Libcurl and c-ares

This is a follow-up to my previous post about async cancelation of a DNS
lookup.

1. Building libcurl with c-ares support.

As suggested by Daniel, I defined USE_ARES in the libcurl project and built
it. There were no compiler or link errors, so I assume libcurl has built-in
c-ares support and it doesn't require any additional h or lib files. Could
you please confirm this. BTW, why doesn't libcurl use ares by default?

2. Using c-ares within curl

I ran the app with the new dll and I didn't notice any obvious difference,
which is what I expected. Now, what are the enhancements that I could take
advantage of at the libcurl API level? Are there new functions or callbacks
that I can use to do async cancel, or to get progress status during a
lengthy DNS request?

Thanks all,

Adrian

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: Libcurl and c-ares

On Thu, Dec 31, 2009 at 7:02 PM, Adrian Michel wrote:

> I assume libcurl has built-in c-ares support and it doesn't require
> any additional h or lib files. Could you please confirm this.

If you built also the curl command-line tool, you can try
curl --version and see if async DNS is mentioned on the
"features" line.

Or you could compile a simple test app, something like:

#include <curl/curl.h>
int main()
{
 curl_version_info_data*info=curl_version_info(CURLVERSION_NOW);
 if (info->features&CURL_VERSION_ASYNCHDNS) {
   printf( "ares enabled\n");
 } else {
   printf( "ares NOT enabled\n");
 }
 return 0;
}

> BTW, why doesn't libcurl use ares by default?

There was a brief mention about this back in August:
  http://curl.haxx.se/mail/lib-2009-08/0014.html

 - Jeff
-------------------------------------------------------------------

Re: [PATCH] FTP PRET support for drftpd servers

So, I copied the test files to tests/data and ran ./runtests.pl 1107.
This is what happens:

********* System characteristics ********
* curl 7.20.0-CVS (x86_64-unknown-linux-gnu)
* libcurl/7.20.0-CVS OpenSSL/0.9.8l zlib/1.2.3 libidn/1.15
* Features: IDN IPv6 Largefile NTLM SSL libz
* System: Linux 2.6.31-gentoo-r6 #2 SMP Sat Dec 5 16:09:26 CET 2009
x86_64 Intel(R) Core(TM) i7 CPU 920  <at>  2.67GHz GenuineIntel GNU/Linux
* Server SSL:        OFF  libcurl SSL:  ON
* debug build:       OFF  track memory: OFF
* valgrind:          OFF  HTTP IPv6     ON
* FTP IPv6           ON   Libtool lib:  OFF
* SSL library:       OpenSSL
* Ports:
*   HTTP/8990 FTP/8992 FTP2/8995
*   TFTP/8997 HTTP-IPv6/8994 FTP-IPv6/8996
*   SSH/8999 SOCKS/9000 POP3/9001 IMAP/9002 SMTP/9003
*****************************************
]est 1107...[FTP RETR PASV with PRET

 data FAILED:
--- log/check-expected  2010-01-01 12:44:35.000000000 +0100
+++ log/check-generated 2010-01-01 12:44:35.000000000 +0100
 <at>  <at>  -1,6 +0,0  <at>  <at> 
-data
-    to
-      see
-that FTP
-works

Re: [PATCH] FTP PRET support for drftpd servers

On Fri, 1 Jan 2010, Ingmar Runge wrote:

> This is what happens:

> So it looks like it's running the wrong binary? Am I missing something here?

First, please don't top-post. Then, I couldn't repeat that failure of yours. I 
would guess that you simply had some mistake in your build or something.

I did however edit the test cases a little bit and I've now committed your 
patch with my test cases to CVS. Please verify that everything is now working 
as expected!

Thanks for your contribution!

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: ftp enhancement - FTP wildcard download

On Thu, 31 Dec 2009, Pavel Raiskup wrote:

>> I would prefer that metadata callback to get called before any data is 
>> delivered, so that an app could get to know the file name etc before any 
>> data is downloaded.
>
> Yes, calling this callback means starting of new file transfer and means end 
> of previous file transfer (gives info about download result, bytes 
> transferred, etc.)

Another approach would be to require a new curl_easy_perform() for each 
subsequent transfer, and just have a not modified URL mean that it will 
continue on the wildcard-transfer. It would solve how to deal with return 
codes and progress meter etc. I think I'd prefer that.

>> Also, if done client-side it might be easier to provide the same matching 
>> concept not only between different FTP servers but also in the future 
>> between FTP and SFTP downloads etc.
>
> I think we should generally expect different type of LIST response too  
> ..

Hm. Are you considering doing a LIST parsing? In my book that is really nasty 
stuff that will take time and lots of work to get working even half decent. 
I was kind of hoping we could get away with doing NLST...

> I thought about it .. I propose, that there could be new CURLOPT_RECURSIVE 
> or something like this and default should be "false" probably?

I propose we simply ignore recursiveness to start with. Recursive is not a 

RE: Async cancel lengthy DNS lookup

I managed to get libcurl to use c-ares as dns resolver. I then I tested with
a domain that can't be resolved. The call to curl_easy_perform takes a while
(15 seconds or so), but I'm still not seeing calls to the progress callback.

Here's my code, (I removed the error handling and anything else that is not
libcurl related to make it clearer), and xprogress does not get called in
this case. It gets called when it gets to the http part though.

/*******************/

// make sure c-ares is used
curl_version_info_data* info= curl_version_info(CURLVERSION_NOW);
if(info->features&CURL_VERSION_ASYNCHDNS)
  std::cout << "ares enabled";
else
  std::cout << "ares NOT enabled";

// start here
curl_easy_setopt( _curl, CURLOPT_URL, url

// these two are optional (there are booleans that are tested)
    curl_easy_setopt( _curl, CURLOPT_SSL_VERIFYPEER, 0L)
    curl_easy_setopt( _curl, CURLOPT_SSL_VERIFYHOST, 0L)

curl_easy_setopt( _curl, CURLOPT_WRITEFUNCTION, responseCallback )
curl_easy_setopt( _curl, CURLOPT_FILE, (void *)&rs)
curl_easy_setopt( _curl, CURLOPT_NOPROGRESS, 0L)
curl_easy_setopt( _curl, CURLOPT_PROGRESSFUNCTION, xprogress)
curl_easy_setopt( _curl, CURLOPT_PROGRESSDATA, this)
curl_easy_setopt( _curl, CURLOPT_HTTPPOST, fp )

RE: Some very early feedback in SMTP testing.

On Mon, 14 Dec 2009, Patrick Monnerat wrote:

> Remember a data line beginning with a dot should be preceded by an 
> additional dot 

This is now supported, and test case 803 verifies it!

> Just a small remark abour CURLOPT_MAIL_RCPT: in the current version, you can 
> only send a mail to a single address. The protocol allows you to specify 
> several RCPT TO statements for the same mail. I would suggest to change 
> CURLOPT_MAIL_RCPT char* argument into a struct slist* for that purpose.

Right, this is now what I've made it into.

> Also forcing <> around the CURLOPT_MAIL_FROM and CURLOPT_MAIL_RCPT values 
> supposes they are e-mail addresses alone, making arguments as "name surname 
> <e-mail-address>" infeasible; extra mail-parameters and rcpt-parameters 
> cannot be specified either.

Indeed, I removed those hardcoded things as well.

Thanks a lot for your continued feedback.

POP3, IMAP and SMTP can now be tested with our standard test suite as the test 
FTP server now has been made capable of speaking the other protocols too.

I will appreciate more feedback and testing of these new protocols as I 
believe they are now starting to become really usable!

--

-- 

RE: Async cancel lengthy DNS lookup

On Fri, 1 Jan 2010, Adrian Michel wrote:

> I managed to get libcurl to use c-ares as dns resolver. I then I tested with 
> a domain that can't be resolved. The call to curl_easy_perform takes a while 
> (15 seconds or so), but I'm still not seeing calls to the progress callback.

No, I just read the source code and I really can't explain why but there's 
indeed a lack of callback-usage during the name resolving. I consider this a 
bug we need to fix. I'll take a look at it asap.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

RE: Async cancel lengthy DNS lookup

On Fri, 1 Jan 2010, Daniel Stenberg wrote:

> No, I just read the source code and I really can't explain why but there's 
> indeed a lack of callback-usage during the name resolving. I consider this a 
> bug we need to fix. I'll take a look at it asap.

Okay, have a look at the attached patch and see if this doesn't make the 
progress callback get called properly!?

--

-- 

  / daniel.haxx.se

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Re: ftp enhancement - FTP wildcard download

>>> ... CLIENT SIDE MATCH
>>> Also, if done client-side it might be easier to provide the same  
>>> matching concept not only between different FTP servers but also in  
>>> the future between FTP and SFTP downloads etc.
>>
>> I think we should generally expect different type of LIST response too  
>>  ..
>
> Hm. Are you considering doing a LIST parsing? In my book that is really  
> nasty stuff that will take time and lots of work to get working even  
> half decent. I was kind of hoping we could get away with doing NLST...

I proposed it because I'd like to know if each filename (in NLST or LIST)
is a directory or regular file. And NLST is quite limiting for that. I can
imagine it could be relatively difficult to parse LIST, but there could be
other advantages like known file-size (could save one ftp command for each
matched file!) last-change time (e.g. for selective download maybe).

Even if we use NLST we should tell the the app information if filename  
means
directory or regular file - it is almost necessary for recursive download
--> even if is it not implemented in libcurl and recursive download is done
on users application (using wildcard as the base). If I use NLST only,
this information (dir/file) will depend only on ftp's state-code --> I'll
try to download regular file (matched filename) and if it fails (550) it is
directory .. 

I'm not sure, if there is 100% way to get necessary information from the
LIST response (where to find filename and permission bits there) and if
servers are so incompatible to do this job impossible. But I'm sure that I