Everything that concerns curl. Questions, bug reports, scripts and related tools and subjects welcome

xNokia | 1 Mar 2011 01:30

Store file troubleshoot with CURL [Serious Issue]

Hi All,

There are an issue with curl when it comes to there are no filename in the URL and no file name on the header as it on megaupload website, if I tried to download a file from this URL

http://www.megaupload.com?d=JLMHQ16K by using the command below

curl -OLJg --cookie /home/AG/cookies/rapidshare2 -w ",%{url_effective},%{size_download},%{content_type}\n" http://www.megaupload.com/?d=JLMHQ16K

The stored filename will be ?d=JLMHQ16K, while it should be crashreport.exe, I tried to edit the curl source code to let it to capture the file name from the latest followed location instead of the initial URL but it doesn't work as I want as it always give coredumped error, I wonder if there anyway to sort the issue by having the CURL to capture the filename from the latest followed location if there are no filename in the header.

Part of the edits I made:-

First shot:

if(!outfile) {

/* NOHEADER FILENAME */

// outfile = get_url_file_name(url);

outfile = "NOHE";

Second shot:

if ("NOHE" == outs.filename) {

char *this_url=NULL;

curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &this_url);

// /* extract the file name from the URL */

outs.filename = get_url_file_name(this_url);

}

Thanks for taking the time to read this email and I will be looking forward for a reply as I'm working on a project which need this function to be sorted.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

Bryan Schaefer | 1 Mar 2011 02:46

Re: is --interface or CURLOPT_INTERFACE bugged?

Hi Dan,

Thanks for the reply. As I continued troubleshooting I discovered that routing was not working in my environment, and I am studying iptables to figure it out. I've got 6 ethernet devices (3 external, 1 dmz, 2 internal), and getting the routing to work properly is proving tricky.

From: Dan Fandrich <dan <at> coneharvesters.com>
To: curl-users <at> cool.haxx.se
Sent: Mon, February 28, 2011 2:13:35 PM
Subject: Re: is --interface or CURLOPT_INTERFACE bugged?

On Sat, Feb 26, 2011 at 04:16:55PM -0800, Bryan Schaefer wrote:
> However, when i try either of the following:
>
> curl --interface eth0 http://freedns.afraid.org (or any interface, or an IP of
> any interface)
>
> or
>
> my $curl = WWW::Curl::Easy->new;
> $curl->setopt(CURLOPT_INTERFACE, 'eth0');
> $curl->setopt(CURLOPT_URL, 'http://freedns.afraid.org');
> $output = $curl->perform;
> print $output;
>
> it times out, returning code 7, could not connect to host.
& gt;
> Am I doing something wrong, or is the interface option wrong?

This is most easily explained by eth0 not being the correct interface to
use :-)

But you're using --interface correctly, so there must be something
going on. What does the -v option show? Does "ping -I eth0 freedns.afraid.org"
work?

>>> Dan
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

kalin m | 1 Mar 2011 11:32

Re: curl ftps


hey Dan...  

thanks for the reply...

no matter what flags i add to the command it always hangs on:

 * SSLv3, TLS handshake, Client hello (1):
} [data not shown]

curl: (35) Unknown SSL protocol error in connection to xx.xx.xx.xx:990 

if i just use ftp://..  curl tries port 21. so if i put :990 after the domain i get the same result as the above.
i did use --ftp-ssl-control. same result.

i tried this from 3 different os machines. only one machine works - an os x. with OpenSSL 1.0.0c 

this is what works on the os x:

curl --ftp-ssl -vvvv -3 -k 'ftps://user:pass <at> xx.xx.xx.xx/in/' -T test113

the system i need it to work on is:

curl 7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz 

by the way i was trying to reinstall openssl and curl. openssl was installed successfully. when it comes
time for curl, running

# ./configure --with-ssl

or 

# ./configure --with-ssl=/usr/local/ssl

i get:

.....
configure: WARNING: SSL disabled, you will not be able to use HTTPS, FTPS, NTLM and more.
configure: WARNING: Use --with-ssl, --with-gnutls, --with-polarssl, --with-nss or --with-axtls to
address this.
......

  SSL support:     no      (--with-{ssl,gnutls,nss,polarssl,axtls} )

and did make clean a few time to make sure i was starting from scratch... 

the new openssl version is OpenSSL 1.0.0d 8 Feb 2011 but curls configure script is not seeing it...

the current curl is still build with 0.9.8g...

this is getting a bit embarrassing.

thanks...

On 2/28/11 2:06 PM, Dan Fandrich wrote:
> On Fri, Feb 25, 2011 at 07:17:29PM -0500, kalin m wrote:
>> i've been trying to use curl to upload a file to an ftps machine with
>> a filezilla server. and it always hangs at the time of the actual file
>> transfer. i would need some help to figure out why...
>>
>> here is what happens:
>>
>> # curl -vvvv -T test4 --ftp-ssl -d -k -3 --cacert 'tatts.crt' 'ftps://user:pass <at> xx.xx.xx.xx/in/'
> 
> Implicit SSL FTP is not well standardized, so it's not surprising you're
> finding issues.  Some servers are set up on port 990 with stunnel, so it's
> impossible to encrypt data traffic with those.  Those servers do the
> equivalent of --ftp-ssl-control.  If you don't need the data traffic
> encrypted, try adding that option.  The best route would be to use explicit
> SSL; since you already have the --ftp-ssl option, just change the ftps:// to
> ftp:// and see what happens.
> 
>>>> Dan
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-users
> FAQ:        http://curl.haxx.se/docs/faq.html
> Etiquette:  http://curl.haxx.se/mail/etiquette.html
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

Dan Fandrich | 1 Mar 2011 22:49

Re: curl ftps

On Tue, Mar 01, 2011 at 05:32:55AM -0500, kalin m wrote:
> thanks for the reply...
> 
> no matter what flags i add to the command it always hangs on:
> 
>  * SSLv3, TLS handshake, Client hello (1):
> } [data not shown]
> 
> curl: (35) Unknown SSL protocol error in connection to xx.xx.xx.xx:990 
> 
> if i just use ftp://..  curl tries port 21. so if i put :990 after the domain i get the same result as the above.
> i did use --ftp-ssl-control. same result.

Yes, with ftp:// curl tries port 21. But with --ftp-ssl, it will
attempt to switch to SSL after connection. This is the only standards-
compliant method of making an FTP SSL connection.

> i tried this from 3 different os machines. only one machine works - an os x. with OpenSSL 1.0.0c 
> 
> this is what works on the os x:
> 
> curl --ftp-ssl -vvvv -3 -k 'ftps://user:pass <at> xx.xx.xx.xx/in/' -T test113

Are you saying that curl actually does connect to the desired server
successfully from one machine? If it works from there, it should work
the same way from other machines. If not, then there's a good chance that
there's a firewall in the way of the other clients. Note that only one -v
is necessary.

> the system i need it to work on is:
> 
> curl 7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18
> Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp 
> Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz 
> 
> by the way i was trying to reinstall openssl and curl. openssl was installed successfully. when it comes
time for curl, running
> 
> # ./configure --with-ssl
> 
> or 
> 
> # ./configure --with-ssl=/usr/local/ssl
> 
> 
> i get:
> 
> .....
> configure: WARNING: SSL disabled, you will not be able to use HTTPS, FTPS, NTLM and more.
> configure: WARNING: Use --with-ssl, --with-gnutls, --with-polarssl, --with-nss or --with-axtls to
address this.
> ......
> 
>   SSL support:     no      (--with-{ssl,gnutls,nss,polarssl,axtls} )
> 
> 
> and did make clean a few time to make sure i was starting from scratch... 
> 
> the new openssl version is OpenSSL 1.0.0d 8 Feb 2011 but curls configure script is not seeing it...
> 
> the current curl is still build with 0.9.8g...

Are you building 7.21.4 or 7.18.2?  If there was a build problem in
7.18.2, it may have been fixed since then. Take a look at the config.log file
to get some clues as to why it isn't picking up the new OpenSSL version.

>>> Dan
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

kalin m | 2 Mar 2011 02:08

Re: curl ftps

On 3/1/11 4:49 PM, Dan Fandrich wrote:
> On Tue, Mar 01, 2011 at 05:32:55AM -0500, kalin m wrote:
>> thanks for the reply...
>>
>> no matter what flags i add to the command it always hangs on:
>>
>>   * SSLv3, TLS handshake, Client hello (1):
>> } [data not shown]
>>
>> curl: (35) Unknown SSL protocol error in connection to xx.xx.xx.xx:990
>>
>> if i just use ftp://..  curl tries port 21. so if i put :990 after the domain i get the same result as the above.
>> i did use --ftp-ssl-control. same result.
> 
> Yes, with ftp:// curl tries port 21. But with --ftp-ssl, it will
> attempt to switch to SSL after connection. This is the only standards-
> compliant method of making an FTP SSL connection.

that's not really what happens. even with --ftp-ssl it tries port 21 and eventually times out.

> 
>> i tried this from 3 different os machines. only one machine works - an os x. with OpenSSL 1.0.0c
>>
>> this is what works on the os x:
>>
>> curl --ftp-ssl -vvvv -3 -k 'ftps://user:pass <at> xx.xx.xx.xx/in/' -T test113
> 
> Are you saying that curl actually does connect to the desired server
> successfully from one machine?

yes.

> If it works from there, it should work
> the same way from other machines.

right.

>> .....
>> configure: WARNING: SSL disabled, you will not be able to use HTTPS, FTPS, NTLM and more.
>> configure: WARNING: Use --with-ssl, --with-gnutls, --with-polarssl, --with-nss or --with-axtls to
address this.
>> ......
>>
>>    SSL support:     no      (--with-{ssl,gnutls,nss,polarssl,axtls} )

> Are you building 7.21.4 or 7.18.2?  If there was a build problem in
> 7.18.2, it may have been fixed since then. Take a look at the config.log file
> to get some clues as to why it isn't picking up the new OpenSSL version.
> 

i did dig into the log. basically the problem was this:

http://curl.haxx.se/mail/archive-2007-11/0013.html

and this helped a lot:

http://curl.haxx.se/mail/tracker-2008-08/0026.html

and then this (cause i did get that awesome -fPIC error):

http://help.directadmin.com/item.php?id=314 

so in short - after some pain i did get this resolved. i'd think that all this should have been resolved in the
last 3 and a half years but i guess it's not a priority.

thank you for your help...

>>>> Dan
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-users
> FAQ:        http://curl.haxx.se/docs/faq.html
> Etiquette:  http://curl.haxx.se/mail/etiquette.html
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

Dan Fandrich | 2 Mar 2011 02:23

Re: curl ftps

On Tue, Mar 01, 2011 at 08:08:39PM -0500, kalin m wrote:
> On 3/1/11 4:49 PM, Dan Fandrich wrote:
> > Yes, with ftp:// curl tries port 21. But with --ftp-ssl, it will
> > attempt to switch to SSL after connection. This is the only standards-
> > compliant method of making an FTP SSL connection.
> 
> that's not really what happens. even with --ftp-ssl it tries port 21 and eventually times out.

If you'd like advice, you'll have to supply more information. A debug log
is a minimum.

> >> i tried this from 3 different os machines. only one machine works - an os x. with OpenSSL 1.0.0c
> >>
> >> this is what works on the os x:
> >>
> >> curl --ftp-ssl -vvvv -3 -k 'ftps://user:pass <at> xx.xx.xx.xx/in/' -T test113
> > 
> > Are you saying that curl actually does connect to the desired server
> > successfully from one machine?
> 
> yes.
> 
> > If it works from there, it should work
> > the same way from other machines.
> 
> right.
> 
> >> .....
> >> configure: WARNING: SSL disabled, you will not be able to use HTTPS, FTPS, NTLM and more.
> >> configure: WARNING: Use --with-ssl, --with-gnutls, --with-polarssl, --with-nss or --with-axtls
to address this.
> so in short - after some pain i did get this resolved. i'd think that all this should have been resolved in the
last 3 and a half years but i guess it's not a priority.

Did you follow the advice in that bug entry? Did you report a problem to
the OpenSSL developers?  I'm not aware of any outstanding problem in the
curl build tools in this area.

>>> Dan
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

kalin m | 2 Mar 2011 02:39

Re: curl ftps


On 3/1/11 8:23 PM, Dan Fandrich wrote:
>>
>> that's not really what happens. even with --ftp-ssl it tries port 21 and eventually times out.
> 
> If you'd like advice, you'll have to supply more information. A debug log
> is a minimum.

you are right. here:

# curl -vvvv --ftp-ssl -k -3 'ftp://user:pass <at> xx.xx.xx.xx/in/' -T test900 --ftp-ssl-control
* About to connect() to xx.xx.xx.xx port 21 (#0)
*   Trying xx.xx.xx.xx... Connection timed out
* couldn't connect to host
* Closing connection #0

curl: (7) couldn't connect to host

> 
>>>> i tried this from 3 different os machines. only one machine works - an os x. with OpenSSL 1.0.0c
>>>>
>>>> this is what works on the os x:
>>>>
>>>> curl --ftp-ssl -vvvv -3 -k 'ftps://user:pass <at> xx.xx.xx.xx/in/' -T test113
>>>
>>> Are you saying that curl actually does connect to the desired server
>>> successfully from one machine?
>>
>> yes.
>>
>>> If it works from there, it should work
>>> the same way from other machines.
>>
>> right.
>>
>>>> .....
>>>> configure: WARNING: SSL disabled, you will not be able to use HTTPS, FTPS, NTLM and more.
>>>> configure: WARNING: Use --with-ssl, --with-gnutls, --with-polarssl, --with-nss or --with-axtls
to address this.
>> so in short - after some pain i did get this resolved. i'd think that all this should have been resolved in
the last 3 and a half years but i guess it's not a priority.
> 
> Did you follow the advice in that bug entry?

yes. the one on the very bottom: Comment By: Dan Fandrich (dfandrich) Date: 2008-07-24 20:50. (i guess this
is your comment?!)

> Did you report a problem to
> the OpenSSL developers? 

no, i have not. i guess i should.

> I'm not aware of any outstanding problem in the
> curl build tools in this area.

again, thanks for you help with this. much appreciated.

>>>> Dan
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-users
> FAQ:        http://curl.haxx.se/docs/faq.html
> Etiquette:  http://curl.haxx.se/mail/etiquette.html
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

Don Cohen | 1 Mar 2011 22:48

curl: (35) error:14094412:SSL routines:SSL3_READ_BYTES:sslv3


Here's my command 
 curl -E /root/key-cert https://ewsg-cert.prudent...
and the result:
 curl: (35) error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert
 bad certificate 

This seems weird to me:
  Note that this certificate is the 
  private key and the private certificate concatenated! 

I assume the private certificate is just the certificate.
In any case my file key-cert looks like
-----BEGIN RSA PRIVATE KEY-----
 ...
-----END RSA PRIVATE KEY----- 
-----BEGIN CERTIFICATE----- 
 ...
-----END CERTIFICATE----- 

What does the error above mean and what could I possibly be doing
wrong to cause it?
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

Daniel Stenberg | 2 Mar 2011 22:07

Re: curl ftps

On Tue, 1 Mar 2011, kalin m wrote:

> # curl -vvvv --ftp-ssl -k -3 'ftp://user:pass <at> xx.xx.xx.xx/in/' -T test900 --ftp-ssl-control
> * About to connect() to xx.xx.xx.xx port 21 (#0)
> *   Trying xx.xx.xx.xx... Connection timed out
> * couldn't connect to host
> * Closing connection #0
>
> curl: (7) couldn't connect to host

This problem has nothing to do with FTPS. You simply fail to establish a 
TCP connection to the FTP server on port 21. Presumably due to a firewall or 
similar.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

headers

Gabriel Thörnblad | 3 Mar 2011 14:26

Using options -C and -z

Hi all,
I am trying to use curl to syncronize large files between two machines 
with a misbehaving connection. Since the files are large, the -C - 
(--continue-at) is very nice feature. The -z (--time-cond) also works as 
expected and only tries to download files that have been updated since 
the last download. My problem is when I am trying to use these options 
at the same time while automating the procedure.

This is the command I am using:
curl -R -z "$localfile" -C - -o "$localfile" "$URL"

I would have hoped that if curl realizes that the remote file is newer 
than the local, it would discard the "-C" option and restart the 
download from byte 0. Unfortunately for me, curl now tries to resume 
from the last position in the local(older) file and corrupts my download.

Have I misunderstood something, or am I trying to use curl in ways it 
was not intented?

Cheers,
Gabriel

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

29	May 2013
36	April 2013
44	March 2013
39	February 2013
27	January 2013
18	December 2012
47	November 2012
41	October 2012
37	September 2012
40	August 2012
30	July 2012
78	June 2012
122	May 2012
101	April 2012
69	March 2012
43	February 2012
72	January 2012
51	December 2011
50	November 2011
36	October 2011
43	September 2011
89	August 2011
51	July 2011
62	June 2011
55	May 2011
84	April 2011
78	March 2011
55	February 2011
98	January 2011
43	December 2010
66	November 2010
90	October 2010
65	September 2010
73	August 2010
67	July 2010
46	June 2010
89	May 2010
112	April 2010
83	March 2010
58	February 2010
38	January 2010
59	December 2009
104	November 2009
42	October 2009
67	September 2009
76	August 2009
91	July 2009
79	June 2009
75	May 2009
122	April 2009
121	March 2009
134	February 2009
157	January 2009
96	December 2008
84	November 2008
88	October 2008
110	September 2008
167	August 2008
183	July 2008
113	June 2008
124	May 2008
141	April 2008
105	March 2008
86	February 2008
98	January 2008
107	December 2007
127	November 2007
155	October 2007
92	September 2007
113	August 2007
63	July 2007
129	June 2007
101	May 2007
108	April 2007
132	March 2007
148	February 2007
171	January 2007
86	December 2006
154	November 2006
141	October 2006
165	September 2006
78	August 2006
87	July 2006
83	June 2006
84	May 2006
77	April 2006
91	March 2006
105	February 2006
132	January 2006
102	December 2005
152	November 2005
138	October 2005
123	September 2005
169	August 2005
95	July 2005
145	June 2005
107	May 2005
153	April 2005
153	March 2005
168	February 2005
175	January 2005
130	December 2004
172	November 2004
105	October 2004
108	September 2004
80	August 2004
132	July 2004
78	June 2004
139	May 2004
73	April 2004
147	March 2004
101	February 2004
101	January 2004
110	December 2003
139	November 2003
205	October 2003
122	September 2003
169	August 2003
207	July 2003
133	June 2003
181	May 2003
231	April 2003
139	March 2003
145	February 2003
171	January 2003
124	December 2002
123	November 2002
202	October 2002
81	September 2002
108	August 2002
76	July 2002

Mon	Tue	Wed	Thu	Fri	Sat	Sun

	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30	31