Graeme Clark | 17 Oct 13:01 2014

Dev Versions Missing

http://curl.haxx.se/download.html 

The two Win32 - MSVC dev links 7.18.0 and 7.19.3 are broken. Thought i'd just let someone know. :)

Graeme
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Nick Zitzmann | 17 Oct 07:49 2014

Important note for curl users on OS X Yosemite 10.10

It's time to update this note I wrote for Mavericks,
<http://curl.haxx.se/mail/archive-2013-10/0036.html>, for Yosemite users.

In Yosemite, they switched from version 7.30.0 to 7.37.1. Apple's own Secure Transport engine, not
OpenSSL, is still used for TLS. And there have been a few changes since my last note that affect you if you use
curl to access servers that use TLS:

1. You can now use the --cacert option again, for the purpose of connecting to servers with self-signed
certificates. The catch? It only works with a single PEM- or DER-encoded certificate, and it ignores
additional certificates in the file. This is a known problem in that version.

If this becomes a problem for you, then you can work around it by either:
1a. Using a newer version of curl than the one that comes with Yosemite. curl 7.38.0 supports certificate
bundles when using the Secure Transport engine.
    -or-
1b. Import the certificate bundle into your Keychain, and then discontinue your use of the --cacert option altogether.

I also think that using the --cacert and --insecure options together will result in an error, so don't do
that. :)

2. You can now use the -E/--cert option, for the purpose of authenticating with a TLS host using a client
certificate and private key. When using the option, you can either specify:
2a. The name of the certificate as it appears in your Keychain (the certificate's private key has to be
present in the same Keychain in order for this to work),
    -or-
2b. A path to a PKCS#12-encoded file on a disk, which contains both the certificate and the private key. (If
it's in the present working directory, you need to add a ./ to the start of the path, or curl will assume you
want to search the Keychain.)

Note that the file **must** be in P12 (PKCS#12) format. We can't load client certificates in PEM or DER
format, as well as their private keys, because the API that would be necessary to make that work is
unfortunately private, and I'd rather not have to explain to all of you why I got your curl-utilizing apps
rejected from the App Store.

3. The -2/--sslv2 option will now raise an error if you try to use it. (Previously, the option was ignored.)
Support for SSLv2 was removed from Secure Transport back in OS X 10.8, and it's not coming back. If you need
to access a very old (1995-era) Web site that does not support at least SSLv3 or later, then you'll still
need to build your own curl and use OpenSSL instead.

4. The --ssl-allow-beast option will now work, but I don't recommend using it unless you **really** know
what you're doing. By default, curl will try to work around the BEAST problem when connecting to a site that
uses CBC over TLS 1.0.

Nick Zitzmann
<http://www.chronosnet.com/>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Bruno Thomsen | 15 Oct 12:48 2014
Picon

[PATCH] mk-ca-bundle: added SHA-384 signature algorithm

Certificates based on SHA-1 are being phased out[1].
So we should expect a rise in certificates based on SHA-2.
Adding SHA-384 as a valid signature algorithm.

[1] https://blog.mozilla.org/security/2014/09/23/phasing-out-certificates-with-sha-1-based-signature-algorithms/

Signed-off-by: Bruno Thomsen <bth <at> kamstrup.dk>
---
 docs/mk-ca-bundle.1 | 2 +-
 lib/mk-ca-bundle.pl | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/docs/mk-ca-bundle.1 b/docs/mk-ca-bundle.1
index aa38612..7d38dba 100644
--- a/docs/mk-ca-bundle.1
+++ b/docs/mk-ca-bundle.1
 <at>  <at>  -87,7 +87,7  <at>  <at>  each certificate and output when run in plain text mode.

 Valid algorithms are:
 .RS
-ALL, NONE, MD5 (default), SHA1, SHA256, SHA512
+ALL, NONE, MD5 (default), SHA1, SHA256, SHA384, SHA512
 .RE
 .IP -u
 unlink (remove) certdata.txt after processing
diff --git a/lib/mk-ca-bundle.pl b/lib/mk-ca-bundle.pl
index 51af5c9..4278e82 100755
--- a/lib/mk-ca-bundle.pl
+++ b/lib/mk-ca-bundle.pl
 <at>  <at>  -56,7 +56,7  <at>  <at>  $opt_d = 'release';
 # If the OpenSSL commandline is not in search path you can configure it here!
 my $openssl = 'openssl';

-my $version = '1.23';
+my $version = '1.24';

 $opt_w = 76; # default base64 encoded lines length

 <at>  <at>  -97,6 +97,7  <at>  <at>  my  <at> valid_signature_algorithms = (
   "MD5",
   "SHA1",
   "SHA256",
+  "SHA384",
   "SHA512"
 );

--

-- 
1.9.1

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Al Grant | 11 Oct 23:48 2014
Picon

Newbie help : logging into website with cookies

Hi All,

I would really appreicate any help on this. I have been trying to
login to a website and POST data to a second page for about 3 days now
without success.

I can acheive a login by copying the curl commands from firefox, but
if I try to script it, I fail. In essence when I try to script it I am
removing the -H "Cookie:...." for -b

Here is what I get from FF:

curl "https://www.anpronline.net/j_spring_security_check" -H "Host:
www.anpronline.net" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1;
WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" -H "Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H
"Accept-Language: en-GB,en;q=0.5" -H "Accept-Encoding: gzip, deflate"
-H "Referer: https://www.anpronline.net/index.html" -H "Cookie:
__utma=86946376.583746695.1412828339.1413009922.1413057495.9;
__utmz=86946376.1412828339.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not"%"20provided);
JSESSIONID=113idx1ipor6y10oul7yy6uaoi;
AWSELB=A7A737371AF52D0B544DAF902D9C2A0C5FBFC37325B3CB707E1EFDCF041AE1E91355018DDFD9065F26A21E422A304806A548114E34518568CA193FB0649451B57AE20FD01E;
__utmc=86946376; __utmb=86946376.10.10.1413057495; __utmt=1" -H
"Connection: keep-alive" --data
"j_username=al"%"40test.co.nz&j_password=SECRET"

curl "https://www.anpronline.net/blacklists/bl.html" -H "Host:
www.anpronline.net" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1;
WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" -H "Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H
"Accept-Language: en-GB,en;q=0.5" -H "Accept-Encoding: gzip, deflate"
-H "Referer: https://www.anpronline.net/blacklists.html" -H "Cookie:
__utma=86946376.583746695.1412828339.1413009922.1413057495.9;
__utmz=86946376.1412828339.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not"%"20provided);
JSESSIONID=kyr8s0yhfj408gbfauj4yroh;
AWSELB=A7A737371AF52D0B544DAF902D9C2A0C5FBFC37325B3CB707E1EFDCF041AE1E91355018DDFD9065F26A21E422A304806A548114E34518568CA193FB0649451B57AE20FD01E;
__utmc=86946376; __utmb=86946376.14.10.1413057495; __utmt=1" -H
"Connection: keep-alive" --data "name=test&siteId=0"

And to script it:

#!/bin/bash
#new
curl --cookie-jar /tmp/anpronline.cookie "https://anpronline.net"

curl "https://www.anpronline.net/j_spring_security_check" -H "Host:
www.anpronline.net" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1;
WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" -H "Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H
"Accept-Language: en-GB,en;q=0.5" -H "Accept-Encoding: gzip, deflate"
-H "Referer: https://www.anpronline.net/index.html" "Connection:
keep-alive" -b /tmp/anpronline.cookie --data
"j_username=al"%"40test.co.nz&j_password=SECRET"

curl "https://www.anpronline.net/blacklists/bl.html" -H "Host:
www.anpronline.net" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1;
WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" -H "Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H
"Accept-Language: en-GB,en;q=0.5" -H "Accept-Encoding: gzip, deflate"
-H "Referer: https://www.anpronline.net/blacklists.html" -b
/tmp/anpronline.cookie -H "Connection: keep-alive" --data
"name=test&siteId=0"

Could someone please help?

Many thanks,

--

-- 
"Beat it punk!"
- Clint Eastwood
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Rodrigo Zanatta Silva | 11 Oct 22:38 2014
Picon

The fastest way to download a list of URL

I am writing a console program and need to download 10K URLs. I asked this question here in stack overflow.

I just realize I can write files in any way I want. So, I can write a bash script with all curl command. And because I want create N threads, I can write N bash script files  with 10K/N lines and open the N files at same time in background. 

Is this the easy and best strategy to speed up the downloads?

There isn't a program/command or anything that I input with a list of URL with the path to save and it will download in N threads as fast as my computer can do it?
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
David M | 9 Oct 22:40 2014
Picon

curl 7.29.0 on centos7 unexpected timeout

Hi

I am running latest curl binaries bundled with CentOS7 to measure responsiveness of a website using an example found online.

# rpm -q curl
curl-7.29.0-19.el7.x86_64

A review of the data showed latency I was not expecting.

The curl command I am running is: curl -4 -s http://www.example.com

When I run curl through strace there was a noticeable, always repeatable, delay between lines 4 and 5 of the following output:

13578 00:54:17.693069 [00007f95081a0f50] close(3) = 0
13578 00:54:17.693274 [00007f950818ba57] madvise(0x7f94fd350000, 8368128, MADV_DONTNEED) = 0
13578 00:54:17.693406 [00007f9508463e82] _exit(0) = ?
13578 00:54:17.693494 [????????????????] +++ exited with 0 +++
13577 00:54:17.835138 [00007f9508186a4d] <... poll resumed> ) = 0 (Timeout)
13577 00:54:17.835321 [00007fff1affe7d2] clock_gettime(CLOCK_MONOTONIC, {100585, 105149583}) = 0
13577 00:54:17.835499 [00007fff1affe7d2] clock_gettime(CLOCK_MONOTONIC, {100585, 105319542}) = 0

Running curl with additional strace flags (later in the day so timestamps are different) showed a poll timeout that accounted for the lost milliseconds.

14:02:54.851738 [00007f418fdc4fe1] clone(child_stack=0x7f4185783eb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f41857849d0, tls=0x
7f4185784700, child_tidptr=0x7f41857849d0) = 26910 <0.000183>
14:02:54.852046 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 181497670}) = 0 <0.000023>
14:02:54.852179 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 181632916}) = 0 <0.000035>
14:02:54.852271 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 181734496}) = 0 <0.000018>
14:02:54.852352 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 181802453}) = 0 <0.000023>
14:02:54.852433 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 181890580}) = 0 <0.000018>
14:02:54.852510 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182085558}) = 0 <0.000168>
14:02:54.852750 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182211352}) = 0 <0.000037>
14:02:54.852848 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182310406}) = 0 <0.000030>
14:02:54.852939 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182388637}) = 0 <0.000026>
14:02:54.853029 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182490719}) = 0 <0.000032>
14:02:54.853119 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182583486}) = 0 <0.000026>
14:02:54.853199 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182649849}) = 0 <0.000022>
14:02:54.853320 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 182832561}) = 0 <0.000099>
14:02:54.853483 [00007f418fdbaa4d] poll(0, 0, 150) = 0 (Timeout) <0.150283>
14:02:55.003886 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 333381094}) = 0 <0.000054>
14:02:55.004075 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 333548521}) = 0 <0.000050>
14:02:55.004206 [00007fffaabfe7d2] clock_gettime(CLOCK_MONOTONIC, {2381, 333677667}) = 0 <0.000225>

I self-compiled 7.29.0 and installed it in /curl/ on the same host.

# /curl/7.29.0/bin/curl -V
curl 7.29.0 (x86_64-unknown-linux-gnu) libcurl/7.29.0 OpenSSL/1.0.1e zlib/1.2.7 libidn/1.28
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: IDN IPv6 Largefile NTLM NTLM_WB SSL libz

# curl -V (this is curl from CentOS)
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.15.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz

The behavior is responsive as expected and the trace output diverges significantly from the bundled curl.

Here is gdb trace output (traced on system calls) for my compiled curl.

Catchpoint 1 (returned from syscall close), 0x00007ffff76927f0 in __close_nocancel () at ../sysdeps/unix/syscall-template.S:81
81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) c
Continuing.

Catchpoint 1 (call to syscall rt_sigaction), 0x00007ffff75e074a in __GI___libc_sigaction (sig=13, act=act <at> entry=0x7fffffffdf30, oact=oact <at> entry=0x7fffffffdfd0) at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:64
64        result = INLINE_SYSCALL (rt_sigaction, 4,
(gdb) c
Continuing.

Catchpoint 1 (returned from syscall rt_sigaction), 0x00007ffff75e074a in __GI___libc_sigaction (sig=13, act=act <at> entry=0x7fffffffdf30, oact=oact <at> entry=0x7fffffffdfd0) at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:64
64        result = INLINE_SYSCALL (rt_sigaction, 4,
(gdb) c
Continuing.

Catchpoint 1 (call to syscall brk), 0x00007ffff7697fec in __brk (addr=addr <at> entry=0x0) at ../sysdeps/unix/sysv/linux/x86_64/brk.c:31
31        __curbrk = newbrk = (void *) INLINE_SYSCALL (brk, 1, addr);
(gdb) c
Continuing.

Catchpoint 1 (returned from syscall brk), 0x00007ffff7697fec in __brk (addr=addr <at> entry=0x0) at ../sysdeps/unix/sysv/linux/x86_64/brk.c:31
31        __curbrk = newbrk = (void *) INLINE_SYSCALL (brk, 1, addr);
(gdb) c
Continuing.


Here is gdb trace output (traced on system calls) for the bundled curl and shows where it diverges from the previous output:
Nowhere in my compiled version gdb output is there a reference to pthread_once.

Catchpoint 1 (returned from syscall close), 0x00007ffff77532a0 in __close_nocancel () at ../sysdeps/unix/syscall-template.S:81
81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) c
Continuing.

Catchpoint 1 (call to syscall rt_sigaction), 0x00007ffff73b974a in __GI___libc_sigaction (sig=sig <at> entry=13, act=act <at> entry=0x7fffffffdf50, oact=oact <at> entry=0x7fffffffdff0) at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:64
64        result = INLINE_SYSCALL (rt_sigaction, 4,
(gdb) c
Continuing.

Catchpoint 1 (returned from syscall rt_sigaction), 0x00007ffff73b974a in __GI___libc_sigaction (sig=sig <at> entry=13, act=act <at> entry=0x7fffffffdf50, oact=oact <at> entry=0x7fffffffdff0) at ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:64
64        result = INLINE_SYSCALL (rt_sigaction, 4,
(gdb) c
Continuing.

Catchpoint 1 (call to syscall futex), pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:128
128     4:      addq    $8, %rsp
(gdb) c
Continuing.




I would prefer to go back to the bundled curl so I hope there is a simple config change I can make.

Does anyone know what is causing the threaded behavior and more importantly to me the timeout I have described above with curl bundled on CentOS7?

Let me know if additional detail is needed.



-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Sky (Jim Schuyler | 9 Oct 05:58 2014
Picon

CURLOPT_CERTINFO truncated to 2048 chars

I’m using php5-curl for HTTPS and use the CURLOPT_CERTINFO option to report back the certificates that are seen and the narrative of the certificate checking process.

The information returned for a cert in that flow is truncated to 2048 bytes from the start of “——BEGIN CERTIFICATE——“ to wherever the 2048 bytes end. Sometimes the ——END CERTIFICATE—— is within this range and sometimes not. 

(I emphasize that the report itself may be way longer than 2048, but the cert info itself in all cases is truncated to 2048 characters before the report runs on with “*” and the next line of the report.)

Consequently I can’t actually see, save or compare the full cert that was presented by the server.

In addition, sometimes when the full certificate is shorter than 2048 bytes there will be junk at the end after the ——END CERTIFICATE—— and other times it’s cleanly ended at the actual end of the cert. The kind of junk that appears there is another story — sometimes looks like background memory and sometimes like cert info, but not from the cert that’s being checked.

I”m using libcurl 7.35 (it’s the latest I have available on a clean Ubuntu 14.04 LTS install) and have checked the email list archives going a year back but found nothing regarding this being reported or fixed. If there’s a way to use apt-get to upgrade to the current version, I can try it, but I don’t really know how to do that.

I’m also happy to go check the current code and have downloaded the source, and can wade into that next, but perhaps you know already where to look.

-Sky

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Sky (Jim Schuyler)
—The future has arrived, and the label says “some assembly required.” 










^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Keeping the flame of free speech 
      and human rights alive online

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 7 Oct 20:01 2014
Picon

Re: http/2 multiple post requests

On Tue, 7 Oct 2014, Scott Mitchell wrote:

> 1)  Regardless of how my server is behaving I'm not able to justify curl
> creating a second connection (after it has already issued both requests on
> the first connection) here.

That looks like a bug.

> 2) Also, why is the output (-o and --trace-ascii) from the first request
> being erased (or not generated at all)?

I don't know. Debug it and figure it out?

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Ed Judge | 5 Oct 15:48 2014
Picon

AWS S3

Wondering if anyone has used curl to download files from AWS S3 and if there is a good example of how to do it.

Thanks,
Ed
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Bisera Milosheska | 4 Oct 00:39 2014
Picon

curl and nghttp2

Hi,

I am trying to compare the page load times for HTTP and HTTP/2 over plain TCP and over TLS. I have managed to establish an HTTP/2 over TLS connection to a nghttpd server, but I do not manage to establish the other types of connections. I get a result only with the following command:

$ curl --http2 -k 'https://10.0.0.110:8080/index.html' -s -D - -o /dev/null


HTTP/2.0 200

server:nghttpd nghttp2/0.6.3-DEV

content-length:29287

cache-control:max-age=3600

date:Fri, 03 Oct 2014 21:45:40 GMT

last-modified:Fri, 03 Oct 2014 16:24:11 GMT


Does anyone have any smart ideas on how to do this? Thanks in advance.

Regards,
Bisera

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Mike Akiba | 29 Sep 21:41 2014
Picon

Multiple filename do not work with -O

I want to download three files:

one.jpg
two.jpg
three.jpg

After I run:

mike <at> mike-pc:~/www$ curl -# -R -O http://www.example.com/{one,two,three}.jpg

I ONLY get one.jpg on my disk:

mike <at> mike-pc:~/www$ ls
one.jpg

two.jpg and three.jpg goes to stdout though.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Gmane