listmeister | 20 Apr 15:00 2014

Logging into TIAA-CREF.org with a Tcl+cURL script -- (broken script posted)

TIAA-CREF has a two step login process.  The first form accepts
username only, and the next page asks for a password and an extra
security question (which never changes).

The script below is able to send the username and get a form asking
for the password.  But when it comes to password submission, the login
page comes back (despite no HTTP errors).  The cookie changes a few
times (which is possibly normal), but there seems to be *fewer*
cookies in the header than what "live http headers" shows firefox to
be exchanging.  

Any ideas?  Any tiaa-cref customers want to collaborate on this?

The script so far:

#!/usr/bin/tclsh8.5

package require TclCurl
package require htmlparse

set url(top)     "https://publictools.tiaa-cref.org"
set userAgent    "Mozilla/5.0"
set login_suburl "unset"

proc extract_form {args} {
    #this procedure is needed to get the first login URL, which contains a unique string instead of a cookie
    global login_suburl

    foreach {tag slash param text} $args {break}
    if {$tag == "form" && [regexp {action..([^\"]*)\".*} $param -> action]} {
(Continue reading)

dev | 18 Apr 18:39 2014

nightly autobuild setup snags


A few weeks ago I was trying to build curl/libcurl on Solaris 10 sparc
and ran into a few test fails. Specifically three tests 815, 816, and
also 1 513 .  After a bit of maillist discussion I felt it may be of
some
benefit to try a nightly autobuild setup which may help here.

So after a bit of sed/awk/grep-fu I have a nightly script which can pull
down the latest tarball. Sorry, git is out of the question until I can
build a recent version of git. Which seems to need every dependency in
the Linux/GNU stack plus the kitchen sink. So tarballs for now.

First problem was that curl-7.37.0-20140418/tests/testcurl.pl wants me
to input my name and email and a system description. I guess I have to
make a response file for that.

Manually running the script didn't get very far because of this :

node002$ ./curl_daily_fetch.sh
INFO : good download
     : HTTP request sent, awaiting response... 200 OK
INFO : about to try autobuild
     : /usr/local/build/curl_autobuild/build/curl-7.37.0-20140418
please enter your name
Dennis Clarke
please enter your contact email address
dev <at> cor0.com
please enter a one line system description
Oracle Solaris 10 8/11 s10s_u10wos_17b kernel=150400-01 cc=Sun C 5.12
UltraSPARC-T2+
(Continue reading)

Claudiu | 16 Apr 16:12 2014
Picon

[re-post] how to make sure that a site's certificate hasn't been revoked?

[Sorry for the formatting in the earlier mail, forgot to turn on "plain text" on my e-mail client. Hope it
works better this time...]

Let's say I want to download the google page with curl, but want to make sure that the certificate for
google.com hasn't been revoked.
So I tried the following command: 

$ curl https://www.google.com --cacert GeoTrust_Global_CA.pem --crlfile gtglobal.pem -v

, where GeoTrust_Global_CA.pem is the root certificate of Google's CA (GeoTrust) and gtglobal.pem is the
CRL (certificate revocation list) associated with the certificate.

The problem is I get the following error: 

* About to connect() to www.google.com port 443 (#0)
*   Trying 81.24.29.106... connected
* successfully set certificate verify locations:
*   CAfile: GeoTrust_Global_CA.pem
  CApath: /etc/ssl/certs
* successfully load CRL file:
*   CRLfile: gtglobal.pem
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
(Continue reading)

Claudiu | 16 Apr 15:28 2014
Picon

how to make sure that a site's certificate hasn't been revoked?

Let's say I want to download the google page with curl, but want to make sure that the certificate for google.com hasn't been revoked.
So I tried the following command:

$ curl https://www.google.com --cacert GeoTrust_Global_CA.pem --crlfile gtglobal.pem -v

, where GeoTrust_Global_CA.pem is the root certificate of Google's CA (GeoTrust) and gtglobal.pem is the CRL (certificate revocation list) associated with the certificate.

The problem is I get the following error:

* About to connect() to www.google.com port 443 (#0)
*   Trying 81.24.29.91... connected
* successfully set certificate verify locations:
*   CAfile: GeoTrust_Global_CA.pem
  CApath: /etc/ssl/certs
* successfully load CRL file:
*   CRLfile: gtglobal.pem
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

I didn't expect an error, since Google should have a valid certificate.

Do you know how I could issue a curl command that does this correctly?

P.S.: If you're wondering how I got those specific files (GeoTrust_Global_CA.pem and gtglobal.pem) from the curl command, this is how I proceeded:
- I first looked at what CA issued the certificate for https://www.google.com. Turns out it is GeoTrust Global CA
- I downloaded the GeoTrust Global CA root certificate from their website: https://www.geotrust.com/resources/root_certificates/certificates/GeoTrust_Global_CA.pem
- I downloaded the corresponding CRL (certificate revocation list) from here: http://crl.geotrust.com/crls/gtglobal.crl

Thanks
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Sandeep Sastry | 16 Apr 14:26 2014
Picon

Re: Uploding large files on Windows fails consistently

Hi,

 

This is in continuation to a very old mail. It took so long to reply because I had to take permission from my company to share the URL and credentials (public cloud domain) where anyone can try the below command and see the results for themselves.

 

Recap: I am trying to upload a file greater than 1Gb from a windows 7 client (64 bit) using the curl version ( mentioned below) which works fine, but if I introduce low speed time out with 120 secs and for low transfer rate of 10 bytes, the upload fails with operation timeout (error 28) consistently.

Strangely I do not see this problem on Linux and Mac when I enable low speed/time settings . I am not able to conclude if it is a problem with the Windows OS handling of sockets or an issue with curl.

 

curl --version

curl 7.35.0 (x86_64-pc-win32) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 WinIDN libssh2/1.4.3

Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp

scp sftp smtp smtps telnet tftp

Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM SPNEGO SSL SSPI libz

 

 

curl -u mycloud <at> s.com:123456 -X PUT -T “path of the file on your system to upload”  http://sandeep.bix.78ppsnzv1uma.bix.com/destinationfilename

 

Could someone out there try uploading a file just above 1Gb using windows command line interface (using credentials and the url I mentioned above) and let me know if it works for you.

 

Thanks and Best Regards,

-Sandeep

 

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Ryan | 16 Apr 05:28 2014

How to make cURL support other SSL library

Dear all

Our group has a tool (mainly shell scripts) developed on basis of cURL. Now our internal security requires the tool to use IBM GSK(Global Security Toolkit) library for SSL. I read the document of cURL, and the GSK is not on the list of supported SSL. So I am wondering how to make it to support GSK, and how big the effort is.

Thanks a lot in advance !
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 12 Apr 23:48 2014
Picon

Yeah, Heartbleed

Hey all,

(cross-posted to both curl-users and curl-library to reach widely, please send 
responses to the proper single list.)

Nobody missed Heartbleed[1] this past week I'm sure. If you did, you must've 
been on an awesomely disconnected vacation.

Anyway, I've gotten numerous questions about curl in this context so I wanted 
to spell out the details once and for all.

Heartbleed is a flaw in OpenSSL in a certain version span. Clients are *also* 
vulnerable to this flaw, which means that if you run curl or libcurl with a 
vulnerable OpenSSL version a rogue server can read client memory.

Again, this is an OpenSSL flaw but since OpenSSL is a library, applications 
that use it will be affected. If you use libcurl using OpenSSL then you are 
affected too.

This is not a flaw in curl nor libcurl, we will not and cannot release 
anything to adress this problem.

Things to do to avoid being affected include:

  - run a fixed OpenSSL version, or an older version from before the flaw was
    introduced

  - build libcurl against the numerous other fine TLS libraries that we support

[1] = http://heartbleed.com/

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Rene Bon Ciric | 12 Apr 01:25 2014

Problem with Curl

Hello,

I am having a problem with curl. It can't access https://rubygems.org/ at all.

The site works fine. It can access https://whichever-other-site.tld/

Output:
http://fpaste.org/93688/13972572/

Working example:
http://fpaste.org/93689/39725745/

It seems like an NSS problem. I already tried deleting /etc/pki/nssdb
and reinstalling; but the results are the same.

Can't uninstall it completely since everything depends on it.

Any hints or ideas are welcome.

-- 
Renich Bon Ciric
Consultant

# Website
http://evalinux.com/

# Address
José María Vigil #1458,
Villaseñor, 44600,
Guadalajara, Jalisco, México

# Tel
+52 (33) 1589-4554

# Mobile
+52 (33) 3576-5013

# VPS
http://goo.gl/yNn9aX

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Priyanka Shah | 12 Apr 01:55 2014

curl crash in setsockopt callback

Hello,
I need to set a particular socket option through libcurl. I register a sockopt callback, and set the socket option when callack is called. When I ran this code, curl crashed with the following trace:


#0  0xf4586435 in sockopt_callback(void*, int, curlsocktype) ()
#1  0xf41f1011 in singleipconnect () from /usr/lib/libcurl.so.5
#2  0xf41f1b04 in Curl_connecthost () from /usr/lib/libcurl.so.5
#3  0xf41e0aa6 in Curl_setup_conn () from /usr/lib/libcurl.so.5
#4  0xf41e0ca6 in Curl_connect () from /usr/lib/libcurl.so.5
#5  0xf41ecd11 in Curl_do_perform () from /usr/lib/libcurl.so.5
#6  0xf41eda43 in curl_easy_perform () from /usr/lib/libcurl.so.5

---------
Here's my code to register for sockopt callback.

            curl_easy_setopt(client, CURLOPT_SOCKOPTDATA, uint32_t data);
            curl_easy_setopt(client, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);


And this is the callback's implementation:

    int sockopt_callback(void *clientp, curl_socket_t curlfd, curlsocktype purpose)
    {
        if(clientp == NULL) {
             printf("error");
             return CURL_SOCKOPT_OK;
        }

        uint32_t data = *(uint32_t*) clientp;
        sock_ctx lSockCtx;
        memset(&lSockCtx, 0, sizeof(lSockCtx));
        lSockCtx.data = data;
        if (setsockopt(curlfd, SOL_IP, <opt-name>, &lSockCtx, sizeof(lSockCtx)) < 0) {
            return CURL_SOCKOPT_ERROR;
        }
        return CURL_SOCKOPT_OK;
    }

I am not sure where to start debugging. Is there something here I am missing? Any help would be appreciated.
Thanks,
Priyanka.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
frank ernest | 9 Apr 22:13 2014
Picon

Multiple curls one cookie file.

If I start multiple curl instances and tell them all to use the same cookie file will they trip over each
other? Cause race conditions? Or corrupt the file?
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

frank ernest | 9 Apr 21:42 2014
Picon

Can curl handle an IRI?

I was reading about how, in atom feeds the URIs are not URLs but instead they are IRIs. If I threw an IRI at curl
would it fetch the resource?
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html


Gmane