headers
Colin Fleming | 1 Nov 2009 03:31
Picon

Problem with http-client in chicken 4.2.0

Hi all,

I just reinstalled my computer, and as part of that I upgraded from Chicken 3 to 4.2.0. I'm having some problems with the new http-client interface and I'm not sure if I'm just doing something daft. When trying to use the Google Reader API I always get a 403:

#;4> (call-with-input-request "https://www.google.com/accounts/ClientLogin?Email=<email>&Passwd=<password>&source=GooglecURL-Example&service=reader" #f port->string)

Error: (call-with-input-request) Client error: 403 Forbidden

    Call history:

    uri-common.scm: 150  uri-generic#uri-port       
    uri-generic.scm: 97   URI-authority       
    uri-generic.scm: 98   URIAuth-port       
    uri-common.scm: 151  uri-generic#uri-scheme       
    uri-common.scm: 151  alist-ref       
    http-client.scm: 79   uri-common#uri-host       
    http-client.scm: 79   uri-common#uri-host       
    http-client.scm: 448  raise            <--

Calling that same URL with curl works fine, and this worked in the chicken 3 http-client using http:GET. Calling, for example, http://www.google.com works fine too. Is there something obvious I'm missing? Is there any easy way to debug this?

Thanks,
Colin

_______________________________________________
Chicken-users mailing list
Chicken-users <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Permalink | Reply | 11 comments |
headers
John Cowan | 1 Nov 2009 04:04

Re: Problem with http-client in chicken 4.2.0

Colin Fleming scripsit:

> #;4> (call-with-input-request "
> https://www.google.com/accounts/ClientLogin?Email=<email>&Passwd=<password>&source=GooglecURL-Example&service=reader"

That's an https: URL.  Does http-client support those?

--

-- 
Híggledy-pìggledy / XML programmers            John Cowan
Try to escape those / I-eighteen-N woes;        http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      cowan <at> ccil.org
Unicode weenies and / François Yergeaus.
Permalink | Reply | 10 comments |
headers
Colin Fleming | 1 Nov 2009 10:50
Picon

Re: Problem with http-client in chicken 4.2.0

I assume so - it doesn't explicitly say so in the doc, but the previous version did and this version requires openssl. I'm assuming it must be something that's changed in the way the request is constructed - the user agent, maybe?

Cheers,
Colin

2009/11/1 John Cowan <cowan <at> ccil.org>
Colin Fleming scripsit:

> #;4> (call-with-input-request "
> https://www.google.com/accounts/ClientLogin?Email=<email>&Passwd=<password>&source=GooglecURL-Example&service=reader"

That's an https: URL.  Does http-client support those?

--
Híggledy-pìggledy / XML programmers            John Cowan
Try to escape those / I-eighteen-N woes;        http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      cowan <at> ccil.org
Unicode weenies and / François Yergeaus.

_______________________________________________
Chicken-users mailing list
Chicken-users <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Permalink | Reply | 9 comments |
headers
Colin Fleming | 1 Nov 2009 12:53
Picon

Re: Problem with http-client in chicken 4.2.0

Using netcat and stunnel to see what's going on, it's very strange, I can make the exact same request with http-client and curl, curl works but http-client returns a 403:

GET /accounts/ClientLogin?Email=<email>&Passwd=<passwd>&source=Google-cURL-Example&service=reader HTTP/1.1
User-Agent: curl/7.19.6 (i386-apple-darwin10.0.0) libcurl/7.19.6 OpenSSL/0.9.8k zlib/1.2.3
Host: localhost:8889
Accept: */*

GET /accounts/ClientLogin?Email=<email>;Passwd=<passwd>;source=Google-cURL-Example;service=reader HTTP/1.1
Accept: */*
User-Agent: curl/7.19.6 (i386-apple-darwin10.0.0)
Host: localhost:8889

The first works, the second gives me a BadAuthentication error. That's http-client pretending to be curl. My call looks like the following:

(call-with-input-request
  (make-request method: 'GET
                uri: (uri-reference "http://localhost:8889/accounts/ClientLogin?Email=<email>&Passwd=<passwd>&source=Google-cURL-Example&service=reader")
                headers: (headers `((accept . (*/*)))))
  #f
  port->string)

The request is to localhost:8889 for the netcat/stunnel proxy. Any ideas why this might fail?

Cheers,
Colin


2009/11/1 Colin Fleming <colin.mailinglist <at> gmail.com>
I assume so - it doesn't explicitly say so in the doc, but the previous version did and this version requires openssl. I'm assuming it must be something that's changed in the way the request is constructed - the user agent, maybe?

Cheers,
Colin

2009/11/1 John Cowan <cowan <at> ccil.org>

Colin Fleming scripsit:

> #;4> (call-with-input-request "
> https://www.google.com/accounts/ClientLogin?Email=<email>&Passwd=<password>&source=GooglecURL-Example&service=reader"

That's an https: URL.  Does http-client support those?

--
Híggledy-pìggledy / XML programmers            John Cowan
Try to escape those / I-eighteen-N woes;        http://www.ccil.org/~cowan
Incontrovertibly / What we need more of is      cowan <at> ccil.org
Unicode weenies and / François Yergeaus.


_______________________________________________
Chicken-users mailing list
Chicken-users <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Permalink | Reply | 8 comments |
headers
Peter Bex | 1 Nov 2009 14:14
Picon
Picon
Favicon

Re: Problem with http-client in chicken 4.2.0

On Sun, Nov 01, 2009 at 12:53:06PM +0100, Colin Fleming wrote:
> Using netcat and stunnel to see what's going on, it's very strange, I can
> make the exact same request with http-client and curl, curl works but
> http-client returns a 403:
> 
> GET
> /accounts/ClientLogin?Email=<email>&Passwd=<passwd>&source=Google-cURL-Example&service=reader
> HTTP/1.1
> User-Agent: curl/7.19.6 (i386-apple-darwin10.0.0) libcurl/7.19.6
> OpenSSL/0.9.8k zlib/1.2.3
> Host: localhost:8889
> Accept: */*
> 
> The first works, the second gives me a BadAuthentication error. That's
> http-client pretending to be curl. My call looks like the following:
> 
> The request is to localhost:8889 for the netcat/stunnel proxy. Any ideas why
> this might fail?

Possibly your request to google results in a 302 or 301 redirect with a
cookie header, which causes a second request to be sent.  I've
recently discovered that a *lot* of popular internet software doesn't
treat cookie names in a case-insensitive fashion (which it should,
according to the relevant RFCs), and intarweb does case normalization
on cookies, causing trouble with some software.

So maybe the original request is okay, but the other one is causing the
problems because the cookie isn't sent with exactly the same name.
I don't know the google API, so I'm just guessing here.  Try tcpdump to
check out the entire HTTP conversation.

Cheers,
Peter
--

-- 
http://sjamaan.ath.cx
--
"The process of preparing programs for a digital computer
 is especially attractive, not only because it can be economically
 and scientifically rewarding, but also because it can be an aesthetic
 experience much like composing poetry or music."
							-- Donald Knuth
Permalink | Reply | 1 comment |
headers
Colin Fleming | 1 Nov 2009 15:26
Picon

Re: Problem with http-client in chicken 4.2.0

Hi Peter,

Thanks for the suggestion - there doesn't seem to be any redirect involved, I just get a 403 back. I'm not near my project right now, but I just tried with curl and I get a 200 directly from that request with the information. It's very strange. I'll try tcpdump this evening.

Cheers,
Colin

2009/11/1 Peter Bex <Peter.Bex <at> xs4all.nl>
On Sun, Nov 01, 2009 at 12:53:06PM +0100, Colin Fleming wrote:
> Using netcat and stunnel to see what's going on, it's very strange, I can
> make the exact same request with http-client and curl, curl works but
> http-client returns a 403:
>
> GET
> /accounts/ClientLogin?Email=<email>&Passwd=<passwd>&source=Google-cURL-Example&service=reader
> HTTP/1.1
> User-Agent: curl/7.19.6 (i386-apple-darwin10.0.0) libcurl/7.19.6
> OpenSSL/0.9.8k zlib/1.2.3
> Host: localhost:8889
> Accept: */*
>
> The first works, the second gives me a BadAuthentication error. That's
> http-client pretending to be curl. My call looks like the following:
>
> The request is to localhost:8889 for the netcat/stunnel proxy. Any ideas why
> this might fail?

Possibly your request to google results in a 302 or 301 redirect with a
cookie header, which causes a second request to be sent.  I've
recently discovered that a *lot* of popular internet software doesn't
treat cookie names in a case-insensitive fashion (which it should,
according to the relevant RFCs), and intarweb does case normalization
on cookies, causing trouble with some software.

So maybe the original request is okay, but the other one is causing the
problems because the cookie isn't sent with exactly the same name.
I don't know the google API, so I'm just guessing here.  Try tcpdump to
check out the entire HTTP conversation.

Cheers,
Peter
--
http://sjamaan.ath.cx
--
"The process of preparing programs for a digital computer
 is especially attractive, not only because it can be economically
 and scientifically rewarding, but also because it can be an aesthetic
 experience much like composing poetry or music."
                                                       -- Donald Knuth

_______________________________________________
Chicken-users mailing list
Chicken-users <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Permalink | Reply | 0 comments |
headers
Jörg "F. Wittenberger" | 1 Nov 2009 15:36

Re: Problem with http-client in chicken 4.2.0

I've got no idea what's going on here at all, but
to me the second request looks wrong in any case.

Shouldn't those parameter field separators be "&" and never ";" ???

Am Sonntag, den 01.11.2009, 12:53 +0100 schrieb Colin Fleming:
> Using netcat and stunnel to see what's going on, it's very strange, I
> can make the exact same request with http-client and curl, curl works
> but http-client returns a 403:
> 
> GET
/accounts/ClientLogin?Email=<email>&Passwd=<passwd>&source=Google-cURL-Example&service=reader HTTP/1.1
> User-Agent: curl/7.19.6 (i386-apple-darwin10.0.0) libcurl/7.19.6
> OpenSSL/0.9.8k zlib/1.2.3
> Host: localhost:8889
> Accept: */*
> 
> GET /accounts/ClientLogin?Email=<email>;Passwd=<passwd>;source=Google-cURL-Example;service=reader

                                         ^               ^

>  HTTP/1.1
> Accept: */*
> User-Agent: curl/7.19.6 (i386-apple-darwin10.0.0)
> Host: localhost:8889
> 
> The first works, the second gives me a BadAuthentication error. That's
> http-client pretending to be curl. My call looks like the following:
> 
> (call-with-input-request 
>   (make-request method: 'GET 
>                 uri: (uri-reference
>
"http://localhost:8889/accounts/ClientLogin?Email=<email>&Passwd=<passwd>&source=Google-cURL-Example&service=reader") 
>                 headers: (headers `((accept . (*/*))))) 
>   #f 
>   port->string)
> 
> The request is to localhost:8889 for the netcat/stunnel proxy. Any
> ideas why this might fail?
> 
> Cheers,
> Colin
> 
> 
> 2009/11/1 Colin Fleming <colin.mailinglist <at> gmail.com>
>         I assume so - it doesn't explicitly say so in the doc, but the
>         previous version did and this version requires openssl. I'm
>         assuming it must be something that's changed in the way the
>         request is constructed - the user agent, maybe?
>         
>         Cheers,
>         Colin
>         
>         2009/11/1 John Cowan <cowan <at> ccil.org>
>         
>         
>                 Colin Fleming scripsit:
>                 
>                 > #;4> (call-with-input-request "
>                 >
>                 https://www.google.com/accounts/ClientLogin?Email=<email>&Passwd=<password>&source=GooglecURL-Example&service=reader"
>                 
>                 
>                 That's an https: URL.  Does http-client support those?
>                 
>                 --
>                 Híggledy-pìggledy / XML programmers            John
>                 Cowan
>                 Try to escape those / I-eighteen-N woes;
>                  http://www.ccil.org/~cowan
>                 Incontrovertibly / What we need more of is
>                  cowan <at> ccil.org
>                 Unicode weenies and / François Yergeaus.
>         
> 
> _______________________________________________
> Chicken-users mailing list
> Chicken-users <at> nongnu.org
> http://lists.nongnu.org/mailman/listinfo/chicken-users
> 
> --===============0180303
Permalink | Reply | 5 comments |
headers
Peter Bex | 1 Nov 2009 16:13
Picon
Picon
Favicon

Re: Problem with http-client in chicken 4.2.0

On Sun, Nov 01, 2009 at 03:36:42PM +0100, Jörg F. Wittenberger wrote:
> I've got no idea what's going on here at all, but
> to me the second request looks wrong in any case.
> 
> Shouldn't those parameter field separators be "&" and never ";" ???

Good catch.  It's possible that Google doesn't allow ; as separator.
In that case, parameterize form-urlencoded-separator to "&;" or just "&".

This parameter is for uri-common, which defaults to ";&" because
of the recommendation in the HTML spec to prefer ";" as separator, but
to also allow "&" on input. The reason is the former doesn't have to be
specially encoded in HTML.

Cheers,
Peter
--

-- 
http://sjamaan.ath.cx
--
"The process of preparing programs for a digital computer
 is especially attractive, not only because it can be economically
 and scientifically rewarding, but also because it can be an aesthetic
 experience much like composing poetry or music."
							-- Donald Knuth
Permalink | Reply | 4 comments |
headers
Colin Fleming | 1 Nov 2009 16:35
Picon

Re: Problem with http-client in chicken 4.2.0

Good catch indeed! I'll try this when I get home.

Thanks!
Colin

2009/11/1 Peter Bex <Peter.Bex <at> xs4all.nl>
On Sun, Nov 01, 2009 at 03:36:42PM +0100, Jörg F. Wittenberger wrote:
> I've got no idea what's going on here at all, but
> to me the second request looks wrong in any case.
>
> Shouldn't those parameter field separators be "&" and never ";" ???

Good catch.  It's possible that Google doesn't allow ; as separator.
In that case, parameterize form-urlencoded-separator to "&;" or just "&".

This parameter is for uri-common, which defaults to ";&" because
of the recommendation in the HTML spec to prefer ";" as separator, but
to also allow "&" on input. The reason is the former doesn't have to be
specially encoded in HTML.

Cheers,
Peter
--
http://sjamaan.ath.cx
--
"The process of preparing programs for a digital computer
 is especially attractive, not only because it can be economically
 and scientifically rewarding, but also because it can be an aesthetic
 experience much like composing poetry or music."
                                                       -- Donald Knuth

_______________________________________________
Chicken-users mailing list
Chicken-users <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Permalink | Reply | 3 comments |
headers
Colin Fleming | 1 Nov 2009 18:21
Picon

Re: Problem with http-client in chicken 4.2.0

Unbelievably enough, that was the problem. Thank you all for the help, that would have taken me forever to figure out.

Cheers,
Colin


2009/11/1 Colin Fleming <colin.mailinglist <at> gmail.com>
Good catch indeed! I'll try this when I get home.

Thanks!
Colin

2009/11/1 Peter Bex <Peter.Bex <at> xs4all.nl>
On Sun, Nov 01, 2009 at 03:36:42PM +0100, Jörg F. Wittenberger wrote:

> I've got no idea what's going on here at all, but
> to me the second request looks wrong in any case.
>
> Shouldn't those parameter field separators be "&" and never ";" ???

Good catch.  It's possible that Google doesn't allow ; as separator.
In that case, parameterize form-urlencoded-separator to "&;" or just "&".

This parameter is for uri-common, which defaults to ";&" because
of the recommendation in the HTML spec to prefer ";" as separator, but
to also allow "&" on input. The reason is the former doesn't have to be
specially encoded in HTML.

Cheers,
Peter
--
http://sjamaan.ath.cx
--
"The process of preparing programs for a digital computer
 is especially attractive, not only because it can be economically
 and scientifically rewarding, but also because it can be an aesthetic
 experience much like composing poetry or music."
                                                       -- Donald Knuth


_______________________________________________
Chicken-users mailing list
Chicken-users <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Permalink | Reply | 2 comments |

Gmane