Anna Gulaev | 1 Oct 01:32 2009
Picon

Re: server not receiving POST data

On Wed, Sep 30, 2009 at 1:40 PM, Daniel Stenberg <daniel <at> haxx.se> wrote:
On Wed, 30 Sep 2009, Anna Gulaev wrote:

From the debug info, It appears that the expected stuff is sent, but the receiving script receives as standard input only one of the mime separators. The receiving script essentially just echoes its standard input, wrapped in some HTML.

Then I would say the error is in the server end.

I wrote a little web form, and the server script receives data from that just fine. From curl it receives a MIME separator rather than the data.


-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Joshua Kwan | 1 Oct 01:39 2009

Re: Problems related to probing for proxy authentication methods

On 9/29/09 14:54, "Daniel Stenberg" <daniel <at> haxx.se> wrote:
> libcurl shouldn't require that the connection is persistant, and yes it should
> react on the 407 and do a new connection for the subsequent request. I thought
> that already worked... any chance you can write up a test case for the curl
> test suite that repeats this?

Yup. It's attached. (Took me quite a while to figure out how to work the
system.) It doesn't have a datacheck stage yet, I just wanted to pass along
the problem case to you -- when we do engineer a fix I'd be happy to improve
the test case.

BTW, here is a strace log of connecting to the actual CCProxy server.
My guess is that CCProxy is just broken by resetting the connection just
like that, but I'll leave it up to you whether we should support this kind
of situation or not.

I think the erroneous bit is that CCProxy doesn't reset the connection after
sending the 407, it only resets it after it gets more data. I am feeling
like that is very nonstandard behavior:

connect(3, {sa_family=AF_INET, sin_port=htons(8080),
sin_addr=inet_addr("10.114.25.113")}, 16) = -1 EINPROGRESS (Operation now in
progress)
poll([{fd=3, events=POLLOUT}], 1, 299995) = 1 ([{fd=3, revents=POLLOUT}])
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
sendto(3, "CONNECT foo.ourdomain.com:443 HTTP/1.1\r\nHost:
foo.ourdomain.com:443\r\nUser-Agent: curl/7.19.7-CVS
(x86_64-unknown-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8k zlib/1.2.3.3
libidn/1.15 libssh2/1.2\r\nProxy-Connection: Keep-Alive\r\n\r\n", 248,
MSG_NOSIGNAL, NULL, 0) = 248
poll([{fd=3, events=POLLIN|POLLPRI}], 1, 1000) = 1 ([{fd=3,
revents=POLLIN}])
recvfrom(3, "HTTP/1.0 407 Unauthorized\r\nServer:
CCProxy\r\nProxy-Authenticate: Basic realm=\"CCProxy
Authorization\"\r\nCache-control: no-cache\r\n\r\n<h1>Unauthorized
...</h1>\r\n<h2>IP Address: 10.114.25.112:42880<br>\r\nMAC Address:
<br>\r\nServer Time: 2009-09-30 16:28:07<br>\r\nAuth Result: </h2>", 16384,
0, NULL, NULL) = 271
sendto(3, "CONNECT foo.ourdomain.com:443 HTTP/1.1\r\nHost:
softwareupdate.eng.vmware.com:443\r\nProxy-Authorization: Basic
*********\r\nUser-Agent: curl/7.19.7-CVS (x86_64-unknown-linux-gnu)
libcurl/7.19.5 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
libssh2/1.2\r\nProxy-Connection: Keep-Alive\r\n\r\n", 289, MSG_NOSIGNAL,
NULL, 0) = 289 # We try to send the credentials along, but...
poll([{fd=3, events=POLLIN|POLLPRI}], 1, 1000) = 1 ([{fd=3,
revents=POLLIN}])
recvfrom(3, "", 16384, 0, NULL, NULL)   = 0 # connection is RESET!
close(3)                                = 0

Let me know. I'd love to close the bug out on my end. :)

-Josh

Attachment (test2005): application/octet-stream, 1322 bytes
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Constantine Sapuntzakis | 1 Oct 04:00 2009
Picon

Patch to not shrink SO_SNDBUF on Windows

The current implementation will always set SO_SNDBUF to
CURL_WRITE_SIZE even if the SO_SNDBUF starts out larger.
The patch doesn't do a setsockopt if SO_SNDBUF is already greater than
CURL_WRITE_SIZE.

This should help folks who have set up their computer with large send buffers.

-Costa
Attachment (dont-shrink-sndbuf.diff): application/octet-stream, 632 bytes
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Guenter | 1 Oct 04:09 2009
Picon

Re: NSS Initialization flags

Hi,
Kamil Dudka schrieb:
> As for the particular option, we should give Nelson some time to explain why 
> it is deprecated. I am also going to look at the nss' code myself first to 
> discover what we are talking about :-)
Nelson has now written a huge reply which covers our questions I think :)

Gün.

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Constantine Sapuntzakis | 1 Oct 04:32 2009
Picon

Re: Problems related to probing for proxy authentication methods

On Wed, Sep 30, 2009 at 4:39 PM, Joshua Kwan <jkwan <at> vmware.com> wrote:
> On 9/29/09 14:54, "Daniel Stenberg" <daniel <at> haxx.se> wrote:
>> libcurl shouldn't require that the connection is persistant, and yes it should
>> react on the 407 and do a new connection for the subsequent request. I thought
>> that already worked... any chance you can write up a test case for the curl
>> test suite that repeats this?
>
> I think the erroneous bit is that CCProxy doesn't reset the connection after
> sending the 407, it only resets it after it gets more data. I am feeling
> like that is very nonstandard behavior:
>

Agreed - CCProxy should send a shutdown if it doesn't support another request.

However, the same situation can happen with any proxy or server that
does support persistent connections. curl sends the next header just
as the proxy times out the persistent connection. So it would be
friendlier if curl deal with this situation rather than propagating it
to the client.

Maybe the right way to deal with it is: if the request is not the
first request on the connection and the server resets the connection,
then retry the request on a fresh connection.

-Costa
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Daniel Stenberg | 1 Oct 08:53 2009
Picon

the HOST ftp command

FYI,

I just fell over this 
http://tools.ietf.org/html/draft-hethmon-mcmurray-ftp-hosts-08 which describes 
a new suggested FTP command called HOST that is to be issued before the USER 
command to let a client select which virtual server to use, like for when you 
want to host multiple ftp servers on the same IP address.

I'm not saying we should proceed and write code for it immediately, but it's 
worth noticing just in case servers and other clients actually start using 
it...

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Daniel Stenberg | 1 Oct 09:05 2009
Picon

Re: Patch to not shrink SO_SNDBUF on Windows

On Wed, 30 Sep 2009, Constantine Sapuntzakis wrote:

> The current implementation will always set SO_SNDBUF to CURL_WRITE_SIZE even 
> if the SO_SNDBUF starts out larger. The patch doesn't do a setsockopt if 
> SO_SNDBUF is already greater than CURL_WRITE_SIZE.

Thanks, applied!

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Daniel Stenberg | 1 Oct 09:09 2009
Picon

Re: Problems related to probing for proxy authentication methods

On Wed, 30 Sep 2009, Constantine Sapuntzakis wrote:

>> I think the erroneous bit is that CCProxy doesn't reset the connection 
>> after sending the 407, it only resets it after it gets more data. I am 
>> feeling like that is very nonstandard behavior:
>
> Agreed - CCProxy should send a shutdown if it doesn't support another 
> request.

Indeed. I think it is a proxy bug not to.

> However, the same situation can happen with any proxy or server that does 
> support persistent connections. curl sends the next header just as the proxy 
> times out the persistent connection. So it would be friendlier if curl deal 
> with this situation rather than propagating it to the client.

I agree here too. And libcurl already _does_ handle this case when it does a 
"normal" HTTP request that isn't a CONNECT, exactly for this reason.

> Maybe the right way to deal with it is: if the request is not the first 
> request on the connection and the server resets the connection, then retry 
> the request on a fresh connection.

Sounds about right. The worst part is that the CONNECT function has turned 
into a rather messy thing that would benefit from a little cleanup...

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Daniel Stenberg | 1 Oct 11:18 2009
Picon

Re: NTLM infinite loop behavior when connection cache is full

On Wed, 30 Sep 2009, Joshua Kwan wrote:

> 1) If all the connections have actually been severed on the socket level, 
> why is the connection cache still full? Should it be empty by the point 
> where all sockets have been closed and traffic has ceased? Or does the 
> connection cache hold symbolic connections (i.e. cookie and authentication 
> state preserving but not socket preserving?)

The connection cache should only keep actually open connections. As soon as 
libcurl detects a connection not being alive anymore it should be closed 
properly and removed from the connection cache. Anything else is a buggy 
behavior.

But of course, the connections might've been alive when they were put into the 
cache and they all died while in there. Oh, but you said they were marked 
'inuse' right? That can't be right as only connection still be actually in use 
should have that set. As soon as they are "returned" to the cache they should 
not be inuse anymore.

> 1.5) Our program uses the multi API with the socket_action workflow and it 
> seems like we could plausibly responsible for keeping cURL state consistent 
> with what sockets are actually open or closed. Could this be the source of 
> the problem?

Hm. Well it could possibly be the case if your app somehow doesn't tell 
libcurl about the closing of the sockets and the terminating of the request, 
so that the socket is atually closed (as you see) but libcurl is still 
thinking the requests/connections are alive. This is just me speaking out my 
mind loud, I'm not exactly sure how it would occur.

> 2) Assuming #1 is working as designed, we need to preserve NTLM state even 
> if the initial NEGOTIATE results in a closed connection after the response 
> comes in. I think the state machine might need another value to represent 
> this so the state can be copied over to the new connection.

No. NTLM is made for and only used for negotiating authentication for the 
_connection_. If the connection is closed there's no NTLM state left necesary 
to save as it needs to be restarted on the next connect.

The problem is rather that we can start a new connection and go with NTLM 
there but we can't keep it in the cache so we can't complete the NTLM 
negotiation. You can of course work around this by inreasing the connection 
cache size, but the actual problem is problem seen by this sympthom. The 
connection cache really shouldn't get full of inuse connections, and if that 
happens when it is about to do NTLM I think a better behavior would be to 
instead fail that transfer entirely and as early as possible as it simply 
won't be able to proceed with that.

> The whole thing wouldn't be a problem if the connection cache weren't always 
> full, or if the NTLM subsystem had better support for a disconnection right 
> after the CHALLENGE comes back. url.c:2639 looks like it contains code that 
> will help to preserve the NTLM state over different connections but perhaps 
> it isn't enough as-is?

Uhm, url.c:2639 of what libcurl version?

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Debasis Ray | 1 Oct 11:55 2009

RE: CURLOPT_INFILESIZE_LARGE and curl_off_t

Thanks a lot, one more question.

We got the version 7.10.6 of libcurl and the below of Linux.

[dray]$ more /etc/redhat-release 
Red Hat Enterprise Linux AS release 3 (Taroon Update 8)
[dray]$ uname -a
Linux lndus186 2.4.21-47.ELsmp #1 SMP Wed Jul 5 20:38:41 EDT 2006 i686
i686 i386 GNU/Linux

Will it be safe to upgrade to the latest version of curl 7.19* with the
above version of Linux we have?

Thanks in advance.
DR
-----Original Message-----
From: curl-library-bounces <at> cool.haxx.se
[mailto:curl-library-bounces <at> cool.haxx.se] On Behalf Of Daniel Stenberg
Sent: 30 September 2009 18:12
To: libcurl development
Subject: Re: CURLOPT_INFILESIZE_LARGE and curl_off_t 

On Wed, 30 Sep 2009, Debasis Ray wrote:

> Can someone please advise the version of curl library when the
variables 
> CURLOPT_INFILESIZE_LARGE and curl_off_t were introduced?

CURLOPT_INFILESIZE_LARGE is a define, curl_off_t is a variable type.

CURLOPT_INFILESIZE_LARGE was introduced in 7.11.0, and curl_off_t came
with 
7.11.1

(docs/libcurl/symbols-in-versions is a good place to read for this
info...)

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

_______________________________________________________________________
The information transmitted is intended only for the person or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission, dissemination or other
use of, or taking of any action in reliance upon, this information by persons or entities other than the
intended recipient is prohibited. If you received this in error, please contact the sender and delete the
material from any computer.

_____________________________________________________________________
This e-mail has been scanned for viruses by Verizon Business Internet Managed Scanning Services - powered
by MessageLabs. For further information visit http://www.mci.com
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html


Gmane