headers
acpark | 1 Aug 2008 10:46
Picon

XML File Upload - Help !

Hello,

I am new to curl and trying to upload a file to a remote server and have
come across a 4kb limit to the file size.  Having viewed the trace files
showing the file that I send and the error file I get in return, I am
thinking that the file being sent should be chunked and this is not
happening.  Is there a way of setting the chunk/buffer size ?

I am using version 7.18.2 on sparc solaris.  Command line -
/usr/lib/local/bin/curl -v -k -d  <at> export.xml -H "Content-Type: text/xml"
-u username:password "https://URL/Upload.ws"

The relevant portion of the -v trace is as follows -

> Content-Type: text/xml
> Content-Length: 4283
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Server: Orion/2.0.7
< Date Fri, 01 Aug 2008 08:29:19 GMT
< HTTP/1.1 500 Internal Server Error
< Server: Orion/2.0.7
< Connection Close
< Content-Type: text/xml;charset="utf-8"
< Transfer-Encoding: chunked
<

What follows is an xml error file with an error message of "Parse Error a
[row,col]: 1, 4284 Message: Read time out"
(Continue reading)

Permalink | Reply | 501 comments |
headers
Dan Fandrich | 1 Aug 2008 19:48
Favicon

Re: XML File Upload - Help !

On Fri, Aug 01, 2008 at 09:46:00AM +0100, acpark <at> surefish.co.uk wrote:
> I am new to curl and trying to upload a file to a remote server and have
> come across a 4kb limit to the file size.  Having viewed the trace files
> showing the file that I send and the error file I get in return, I am
> thinking that the file being sent should be chunked and this is not
> happening.  Is there a way of setting the chunk/buffer size ?

I don't think that's the problem.

> I am using version 7.18.2 on sparc solaris.  Command line -
> /usr/lib/local/bin/curl -v -k -d  <at> export.xml -H "Content-Type: text/xml"
> -u username:password "https://URL/Upload.ws"
> 
> The relevant portion of the -v trace is as follows -
> 
> > Content-Type: text/xml
> > Content-Length: 4283
> > Expect: 100-continue
> >
> < HTTP/1.1 100 Continue
> < Server: Orion/2.0.7
> < Date Fri, 01 Aug 2008 08:29:19 GMT
> < HTTP/1.1 500 Internal Server Error
> < Server: Orion/2.0.7
> < Connection Close
> < Content-Type: text/xml;charset="utf-8"
> < Transfer-Encoding: chunked

It looks like the server isn't handling the 100-continue properly.  curl
switches to using 100-continue when the post is larger than a few KB, so
(Continue reading)

Permalink | Reply | 501 comments |
headers
What you get is Not what you see | 2 Aug 2008 14:23
Picon

Re: DNS-over-Socks

I didnt understand the difference. But when I use
--socks5 host:port option
it seems not resolve dns queries via socks.

On Thu, Jul 31, 2008 at 7:22 PM, Fabian Keil
<freebsd-listen <at> fabiankeil.de> wrote:
> "What you get is Not what you see" <wyginwys <at> gmail.com> wrote:
>
>> As far as I understood there is no support for dns queries
>> being tunnelled over socks proxy. Correct me If I am wrong.
>> So any patch available? Or a way to overcome this?
>
> Do you really need to tunnel DNS queries or does it
> suffice if the queries are done by the socks proxy?
>
> If it's the latter, have a look at:
>
> |  --socks5-hostname <host[:port]>
> |        Use the specified SOCKS5 proxy (and let the  proxy  resolve  the
> |        host  name).  If the port number is not specified, it is assumed
> |        at port 1080. (Added in 7.18.0)
>
> Fabian
>
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
(Continue reading)

Permalink | Reply | 0 comments |
headers
Dave Yost | 2 Aug 2008 16:08

http://foo.com/bar

If curl is asked to continue from where it left off, and the file was already correctly downloaded, it exits with a result code of 18. This would be OK I guess if the 18 were used only for this situation, but apparently it is used for other situations.

I suggest that curl should exit with status 0 if the file is already downloaded correctly.

I presume that curl doesn't just look for the count of bytes in the file it's resuming but instead actually verifies that those bytes are correct.

Below is a shell transcript. My prompt uses two numbers:
 * result of previous command
 * history number

Dave

0 413 Z% curl --continue-at - -O  http://bits.netbeans.org/dev/nightly/latest/zip/moduleclusters/netbeans-trunk-nightly-200808010201-identity.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  303k  100  303k    0     0   142k      0  0:00:02  0:00:02 --:--:--  153k
0 414 Z% curl --continue-at - -O  http://bits.netbeans.org/dev/nightly/latest/zip/moduleclusters/netbeans-trunk-nightly-200808010201-identity.zip
** Resuming transfer from byte position 311089
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (18) transfer closed with outstanding read data remaining
18 415 Z%
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Permalink | Reply | 501 comments |
headers
Dan Fandrich | 2 Aug 2008 20:38
Favicon

http://foo.com/bar

On Sat, Aug 02, 2008 at 10:08:37AM -0400, Dave Yost wrote:
> If curl is asked to continue from where it left off, and the file was already
> correctly downloaded, it exits with a result code of 18. This would be OK I
> guess if the 18 were used only for this situation, but apparently it is used
> for other situations.
> 
> I suggest that curl should exit with status 0 if the file is already downloaded
> correctly.

It normally does.  Take a look at tests 92 and 1040, which test this exact
situation.  It sounds to me like the server is not supporting the range
request properly.  If you use the -v option, you should see how it's
handling it.

> I presume that curl doesn't just look for the count of bytes in the file it's
> resuming but instead actually verifies that those bytes are correct.

No, it just looks at the size, not the content.

>>> Dan
--

-- 
http://www.MoveAnnouncer.com              The web change of address service
          Let webmasters know that your web site has moved
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Permalink | Reply | 501 comments |
headers
What you get is Not what you see | 2 Aug 2008 21:55
Picon

Re: DNS-over-Socks

Ok I understood the difference.My case was the latter
and I understood that my curl version is older than the current one
and 7.18.0 so i didnt notice the sock5-hostname option.
Now how about the former?

On Thu, Jul 31, 2008 at 7:22 PM, Fabian Keil
<freebsd-listen <at> fabiankeil.de> wrote:
> "What you get is Not what you see" <wyginwys <at> gmail.com> wrote:
>
>> As far as I understood there is no support for dns queries
>> being tunnelled over socks proxy. Correct me If I am wrong.
>> So any patch available? Or a way to overcome this?
>
> Do you really need to tunnel DNS queries or does it
> suffice if the queries are done by the socks proxy?
>
> If it's the latter, have a look at:
>
> |  --socks5-hostname <host[:port]>
> |        Use the specified SOCKS5 proxy (and let the  proxy  resolve  the
> |        host  name).  If the port number is not specified, it is assumed
> |        at port 1080. (Added in 7.18.0)
>
> Fabian
>
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Permalink | Reply | 2 comments |
headers
Daniel Stenberg | 2 Aug 2008 21:59
Picon
Favicon
Gravatar

Re: DNS-over-Socks

On Sat, 2 Aug 2008, What you get is Not what you see wrote:

> Ok I understood the difference.My case was the latter and I understood that 
> my curl version is older than the current one and 7.18.0 so i didnt notice 
> the sock5-hostname option. Now how about the former?

And this is why top-posting is bad. What's "latter" and "former" in this 
context?

Besides, the mail you responded to accurately said that tunneling DNS queries 
isn't supported (I think that's the "former" you refer to) but sending the 
host name to the proxy to let it do the resolve is (the "latter").

So, what's the question again?

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Permalink | Reply | 1 comment |
headers
Daniel Stenberg | 2 Aug 2008 22:46
Picon
Favicon
Gravatar

Re: broken pipe

On Thu, 31 Jul 2008, brubelsabs wrote:

> I've coded a small program checking some URLs

Then I recommend you to use the curl-library mailing list instead of this.

> #1  0xb7ca62f3 in write () from /lib/tls/i686/cmov/libc.so.6
> #2  0xb799cdd1 in ?? () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #3  0xb799ac73 in BIO_write () from /usr/lib/i686/cmov/libcrypto.so.0.9.8
> #4  0xb7a5eabe in ssl3_write_pending () from
> /usr/lib/i686/cmov/libssl.so.0.9.8

This happens within OpenSSL code. What if you use HTTP only? Is your program 
multi-threaded perhaps without the OpenSSL mutex callbacks set?

> Program terminated with signal SIGPIPE, Broken pipe.

> while(CURLM_CALL_MULTI_PERFORM == curl_multi_perform(_handle,
>                                       &handles));
>
> I've read: http://curl.haxx.se/mail/lib-2006-01/0170.html that this isn't an 
> issue with libcurl but with libssl, and I should implement a signal handler 
> ignoring the SIGPIPE, but this was some time ago and maybe there is now a 
> proper solution? I use libcurl together with libares.

Ah right. If you make OpenSSL send data and the server dies/closes the 
connection, libcurl won't (be able to) ignore the SIGPIPE and this will 
happen. I don't think we have a good way to fix this, and I'd actually even go 
so far to suggest that it is strange that OpenSSL offers no way for us to 
avoid them.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Permalink | Reply | 1 comment |
headers
acpark | 4 Aug 2008 10:40
Picon

Re: XML File Upload - Help !

Dan,

Thank you,  the -0 option fix the problem.

Cheers,
Andrew.

> On Fri, Aug 01, 2008 at 09:46:00AM +0100, acpark <at> surefish.co.uk wrote:
>> I am new to curl and trying to upload a file to a remote server and have
>> come across a 4kb limit to the file size.  Having viewed the trace files
>> showing the file that I send and the error file I get in return, I am
>> thinking that the file being sent should be chunked and this is not
>> happening.  Is there a way of setting the chunk/buffer size ?
>
> I don't think that's the problem.
>
>> I am using version 7.18.2 on sparc solaris.  Command line -
>> /usr/lib/local/bin/curl -v -k -d  <at> export.xml -H "Content-Type: text/xml"
>> -u username:password "https://URL/Upload.ws"
>>
>> The relevant portion of the -v trace is as follows -
>>
>> > Content-Type: text/xml
>> > Content-Length: 4283
>> > Expect: 100-continue
>> >
>> < HTTP/1.1 100 Continue
>> < Server: Orion/2.0.7
>> < Date Fri, 01 Aug 2008 08:29:19 GMT
>> < HTTP/1.1 500 Internal Server Error
>> < Server: Orion/2.0.7
>> < Connection Close
>> < Content-Type: text/xml;charset="utf-8"
>> < Transfer-Encoding: chunked
>
> It looks like the server isn't handling the 100-continue properly.  curl
> switches to using 100-continue when the post is larger than a few KB, so
> that's probably the difference you're seeing.  Try disabling this by
> switching to HTTP 1.0 mode with the -0 switch or disabling the Expect:
> header with -H Expect:
>
>>>> Dan
> --
> http://www.MoveAnnouncer.com              The web change of address
> service
>           Let webmasters know that your web site has moved
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
> FAQ:        http://curl.haxx.se/docs/faq.html
> Etiquette:  http://curl.haxx.se/mail/etiquette.html
>
> ______________________________________________
> This email has been scanned by Netintelligence
> http://www.netintelligence.com/email
>
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Permalink | Reply | 0 comments |
headers
What you get is Not what you see | 4 Aug 2008 11:46
Picon

Re: DNS-over-Socks

On Sat, Aug 2, 2008 at 10:59 PM, Daniel Stenberg <daniel <at> haxx.se> wrote:
> On Sat, 2 Aug 2008, What you get is Not what you see wrote:
>
>> Ok I understood the difference.My case was the latter and I understood
>> that my curl version is older than the current one and 7.18.0 so i didnt
>> notice the sock5-hostname option. Now how about the former?
>
> And this is why top-posting is bad. What's "latter" and "former" in this
> context?
>

Sorry I forgot to not top-posting.

> Besides, the mail you responded to accurately said that tunneling DNS
> queries isn't supported (I think that's the "former" you refer to) but
> sending the host name to the proxy to let it do the resolve is (the
> "latter").
>
> So, what's the question again?
>
> --

Yes the former was tunnelling DNS queries and I learnt it isnt supported.
My question was how to make dns queries via socks proxy.
Thanks for your replies.I will use --socks5-hostname option with an upgraded
curl(mine is 7.16.4) to make proxy resolve the hostname.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Permalink | Reply | 0 comments |

Gmane