Problems with a POST passing over 60k of data

I’m running into problems passing a very large request
(60K+).  This code works fine for smaller requests but
blows up before the RPC is made when I pass a very
large request. Any suggestions? 

CURL *HTTPRPC;
Char* urltest="http://myurl.com/myuri/myservlet";
Char* msg="parm1=fctname&parm2=60kofXMLData…"  // 
this is a BIG request...

curl_easy_setopt(HTTPRPC, CURLOPT_URL, urltest);

/* explicitly specify POST */
curl_easy_setopt(HTTPRPC, CURLOPT_POSTFIELDS, msg);

/* explicitly specify POST SIZE*/
curl_easy_setopt(HTTPRPC, CURLOPT_POSTFIELDSIZE,
strlen(msg));
		
// Tell curl about what function to call with the
results...
curl_easy_setopt(HTTPRPC, CURLOPT_WRITEFUNCTION,
Session__write);

// And what to pass to the function for the userp...
curl_easy_setopt(HTTPRPC, CURLOPT_FILE, &results);

// Now just run it!
CURLcode res = curl_easy_perform(HTTPRPC);

Re: Problems with a POST passing over 60k of data

On Thu, 1 May 2003, M D wrote:

[I'm cc'ing this reply to the libcurl mailing list, as it seems a more
appropriate mailing list for this kind of talk.]

> IÂ’m running into problems passing a very large request (60K+).

Using what libcurl version on what platform?

> This code works fine for smaller requests but blows up before the RPC is
> made when I pass a very large request.

What does "blows up" mean in this context? Please be specific.

> Any suggestions?

Sorry, I don't understand what the problem is.

--

-- 
 Daniel Stenberg -- curl, cURL, Curl, CURL. Groks URLs.

-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Found out why the 60K request is failing but unsure how to fix it

Ok.  I found out why it is failing.  Not sure how to
fix it.

In the 60k of XML string there are tag values like
"<movie>The King &amp; I</movie>"

When the the string hits the server it is truncated to
"<movie>The King " and it fails with an invalid XML
error.

The documentation suggests that specifing
CURLOPT_POSTFIELDSIZE will allow for binary data.

  ~~~

I also tried specifying this in the header:

	headers = curl_slist_append(headers, "Content-Type:
text/xml");
	curl_easy_setopt(curl_handle, CURLOPT_HTTPHEADER,
headers);

The server didn't recognize any parms being sent when
I put that in.

   ~~~

Btw, if I replace the "&amp;" with "" it works (but
this is not what we want)

Re: Found out why the 60K request is failing but unsure how to fix it

On Thu, 1 May 2003, M D wrote:

[again, this is libcurl-related and is better discussed on the libcurl list]

> In the 60k of XML string there are tag values like "<movie>The King &amp;
> I</movie>"
>
> When the the string hits the server it is truncated to "<movie>The King "
> and it fails with an invalid XML error.

What do you mean "it is truncated" ? curl certainly does not truncate any
data at any time.

To me, it sounds as if you've tried to pass that XML over a command line and
the &-letter was eaten by your shell.

> The documentation suggests that specifing CURLOPT_POSTFIELDSIZE will allow
> for binary data.

It does, but the only reason you'd need to set that option is if you have
binary zero in the data. curl doesn't truncate data at &-letters, using that
option or not.

> 	headers = curl_slist_append(headers, "Content-Type: text/xml");
> 	curl_easy_setopt(curl_handle, CURLOPT_HTTPHEADER, headers);
>
> The server didn't recognize any parms being sent when I put that in.

Many servers require you to do something like this, yes.

cURL, secure session cookies and following 302 re-directs

Re: cURL, secure session cookies and following 302 re-directs

Re: Found out why the 60K request is failing but unsure how to fix it

You probably need to urlencode the XML data, or something like that.

If you've got this (from your original message):

    Char* msg="parm1=fctname&parm2=60kofXMLData..."

then having "&" in the string is going to be a problem, because &amp
looks a lot like &parm2, and so on.  This means replacing all
non-alphanumeric chars in the '60kofXMLData' with %xx equivalent.

I don't think libcurl does that for you, right Daniel?

Ralph Mitchell

M D wrote:

> Ok.  I found out why it is failing.  Not sure how to
> fix it.
>
> In the 60k of XML string there are tag values like
> "<movie>The King &amp; I</movie>"
>
> When the the string hits the server it is truncated to
> "<movie>The King " and it fails with an invalid XML
> error.
>
> The documentation suggests that specifing
> CURLOPT_POSTFIELDSIZE will allow for binary data.
>
>   ~~~
>
> I also tried specifying this in the header:
>
>         headers = curl_slist_append(headers, "Content-Type:
> text/xml");
>         curl_easy_setopt(curl_handle, CURLOPT_HTTPHEADER,
> headers);
>
> The server didn't recognize any parms being sent when
> I put that in.
>
>    ~~~
>
> Btw, if I replace the "&amp;" with "" it works (but
> this is not what we want)

-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: Found out why the 60K request is failing but unsure how to fix it

On Fri, 2 May 2003, Ralph Mitchell wrote:

> I don't think libcurl does that for you, right Daniel?

Correct. libcurl does no encoding or translating of contents by itself.

--

-- 
 Daniel Stenberg -- curl, cURL, Curl, CURL. Groks URLs.

-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

How to specify a DNS server

Hello,

I'd like curl not to use my OS default DNS server. Can I give curl another
DNS server IP ??

-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf

Re: How to specify a DNS server

On Fri, 2 May 2003, Christophe PIN wrote:

> I'd like curl not to use my OS default DNS server. Can I give curl another
> DNS server IP ??

Nope. curl uses the ordinary system calls to lookup names.

This is a feature I hope we'll be able to add in the future when/if we manage
to add our asynch DNS resolves.

--

-- 
 Daniel Stenberg -- curl, cURL, Curl, CURL. Groks URLs.

-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf