James Goddard | 1 May 2008 19:13
Picon
Favicon

help with PUT and CURLOPT_READFUNCTION

Hi!

I can successfully PUT a file using CURLOPT_PUT, CURLOPT_INFILE, and CURLOPT_INFILESIZE. However, for large files ( > ~100MB) my machine runs out of memory so I want to be able to "stream" the file to the webserver (instead of storing it in memory and then sending) via CURLOPT_READFUNCTION. Unfortunately, I can't get this to work and cannot find any working examples online. I've given both my working and non-working code below. Interestingly, method 1 below (which does not work) takes about 30 seconds to PUT, while method 2 (works) happens practically instantly.Any help is truly appreciated!

========= DOES NOT WORK =========

if ($method == 'PUT')
{
  $this->fp = fopen($file, 'r');
  $this->fileSize = filesize($file);

  curl_s etopt($curl, CURLOPT_CUSTOMREQUEST, 'PUT');
  curl_setopt($curl, CURLOPT_READFUNCTION, array($this, 'stream_file'));
  curl_setopt($curl, CURLOPT_UPLOAD, true);

  $request_headers[] = 'Content-Length: '.$this->fileSize;  
  $request_headers[] = "Expect: ";  
  $request_headers[] = "Transfer-Encoding: ";

  curl_setopt($this->curl, CURLOPT_HTTPHEADER, $request_headers);

  ...
}

protected function stream_file($curl, $fileData, $fileSize)
{
    return fread($this->fp, $this->fileSize);
}

* About to connect() to 127.0.0.1 port 8000 (#0)
*   Trying 127.0.0.1... * connected
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> PUT /files/foo.mpeg HTTP/1.1
Host: 127.0.0.1:8000
Accept: */*
Accept-Encoding: gzip,deflate
Content-Length: 60153860

  % Total    % Received % Xferd& nbsp; Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 16384    0     0    0 16384      0    453 --:--:--  0:00:36 --:--:--     0* Empty reply from server
100 16384    0     0    0 16384      0    453 --:--:--  0:00:36 --:--:--     0* Connection #0 to host 127.0.0.1 left intact


========= WORKS =========
if ($method == 'PUT')
{
  $fp = fopen($file, 'r'); 
  $fileSize = filesize($file); 
  curl_setopt($curl, CURLOPT_PUT, 1);
  curl_setopt($curl, CURLOPT_INFILE, $fp); 
  curl_setopt($curl, CURLOPT_INFILESIZE, $fileSize);  fclose($fp);
  ...
}

* About to connect() to 127.0.0.1 port 8000 (#0)
*   Trying 127.0.0.1... * connected
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> PUT /files/foo.mpeg HTTP/1.1
Host: 127.0.0.1:8000
Accept: */*
Accept-Encoding: gzip,deflate
Content-Length: 60153860
Expect: 100-continue

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                           &nbs p;     Dload  Upload   Total   Spent    Left  Speed
  0 57.3M    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html
< Content-Length: 18
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
< Keep-Alive: timeout=30, max=100
<
100 57.3M    0    18  100 57.3M      6  21.3M  0:00:02  0:00:02 --:--:-- 33.9M* Connection #0 to host 127.0.0.1 left intact

* Closing connection #0

Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Daniel Stenberg | 1 May 2008 22:40
Picon
Favicon
Gravatar

Re: help with PUT and CURLOPT_READFUNCTION

On Thu, 1 May 2008, James Goddard wrote:

> I can successfully PUT a file using CURLOPT_PUT, CURLOPT_INFILE, and 
> CURLOPT_INFILESIZE. However, for large files ( > ~100MB) my machine runs out 
> of memory so I want to be able to "stream" the file to the webserver 
> (instead of storing it in memory and then sending) via CURLOPT_READFUNCTION.

I'm sorry, but I don't buy this explanation. libcurl itself only supports HTTP 
PUT by streaming the upload, and I'm quite sure the PHP layer doesn't add any 
major magic to that. Thus, the size of the PUT upload has no relevance, it 
always takes roughly the same amount of memory!

> Unfortunately, I can't get this to work and cannot find any working examples 
> online.

I don't know enough PHP/CURL to tell you exactly why.

>  curl_setopt($curl, CURLOPT_CUSTOMREQUEST, 'PUT');

This is not neccesary. CURLOPT_UPLOAD with HTTP implies PUT.

>  $request_headers[] = 'Content-Length: '.$this->fileSize;

I advice you to never ever fiddle with the Content-Length: header yourself. 
You only risk damaging things and libcurl always adds that header itself.

>  $request_headers[] = "Transfer-Encoding: ";

There's really no reason to do this.

--

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

James Goddard | 1 May 2008 23:14
Picon
Favicon

Re: help with PUT and CURLOPT_READFUNCTION

Can anyone who's familiar with the php binding confirm
that the file being PUT is streamed and not first
stored in memory? If so, it will help me pin down why
I can't put files over a certain size without running
out of memory...

Thanks Daniel.

>> I can successfully PUT a file using CURLOPT_PUT,
CURLOPT_INFILE, and
>> CURLOPT_INFILESIZE. However, for large files ( >
~100MB) my machine runs out
>> of memory so I want to be able to "stream" the file
to the webserver
>> (instead of storing it in memory and then sending)
via CURLOPT_READFUNCTION.

> I'm sorry, but I don't buy this explanation. libcurl
itself only supports HTTP
> PUT by streaming the upload, and I'm quite sure the
PHP layer doesn't add any
> major magic to that. Thus, the size of the PUT
upload has no relevance, it
> always takes roughly the same amount of memory!

      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

David Colter | 2 May 2008 00:04
Picon
Favicon

Cookies?

Question about curl handling cookies.

Here's an example of cookies returned from 1 of 20 (or so) pages my  
script will navigate through:

Set-Cookie: SMSESSION=5Rx1y2xkI2p5GYtJ2hBv0G ..... cut ...... i 
+R4eXrmddE; path=/; domain=.xyz.com
Set-Cookie: SMSESSION=5Rx1y2xkI2p5GYtJ2hBv0GWNqQaft ....... cut ......  
i+R4eXrmddE; path=/; domain=.xyz.com
Set-Cookie: GroupMemberships=GroupMem%3A%7B51%7D%7B201%7D%7B202%7D 
%7B363%7D%7B449%7D%7B598%7D%7B614%7D; path=/jetnet
Set-Cookie: EmploymentStatus=ActiveEmployee; path=/jetnet
Set-Cookie: DepartmentCode=007; path=/jetnet
Set-Cookie: JetNetUser=00300611; path=/jetnet
Set-Cookie: PTLogin=; path=/jetnet
Set-Cookie: portalUser=00300611; domain=.aa.com; path=/
Set-Cookie: RetireeStatus=NotRetired; path=/jetnet
Set-Cookie: portalNRTP=5%2F1%2F2008+4%3A39%3A26+PM; domain=.xyz.com;  
path=/
Set-Cookie: Name=DAVID+COLTER; path=/jetnet
Set-Cookie: IntranetFlag=0; path=/jetnet
Set-Cookie: BIGipServerPortal_80_Green=3640937994.20480.0000;  
expires=Thu, 01-May-2008 23:40:13 GMT; path=/

These are returned shortly after login is complete, and further  
Requests to server will not contain all of them. How does curl know  
which to send?

This site's going to be a tough nut to crack. It'd be nice if I could  
trust curl to manage these.  Any comments?

Thanks,
David Colter

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

James Goddard | 2 May 2008 16:42
Picon
Favicon

help with PUT and large files ("select/poll returned error")

When trying to upload a file using CURLOPT_PUT,
CURLOPT_INFILESIZE, and CURLOPT_INFILE
(libcurl/7.16.4) curl sometimes crashes for big files
(those approximately greater than 100MB) and returns
the error "select/poll returned error", and to a much
lesser extent, "Send Failure: Connection reset by
peer".

As stated in this post [
http://curl.haxx.se/mail/archive-2007-06/0112.html ],
the errors are somewhat sporadic so it's been
difficult to pin down the problem.

Here is another relevant post I found [
http://curl.haxx.se/mail/archive-2006-11/0151.html ].

Any suggestions of how to get to the bottom of this problem?

      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

Daniel Stenberg | 2 May 2008 22:18
Picon
Favicon
Gravatar

Re: Cookies?

On Thu, 1 May 2008, David Colter wrote:

[list of cookies]

> These are returned shortly after login is complete, and further Requests to 
> server will not contain all of them. How does curl know which to send?

curl sends all those that match the domain, path etc. Exactly as the cookie 
spec says a client should do...

--

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

Daniel Stenberg | 2 May 2008 22:21
Picon
Favicon
Gravatar

Re: help with PUT and large files ("select/poll returned error")

On Fri, 2 May 2008, James Goddard wrote:

> When trying to upload a file using CURLOPT_PUT, CURLOPT_INFILESIZE, and 
> CURLOPT_INFILE (libcurl/7.16.4) curl sometimes crashes for big files (those 
> approximately greater than 100MB) and returns the error "select/poll 
> returned error", and to a much lesser extent, "Send Failure: Connection 
> reset by peer".

Can you do the same operation with the curl command line tool from the same 
host? Would you be able to try a newer libcurl version?

Are you sending over plain HTTP ? What operating system is used?

--

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

Stephen Pynenburg | 5 May 2008 00:07
Picon

Re: help with PUT and CURLOPT_READFUNCTION

fread(), file_get_contents(), and readfile() all store the data in memory - there's no real way (AFAIK) of just sending right from disk.
However - something that might work is iteratively using stream_get_contents (http://ca3.php.net/manual/en/function.stream-get-contents.php). You could read the file in chunks, then destroy each chunk when you're done reading it.

-Stephen

On Thu, May 1, 2008 at 5:14 PM, James Goddard <james_goddard0511 <at> yahoo.com> wrote:
Can anyone who's familiar with the php binding confirm
that the file being PUT is streamed and not first
stored in memory? If so, it will help me pin down why
I can't put files over a certain size without running
out of memory...

Thanks Daniel.

>> I can successfully PUT a file using CURLOPT_PUT,
CURLOPT_INFILE, and
>> CURLOPT_INFILESIZE. However, for large files ( >
~100MB) my machine runs out
>> of memory so I want to be able to "stream" the file
to the webserver
>> (instead of storing it in memory and then sending)
via CURLOPT_READFUNCTION.

> I'm sorry, but I don't buy this explanation. libcurl
itself only supports HTTP
> PUT by streaming the upload, and I'm quite sure the
PHP layer doesn't add any
> major magic to that. Thus, the size of the PUT
upload has no relevance, it
> always takes roughly the same amount of memory!


     ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Lee French | 7 May 2008 18:33
Picon

Maintining a session

Dear Group,

 

I am in the process of writing a php script that adds items to a shopping cart on another website and then performs a header redirect to the cart page on the other website. However, at the moment my curl session is not maintained on the redirect session (which I expected to be honest). Is it possible for me to maintain the session used during the curl interactivity? This is an example of the code I am using :

 

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://www.cartwebsite.co.uk/index.php?action=showbasket&_prodid=".$_POST['panel_select']."&_qty=".$_POST['panelqty']);

curl_setopt($ch, CURLOPT_HEADER, 0);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt($ch,CURLOPT_FOLLOWLOCATION,0);

curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/mycookie");

curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp/mycookie");

curl_exec($ch);

curl_close($ch);

 

header('location: http://www.cartwebsite.co.uk/index.php?action=showbasket');

 

If I set RETURNTRANSFER to be 0 the session is maintained of course and content echoes however I wish to allow users to continue on the redirect website with their cart items intact. I hope this makes sense and any help you can give me would be great. Many thanks.

 

Kind Regards,

 

Lee French //  iSev Limited  //   website  design - website applications - website hosting

 T. 08700 347 913 // M. 07946 586 598 // E. lee.french <at> isev.co.uk // W. www.isev.co.uk

 

Would you like to see our work? www.isev.co.uk/web-design

 

iSev Limited, e-Innovation Centre SE119, Shifnal Road, Priorslee, Telford, TF2 9FT

 

 

------------------------------------------------------------------------------------------

This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of “iSev Limited”.

 

If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Although we operate anti-virus programmes, we cannot guarantee that this email and any files transmitted with it are virus free and we cannot accept liability for any damage sustained as a result of the software viruses.

 

Please contact the sender if you believe you have received this email in error.  For more information on iSev Limited, visit our website at: http://www.isev.co.uk

 

Registered in England and Wales - Company Number: 6553217.  Registered Offices: iSev Limited, e-Innovation Centre SE119, Shifnal Road, Priorslee, Telford, TF2 9FT

 

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Stephen Pynenburg | 7 May 2008 22:15
Picon

Re: Maintining a session

cURL session and cookies are always separate from the browser's. You'll have to send a Set-Cookie header along with the redirect header, using the cookies that you have saved to /tmp/mycookie .

-Stephen

2008/5/7 Lee French <lee.french <at> isev.co.uk>:

Dear Group,

 

I am in the process of writing a php script that adds items to a shopping cart on another website and then performs a header redirect to the cart page on the other website. However, at the moment my curl session is not maintained on the redirect session (which I expected to be honest). Is it possible for me to maintain the session used during the curl interactivity? This is an example of the code I am using :

 

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://www.cartwebsite.co.uk/index.php?action=showbasket&_prodid=".$_POST['panel_select']."&_qty=".$_POST['panelqty']);

curl_setopt($ch, CURLOPT_HEADER, 0);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt($ch,CURLOPT_FOLLOWLOCATION,0);

curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/mycookie");

curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp/mycookie");

curl_exec($ch);

curl_close($ch);

 

header('location: http://www.cartwebsite.co.uk/index.php?action=showbasket');

 

If I set RETURNTRANSFER to be 0 the session is maintained of course and content echoes however I wish to allow users to continue on the redirect website with their cart items intact. I hope this makes sense and any help you can give me would be great. Many thanks.

 

Kind Regards,

 

Lee French //  iSev Limited  //   website  design - website applications - website hosting

 T. 08700 347 913 // M. 07946 586 598 // E. lee.french <at> isev.co.uk // W. www.isev.co.uk

 

Would you like to see our work? www.isev.co.uk/web-design

 

iSev Limited, e-Innovation Centre SE119, Shifnal Road, Priorslee, Telford, TF2 9FT

 

 

------------------------------------------------------------------------------------------

This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of "iSev Limited".

 

If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Although we operate anti-virus programmes, we cannot guarantee that this email and any files transmitted with it are virus free and we cannot accept liability for any damage sustained as a result of the software viruses.

 

Please contact the sender if you believe you have received this email in error.  For more information on iSev Limited, visit our website at: http://www.isev.co.uk

 

Registered in England and Wales - Company Number: 6553217.  Registered Offices: iSev Limited, e-Innovation Centre SE119, Shifnal Road, Priorslee, Telford, TF2 9FT

 


_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php


_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php

Gmane