Yaron Illouz | 30 Jun 19:37 2015

write different files to disk from an multipart/x-mixed-replace; boundary=END

Hi

 

Hi am trying to get multiple files from a http post reponse.

I know how to write the response as one file but I don't know how to write it as different files to disk.

How do I write the file that arrive in the answer as different files?


For now the output is written to screen as one part (with the --END)

The request is sent from c++ code using curl

    CURL *curl;

    FILE *fp;

    CURLcode res;

 

    curl = curl_easy_init();

    if(curl)

    {

            //fp = fopen(pi_sRequests.c_str(),"wb");

            struct curl_slist *headers=NULL;

 

//headers = curl_slist_append(headers, "Content-Type: multipart/x-mixed-replace;boundary=END");

            curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers );

 

            curl_easy_setopt(curl, CURLOPT_URL, "http://172.16.5.178:8080/module/FileServlet");

            curl_easy_setopt(curl, CURLOPT_POSTFIELDS,pi_sRequests.c_str());

            //curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_to_file);

            //curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);

            res = curl_easy_perform(curl);

 

            /* always cleanup */

            curl_easy_cleanup(curl);

            //fclose(fp);

    }

The response arrive from code writen in java. I took code from Can i attach multiple attachments in one HttpServletResponse

response.setContentType("multipart/x-mixed-replace;boundary=END");

ServletOutputStream out = response.getOutputStream();

out.println("--END");

for(File f:files){

      FileInputStream fis = new FileInputStream(file);

      BufferedInputStream fif = new BufferedInputStream(fis);

      int data = 0;

      out.println("--END");

      while ((data = fif.read()) != -1) {

        out.write(data);

      }

      fif.close();

      out.println("--END");

      out.flush();

}

out.flush();

out.println("--END--");

out.close();

 

 

My question is also on stackoverflow http://stackoverflow.com/questions/31144560/write-multiple-file-from-a-post-reponse-with-curl

 

 

 

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Aaron Couts | 23 Jun 17:59 2015
Picon

command-line tool / SFTP / 644 permissions

I recently ran into an issue while attempting to download a file from my SFTP server using the curl command-line tool.

$ curl sftp://<user>:<password> <at> <server>/<file> -o <file>
curl: (9) Could not open remote file for reading: Permission denied

The remote file has 664 permissions, so there shouldn't be a permissions issue.  My SFTP server logs show that curl is attempting to change the permissions on the file:

Jun 23 15:16:05 mod_sftp/0.9.7[15657]: error changing permissions of '/<file>' to 0100644: Operation not permitted

My SFTP user is not the owner of the file, which is why the chmod command fails.  When I change the permissions from 664 to 644, I can download the file with curl, and my SFTP logs don't show any attempt to change the permissions.

Is there any way to suppress the chmod command that curl is running?  I haven't been able to find anything about this in the documentation, the curl-users archive, or on google.  Hopefully I'm not missing something obvious.

My SFTP server is ProFTPD 1.3.3g with mod_sftp.  I've tested this issue with curl 7.43.0 and 7.19.7.

Thanks,
Aaron



-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Bogdan Harjoc | 22 Jun 12:59 2015
Picon

anyauth for proxy authentication

I can't seem to make curl autodetect that the proxy I specified with --proxy-user requires Digest authentication. It works if I specify --proxy-digest, but if I just say --anyauth it gives up after the first 407 response. 

The command is:

curl --proxy 10.10.0.1:3128 --anyauth --proxy-user user:test --fail http://example.com

With wireshark I see curl sends this in the very first request:

Proxy-Authorization: Basic dXNlcjp0ZXN0

Instead, from the docs it seems it should not send proxy-auth at all, and wait for the 40x response, which contains this:

Proxy-Authenticate: Digest realm="secret", nonce="CemHVQAAAACALrv8qn8AAHRyugkAAAAA", qop="auth", stale=false

Is there a way to resolve this without source code changes ?

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
CJ Ess | 21 Jun 04:01 2015
Picon

Possible bug w/ alternate SSL locations

I think I've found a bug with using alternate SSL locations with curl.

I built a local copy of OpenSSL:

tar -xzf openssl-1.0.2c.tar.gz
cd openssl-1.0.2c
./config --prefix=/home/zxcvbn4038/local zlib
make
make test
make install

Then I tried to use it this way:

cd curl
./buildconf
PKG_CONFIG_LIBDIR=/home/zxcvbn4038/local/lib/pkgconfig/ ./configure --prefix=/home/zxcvbn4038/local

Looking at the configure output it is picking up my alternate openssl package config from ~/local/lib, however to work right it requires SSL_LDFLAGS and SSL_CPPFLAGS to be set which is normally not the case (on Linux at least) and I'm pretty sure they are not, so configure can't find the symbols it is looking for when linking its tests and thinks OpenSSL is not installed. I've tried to confirm however configure is some thick scripting and I can't work out exactly where the tests are being done (I see the code for that is compiled for the test, but not whats invoking gcc or where I might need to add linker flags).

I've also tried to use the alternate library this way:

cd curl
./buildconf
./configure --prefix=/home/zxcvbn4038/local --with-ssl=/home/zxcvbn4038/openssl-1.0.2c

And this time configure finds OpenSSL, enables it, and compiles. However when linking I get these errors:

../lib/.libs/libcurl.so: undefined reference to `SSL_get0_alpn_selected'
../lib/.libs/libcurl.so: undefined reference to `SSL_CTX_set_alpn_protos'

Both of those symbols are present in my alternate OpenSSL library, however they are not present in the system OpenSSL library, so again it appears that we're not setting up the right linker flags in that scenario either.

Both of these seem to be completely reproducible, so I'm hoping that someone more familiar with configure can duplicate my results and put together a fix.


-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Hongyi Zhao | 19 Jun 05:35 2015
Picon

Using curl to list the entries for a ftp/http site recursively.

Hi Developer,

How can I use  curl to list the entries for a ftp/http site recursively?

Regards
--

-- 
Hongyi Zhao <hongyi.zhao <at> gmail.com>
Xinjiang Technical Institute of Physics and Chemistry
Chinese Academy of Sciences
GnuPG DSA: 0xD108493
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Sourin Maiti | 10 Jun 23:55 2015
Picon

Need to send and receive xml data using curl

I use SoapUI to verify my responses below are the sample details. I am planning to automate the processes in linux. How can I achieve it using curl? or wget? I would like to receive the result in a file. 

Raw SoapUI request:

Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
Authorization: Basic R1dTL0FQT1RFU1RJTkdVU0VSOkFwMHRlc3Rpbmc=
Content-Length: 7082
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.galileo.com">
   <soapenv:Header/>
   <soapenv:Body>
      <web:SubmitXml>
X
X
X
X
X
X
X

      </web:SubmitXml>
   </soapenv:Body>
</soapenv:Envelope>



Really Appreciate your response. 

--
with regards,
Sourin Maiti
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
bruce | 10 Jun 03:37 2015
Picon

curl prob

Hi.

Got an issue, probably subtle that I can't seem to wrap my hands around.

A target site can be accessed via the firefox/opera browser. As far as
I can tell, the traffic/cookies/headers are all doable.

I've crafted a test bash/curl but I run into an issue with the process
doing some kind of redirect and the process appears to reset cookies,
and it fails!

The target site www.osu.edu

# the actions for the browser:
#    clear all osu cookies/cache
#    https://www.osu.edu
#    https://courses.osu.edu/psp/csosuct/EMPLOYEE/PUB/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
#    https://courses.osu.edu/psc/csosuct/EMPLOYEE/PUB/c/COMMUNITY_ACCESS.CLASS_SEARCH.GBL
#    -then, select the "AdditionalSearchCriteria" BTN

In trying to mimic this, I crafted the required curls, to have the
required headers/cookie files, etc..

I can post what I created if anyone has bandwidth to take a look.

Like I said, I'm sure it's something subtle, but it's killin' me!

Thanks guys..
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Derek Goring | 10 Jun 00:04 2015
Picon

Using curl to access OpeNDAP

If I enter this:
into a web browser, I get a list of data from NOAA's wave forecast database.

Now, I'd like to automate this using curl but using this:
gives me a globbing error just after the question mark.

What am I doing wrong?
 
________________
Derek Goring
Mulgor Consulting Ltd
24 Brockworth Place
Christchurch, New Zealand
Phone: 64 3 942 5452
Mobile: 64 212 777 639
Website: www.mulgor.co.nz
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Amit Umbarkar | 9 Jun 17:35 2015

FTPS over HTTP proxy unable to setup data connection

I am trying to post a file on FTPS site via HTTP proxy using “implicit” method. Though am able to login & change directories, but hitting upon “* FTP response reading failed” error on each attempt of file STOR command. Here are the detailed logs –

 

/home$ curl --user xxxx:xxxx --tlsv1 --disable-epsv --show-error -Q "CWD inbound" -Q "TYPE I" -Q PWD -Q PASV -Q "STOR abc.txt" --verbose --proxytunnel -x proxy.xyz.com:80 ftps://ftpssl.rbc.com

* About to connect() to proxy.xyz.com port 80

*   Trying 172.19.111.118... * connected

* Connected to proxy.xyz.com (172.19.111.118) port 80

* Establish HTTP proxy tunnel to ftpssl.rbc.com:990

* Proxy auth using (nil) with user ''

* Server auth using Basic with user 'xxxx'

< HTTP/1.0 200 Connection established

* Proxy replied OK to CONNECT request

* successfully set certificate verify locations:

*   CAfile: /usr/share/ssl/certs/ca-bundle.crt

  CApath: none

* SSL connection using RC4-MD5

* Server certificate:

*        subject: /C=CA/ST=Ontario/L=Toronto/O=Royal Bank of Canada/OU=B2Bnetsp20/CN=ftpssl.rbc.com

*        start date: 2012-07-03 00:00:00 GMT

*        expire date: 2015-07-03 23:59:59 GMT

*        subjectAltName: ftpssl.rbc.com matched

*        issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3

* SSL certificate verify ok.

< 220 RBCNG FTPS (Version  Tue Jun  9 10:40:06 2015) server ready.

> USER xxxx

< 331 Password required for xxxx.

> PASS xxxx

< 230 User xxxx logged in.

* We have successfully logged in

> PBSZ 0

< 502 PBSZ Command not implemented.

> PWD

< 257 "/usr/nn/gateway/mbox/fxf/xxxx" is current directory.

* Entry path is '/usr/nn/gateway/mbox/fxf/xxxx'

> CWD inbound

< 250 CWD Command successful.

> TYPE I

< 200 Type set to I

> PWD

< 257 "/usr/nn/gateway/mbox/fxf/xxxx/inbound" is current directory.

> PASV

< 227 Entering Passive Mode (142,245,46,80,24,122)

> STOR abc.txt

< 150 Opening data connection for abc.txt.

> PASV

* FTP response reading failed

* Connection #0 to host proxy.xyz.com left intact

* Closing connection #0

 

When I tried to compare these logs with a windows based CoreFTP Client, they are identical still STOR command invocation. After which CoreFTP tries to establish a socket from proxy for data transfer. Every time it randomly picks up a port for data connection in range of 6200-6299.

 

Please see the snippet below for CoreFTP logs –

 

227 Entering Passive Mode (142,245,8,80,24,104) 

STOR abc.txt

Resolving 142.245.8.80, via proxy (usproxy10.globeop.com)... 

Connect socket #1192 to 172.19.111.118, port 80 (to 6248)...

 

150 Opening data connection for abc.txt. 

226 Transfer complete. 

Abc.txt - 358 bytes transferred 

 

I am assuming this data connection through HTTP proxy is either not mentioned correctly as my initial curl command or it is not implemented in CURL. Can you please advise which is the case? Also, would greatly appreciate if you can help me to resolve this issue.

 

Many thanks for your help!

 

 

Regards,

Amit Umbarkar

This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email in its entirety and any attachments thereto.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 7 Jun 18:58 2015
Picon

the cURL project - video presentation

Heya,

If you haven't been around since forever, you may not know exactly how curl 
came to life, how it runs, who makes it and who pays for everything. Or even 
if you have, you may still appreciate this!

I did a little (50 minutes!) presentation about all this in May that was 
recorded, and now I've made that video available:

http://daniel.haxx.se/blog/2015/06/05/i-lead-the-curl-project-and-this-is-how-it-works/

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Daniel Stenberg | 7 Jun 18:21 2015
Picon

Revamping the --help list?

Hi friends!

There's little doubt that the huge and alpha sorted --help output can be a bit 
intimidating to users and it can easily make the options you really need drown 
among all the others. We are at over 170 options now and the list keeps 
growing.

What to do about it that will be helpful?

spaceone filed this pull-request https://github.com/bagder/curl/pull/252 that 
reorders the list and adds subtitles. The latest incarnation of that 
suggestion can be seen here: https://bpaste.net/show/2ea7b5081e50

I'm however not completely sold on that method as the final solution to the 
problem, so I'm suggesting another take at the problem as detailed in my 
comment to the PR here: 
https://github.com/bagder/curl/pull/252#issuecomment-109327337

What do YOU think?

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Gmane