yuva raj | 1 Jun 18:20 2009
Picon

detecting the problem while transferring data

hello,

I connected to a ftp server using curl and tried to download a file from the server.  In the middle of download I disconnected the connection from server.  

curl stops there without exiting for a long time.  Can anyone suggest me any options that are available in curl, by which I can get the exact error code in such case.

I mean, if there is any problem when data transfer is happening, curl must let me know.

Thanks in advance.

yuvaraj.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Young,Jim | 3 Jun 00:03 2009
Picon

Doing a directory listing with sftp and MOVEit DMZ

I'm having trouble getting curl to play nice with a particular sftp server.  I want it to do a long format
directory listing to see if there are any files to download.  If the directory has two or more files, the
listing seems to work fine.  If there's just one file in the directory, all I get is a new line character.  The
server is running the MOVEit DMZ software on it's side of things.  I've tried the same thing with a standard
Linux server and had no problems.  Standard Windows (WinScp) and UNIX sftp clients and produce the long
listing of the lone file fine and the curl "-l" option produces the short listing correctly.

Has anyone else run into this?  Is there a fix for it?

My particulars:
    Curl is version 7.19.4 running on AIX 5.3
Other parts of the curl "package":
info-4.6-1.aix5.1.ppc.rpm
libidn-1.14-1.aix5.1.ppc.rpm
libssh2-1.1-1.aix5.1.ppc.rpm
bash-4.0-1.aix5.1.ppc.rpm
curl-7.19.4-1.aix5.1.ppc.rpm
openssl-0.9.8k-1.aix5.1.ppc.rpm
pkg-config-0.19-6.aix5.2.ppc.rpm
expat-2.0.1-2.aix5.1.ppc.rpm
zlib-1.2.3-5.aix5.1.ppc.rpm
gettext-0.17-1.aix5.1.ppc.rpm

The curl command is:

    /bin/curl -u mylogin:  --trace /tmp/jay --key /z/jay/.ssh/id_rsa --pubkey /z/jay/.ssh/id_rsa.pub \
       --url sftp://sftphost/svcnmh/ \
       --url sftp://sftphost/svcnmh/835/

The first URL returns:
    drwxrwxrwx   0 0        0               0 Apr 30 15:31 835
    drwxrwxrwx   0 0        0               0 Apr 30 15:31 printer
    drwxrwxrwx   0 0        0               0 May 11 16:03 tmp835
    drwxrwxrwx   0 0        0               0 May 11 16:03 tmpprinter

The second URL returns just a new line character despite the fact that there's a directory named "junk" in
it.  If I put a second file or directory in with it, curl will return the proper listing.

The following is the trace file:
== Info: About to connect() to sftphost port 22 (#0)
== Info:   Trying 123.456.789.163... == Info: connected
== Info: Connected to sftphost (123.456.789.163) port 22 (#0)
== Info: SSH authentication methods available: publickey,password
== Info: Using ssh public key file /z/jay/.ssh/id_rsa.pub
== Info: Using ssh private key file /z/jay/.ssh/id_rsa
== Info: SSH public key authentication failed: Invalid signature for supplied public key, or bad
username/public key combination
== Info: Initialized password authentication
== Info: Authentication complete
=> Send data, 59 bytes (0x3b)
0000: 64 72 77 78 72 77 78 72 77 78 20 20 20 30 20 30 drwxrwxrwx   0 0
0010: 20 20 20 20 20 20 20 20 30 20 20 20 20 20 20 20         0       
0020: 20 20 20 20 20 20 20 20 30 20 41 70 72 20 33 30         0 Apr 30
0030: 20 31 35 3a 33 31 20 38 33 35 0a                 15:31 835.
=> Send data, 63 bytes (0x3f)
0000: 64 72 77 78 72 77 78 72 77 78 20 20 20 30 20 30 drwxrwxrwx   0 0
0010: 20 20 20 20 20 20 20 20 30 20 20 20 20 20 20 20         0       
0020: 20 20 20 20 20 20 20 20 30 20 41 70 72 20 33 30         0 Apr 30
0030: 20 31 35 3a 33 31 20 70 72 69 6e 74 65 72 0a     15:31 printer.
=> Send data, 62 bytes (0x3e)
0000: 64 72 77 78 72 77 78 72 77 78 20 20 20 30 20 30 drwxrwxrwx   0 0
0010: 20 20 20 20 20 20 20 20 30 20 20 20 20 20 20 20         0       
0020: 20 20 20 20 20 20 20 20 30 20 4d 61 79 20 31 31         0 May 11
0030: 20 31 36 3a 30 33 20 74 6d 70 38 33 35 0a        16:03 tmp835.
=> Send data, 66 bytes (0x42)
0000: 64 72 77 78 72 77 78 72 77 78 20 20 20 30 20 30 drwxrwxrwx   0 0
0010: 20 20 20 20 20 20 20 20 30 20 20 20 20 20 20 20         0       
0020: 20 20 20 20 20 20 20 20 30 20 4d 61 79 20 31 31         0 May 11
0030: 20 31 36 3a 30 33 20 74 6d 70 70 72 69 6e 74 65  16:03 tmpprinte
0040: 72 0a                                           r.
== Info: Connection #0 to host sftphost left intact
== Info: Re-using existing connection! (#0) with host ftp3.cerner.com
== Info: Connected to sftphost (123.456.789.163) port 22 (#0)
=> Send data, 1 bytes (0x1)
0000: 0a                                              .
== Info: Connection #0 to host sftphost left intact
== Info: Closing connection #0

A request for a short listing:
    /bin/curl -u mylogin:  --trace /tmp/jay --key /z/jay/.ssh/id_rsa --pubkey /z/jay/.ssh/id_rsa.pub \
       -l --url sftp://sftphost/svcnmh/835/

Returns "junk" and the trace file shows this:

== Info: About to connect() to sftphost port 22 (#0)
== Info:   Trying 123.456.789.163... == Info: connected
== Info: Connected to sftphost (123.456.789.163) port 22 (#0)
== Info: SSH authentication methods available: publickey,password
== Info: Using ssh public key file /z/jay/.ssh/id_rsa.pub
== Info: Using ssh private key file /z/jay/.ssh/id_rsa
== Info: SSH public key authentication failed: Invalid signature for supplied public key, or bad
username/public key combination
== Info: Initialized password authentication
== Info: Authentication complete
=> Send data, 4 bytes (0x4)
0000: 6a 75 6e 6b                                     junk
== Info: Connection #0 to host sftphost left intact
== Info: Closing connection #0

Anything else I should include??

Thanks

Jim Young

----------------------------------------------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are
intended only for the addressee. The information contained in this message is confidential and may
constitute inside or non-public information under international, federal, or state securities laws.
Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly
prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and
notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas
City, Missouri, U.S.A at (+1) (816)221-1024.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Daniel Stenberg | 3 Jun 15:33 2009
Picon

Re: detecting the problem while transferring data

On Mon, 1 Jun 2009, yuva raj wrote:

> I connected to a ftp server using curl and tried to download a file from the 
> server.  In the middle of download I disconnected the connection from 
> server.
>
> curl stops there without exiting for a long time.  Can anyone suggest me any 
> options that are available in curl, by which I can get the exact error code 
> in such case.

You need to use one of the time-out options then probably or --keepalive-time.

> I mean, if there is any problem when data transfer is happening, curl must 
> let me know.

A connection doesn't have to transfer data so an idle transfer is not a 
problem by definition.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Daniel Stenberg | 3 Jun 15:41 2009
Picon

Re: Doing a directory listing with sftp and MOVEit DMZ

On Tue, 2 Jun 2009, Young,Jim wrote:

> Anything else I should include??

Since you're seeing this problem only on a specific SFTP server it's a bit 
hard for us to repeat this in our ends to work on the problem. Any chance you 
can get the libssh2 package and build an example source from there and run 
against this server to help us start tracking this down? I think this is a 
libssh2 problem.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Young,Jim | 3 Jun 19:45 2009
Picon

RE: Doing a directory listing with sftp and MOVEit DMZ

Yes - I might be able to do that.

This server is exposed to the internet.  Would it be helpful to have direct access to a test account on that machine?

-----Original Message-----
From: curl-users-bounces <at> cool.haxx.se [mailto:curl-users-bounces <at> cool.haxx.se] On Behalf Of
Daniel Stenberg
Sent: Wednesday, June 03, 2009 8:42 AM
To: the curl tool
Subject: Re: Doing a directory listing with sftp and MOVEit DMZ 

On Tue, 2 Jun 2009, Young,Jim wrote:

> Anything else I should include??

Since you're seeing this problem only on a specific SFTP server it's a bit 
hard for us to repeat this in our ends to work on the problem. Any chance you 
can get the libssh2 package and build an example source from there and run 
against this server to help us start tracking this down? I think this is a 
libssh2 problem.

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

----------------------------------------------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are
intended only for the addressee. The information contained in this message is confidential and may
constitute inside or non-public information under international, federal, or state securities laws.
Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly
prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and
notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas
City, Missouri, U.S.A at (+1) (816)221-1024.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

Daniel Stenberg | 3 Jun 19:53 2009
Picon

RE: Doing a directory listing with sftp and MOVEit DMZ

On Wed, 3 Jun 2009, Young,Jim wrote:

Please don't top-post!

> This server is exposed to the internet.  Would it be helpful to have direct 
> access to a test account on that machine?

It would indeed. You can mail me privately with that and I could try giving it 
some tests when I can fit it into my schedule.

--

-- 

  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

[cristo] | 4 Jun 21:50 2009
Picon

Problem with login using curl in a shell script

Hello,

I'm writing a shell script that logs in a website and downloads its home page. I am already able to login using curl in a terminal via the following commands:

  1. curl --cookie-jar cjar --output /dev/null http://www.gamers.vg
  2. curl -v --cookie cjar --cookie-jar cjar --data 'remember=OF' --data 'tag=nickname' --data 'password=password' --location --output ./temp/temp.html  http://www.gamers.vg/sec/login/
I'm using the first command to set up the cookies used by the site, and the second one to access the web site and downloading the main page in ./temp/temp.html file.

I'm having problems to access the web site when using those commands in the following shell script:

#!/bin/bash

# local variables
############################

tag=nickname
password=password

# initial cookie
############################

echo initial cookie ...

curl -v --cookie-jar cjar --output /dev/null http://www.gamers.vg

echo login ...

curl -v --cookie cjar --cookie-jar cjar \
    --data \'remember=OF\' \
    --data \'tag=$tag\' \
    --data \'password=$password\' \
    --location \
    --output ./temp/temp.html \
    http://www.gamers.vg/sec/login/

echo done!

The exit status of curl says everything is ok, but when I read temp.html I always have a site's error page saying that I am missing fields of the login form.

I analized the http request made by firefox when I do a normal login and found that there are two hidden fields added to the request that have auto-generated values that change with every submit.

I search in the page's source the code that adds those fields but couldn't find anything.

Can somebody help me to find out why I can login when I call curl from a terminal but not when I call it in a shell script?

Thanks in advance
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Jochen Roderburg | 4 Jun 22:19 2009
Picon

Re: Problem with login using curl in a shell script


Zitat von "[cristo]" <ccristoo <at> gmail.com>:

> Hello,
>
> I'm writing a shell script that logs in a website and downloads its home
> page. I am already able to login using curl in a terminal via the following
> commands:
>
>
>    1. curl --cookie-jar cjar --output /dev/null http://www.gamers.vg
>    2. curl -v --cookie cjar --cookie-jar cjar --data 'remember=OF' --data
>    'tag=nickname' --data 'password=password' --location --output
>    ./temp/temp.html  http://www.gamers.vg/sec/login/
>
> I'm using the first command to set up the cookies used by the site, and the
> second one to access the web site and downloading the main page in
> ./temp/temp.html file.
>
> I'm having problems to access the web site when using those commands in the
> following shell script:
>
> #!/bin/bash
>
> # local variables
> ############################
>
> tag=nickname
> password=password
>
> # initial cookie
> ############################
>
> echo initial cookie ...
>
> curl -v --cookie-jar cjar --output /dev/null http://www.gamers.vg
>
> echo login ...
>
> curl -v --cookie cjar --cookie-jar cjar \
>     --data \'remember=OF\' \
>     --data \'tag=$tag\' \
>     --data \'password=$password\' \
>     --location \
>     --output ./temp/temp.html \
>     http://www.gamers.vg/sec/login/
>
> echo done!
>
> The exit status of curl says everything is ok, but when I read temp.html I
> always have a site's error page saying that I am missing fields of the login
> form.
>
> I analized the http request made by firefox when I do a normal login and
> found that there are two hidden fields added to the request that have
> auto-generated values that change with every submit.
>
> I search in the page's source the code that adds those fields but couldn't
> find anything.
>
> Can somebody help me to find out why I can login when I call curl from a
> terminal but not when I call it in a shell script?
>
> Thanks in advance

Because you are obviously using a different curl command line in your  
script.   ;-)

Why are you quoting the quoting apostrophes there a second time?
So they are sent as part of the posted data and that is certainly not correct.

Best regards,

Jochen Roderburg
RRZK
University of Cologne
Robert-Koch-Str. 10                    Tel.:   +49-221/478-7024
D-50931 Koeln                          E-Mail: Roderburg <at> Uni-Koeln.DE
Germany

-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html

[cristo] | 4 Jun 22:23 2009
Picon

Re: Problem with login using curl in a shell script

-_-

I can't believe I couldn't see it!!.

Thanks a lot!!!



On Thu, Jun 4, 2009 at 3:19 PM, Jochen Roderburg <Roderburg <at> uni-koeln.de> wrote:

Zitat von "[cristo]" <ccristoo <at> gmail.com>:

Hello,

I'm writing a shell script that logs in a website and downloads its home
page. I am already able to login using curl in a terminal via the following
commands:


  1. curl --cookie-jar cjar --output /dev/null http://www.gamers.vg
  2. curl -v --cookie cjar --cookie-jar cjar --data 'remember=OF' --data

  'tag=nickname' --data 'password=password' --location --output
  ./temp/temp.html  http://www.gamers.vg/sec/login/

I'm using the first command to set up the cookies used by the site, and the
second one to access the web site and downloading the main page in
./temp/temp.html file.

I'm having problems to access the web site when using those commands in the
following shell script:

#!/bin/bash

# local variables
############################

tag=nickname
password=password

# initial cookie
############################

echo initial cookie ...

curl -v --cookie-jar cjar --output /dev/null http://www.gamers.vg

echo login ...

curl -v --cookie cjar --cookie-jar cjar \
   --data \'remember=OF\' \
   --data \'tag=$tag\' \
   --data \'password=$password\' \
   --location \
   --output ./temp/temp.html \
   http://www.gamers.vg/sec/login/

echo done!

The exit status of curl says everything is ok, but when I read temp.html I
always have a site's error page saying that I am missing fields of the login
form.

I analized the http request made by firefox when I do a normal login and
found that there are two hidden fields added to the request that have
auto-generated values that change with every submit.

I search in the page's source the code that adds those fields but couldn't
find anything.

Can somebody help me to find out why I can login when I call curl from a
terminal but not when I call it in a shell script?

Thanks in advance


Because you are obviously using a different curl command line in your script.   ;-)

Why are you quoting the quoting apostrophes there a second time?
So they are sent as part of the posted data and that is certainly not correct.

Best regards,


Jochen Roderburg
RRZK
University of Cologne
Robert-Koch-Str. 10                    Tel.:   +49-221/478-7024
D-50931 Koeln                          E-Mail: Roderburg <at> Uni-Koeln.DE
Germany



-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Eric Wong | 7 Jun 11:04 2009
Picon

[PATCH] Correctly interpret "-N" to disable buffering

-N means --no-buffer, but the original option name is --buffer.
So we need to negate it one more time (or one less time).
---
 I came across this while working on another issue that is
 highly dependent on output being unbuffered.

 On a side note, would a patch to change the nobuffer code
 path to use plain write(2) be accepted?  It would save some
 data copies to provide a small speedup.

 src/main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/src/main.c b/src/main.c
index 9f3970d..d00ede8 100644
--- a/src/main.c
+++ b/src/main.c
 <at>  <at>  -2635,7 +2635,7  <at>  <at>  static ParameterError getparameter(char *flag, /* f or -long-flag */
       break;
     case 'N':
       /* disable the output I/O buffering */
-      config->nobuffer = (bool)(!toggle);
+      config->nobuffer = (bool)(longopt ? !toggle : toggle);
       break;
     case 'O': /* --remote-name */
       if(subletter == 'a') { /* --remote-name-all */
--

-- 
Eric Wong
-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html


Gmane