Transferring large files with SCP fails

Hi,

I'm trying to transfer large files with the SCP protocol, using cURL together
with libssh2.

My curl SCP transfer hangs after 192 kB:

$ curl -T /tmp/1 scp://test:test@.../tmp/1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  192k    0     0    0  192k      0  30399 --:--:--  0:00:06 --:--:--     0

Following up on this thread on the cURL mailing list:
http://news.gmane.org/find-root.php?group=gmane.comp.web.curl.general&article=8052

I followed Daniel's suggestion, and ran the 'scp' and 'scp_nonblock' examples.

The 'scp' example works on a small file, but fails on a big one (6 MB):

$ ./scp 127.0.0.1 root **** /tmp/bigfile >/dev/null
Fingerprint: ...
libssh2_channel_read() failed: 583
all done

The 'scp_nonblock' example hangs after a while:

$ ./scp_nonblock 127.0.0.1 root **** /tmp/bigfile >/dev/null
Fingerprint: ...
libssh2_scp_recv()!
libssh2_scp_recv() is done, now receive data!

R: Transferring large files with SCP fails

Hi,

I experienced a similar problem with libssh2 and I had to recompile it with
the latest libssh2 version present in CVS ( if I remember correctly there
was a bug on channel.c corrected using a workaround, last august). 
Try to re-compile it with the latest version and see if it works for you.

Regards,
Antonio

> -----Messaggio originale-----
> Da: libssh2-devel-bounces@... [mailto:libssh2-devel-
> bounces@...] Per conto di Gavrie Philipson
> Inviato: giovedì 1 novembre 2007 14.17
> A: libssh2-devel@...
> Oggetto: [libssh2] Transferring large files with SCP fails
> 
> Hi,
> 
> I'm trying to transfer large files with the SCP protocol, using cURL
> together
> with libssh2.
> 
> My curl SCP transfer hangs after 192 kB:
> 
> $ curl -T /tmp/1 scp://test:test@.../tmp/1
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time
> Current
>                                  Dload  Upload   Total   Spent    Left
> Speed

[ libssh2-Bugs-1824285 ] problems with remote forwarding

Bugs item #1824285, was opened at 2007-11-01 16:37
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1824285&group_id=125852

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: EAV (evaldes)
Assigned to: Nobody/Anonymous (nobody)
Summary: problems with remote forwarding

Initial Comment:
I have attemped to do remote forwarding two different ways:

//libssh2_poll example

listener = libssh2_channel_forward_listen(session,3306);

retval = 0;

while (retval == 0)
{
	ssh_file_descriptor.type = LIBSSH2_POLLFD_LISTENER;

Re: R: Transferring large files with SCP fails

Antonio <tonyspoken <at> ...> writes:

> 
> Hi,
> 
> I experienced a similar problem with libssh2 and I had to recompile it with
> the latest libssh2 version present in CVS ( if I remember correctly there
> was a bug on channel.c corrected using a workaround, last august). 
> Try to re-compile it with the latest version and see if it works for you.

Hi Antonio,

As I wrote, I already tried this with the snapshot libssh2-0.18.0-20071031,
which was the newest at the time of writing...

Do you have more details about what was fixed?

-- Gavrie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: R: Transferring large files with SCP fails

On Sun, 4 Nov 2007, Gavrie Philipson wrote:

> As I wrote, I already tried this with the snapshot libssh2-0.18.0-20071031, 
> which was the newest at the time of writing...
>
> Do you have more details about what was fixed?

The NEWS file and the Changelog in the tarball contain the details. Or you can 
do:

  cvs diff -rRELEASE_0_17

to get a full diff of all changes done since the 0.17 release. (I think I 
should better make a 0.18 release soon...)

Anyway, since these changes didn't fix the problem you see we should rather 
dive in and start debugging your case I guess. I'm a bit limited on time 
myself why I haven't made much sound about this issue so far. My time limit 
makes me focus on my core projects and unfortunately this makes me less active 
in libssh2.

So, get your hard hat on and dive in!

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[ libssh2-Bugs-1826815 ] Uninitialized variables in transport.c

Bugs item #1826815, was opened at 2007-11-06 04:50
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1826815&group_id=125852

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Uninitialized variables in transport.c

Initial Comment:
I have a question about libssh2, especifically about transport.c code.

I'm running valgrind within my program and sometimes it warns me about using uninitialized variables in
transport.c. I checked it and I think it really is doing that, but I wanted to confirm that with you.

At line 304 of transport.c, there's: "remainbuf = p->writeidx - p->readidx;". But 'p->readidx' and
'p->writeidx' are not initialized. They are initialized after this line.

Can you confirm this? Is there a reason for this? If there is, could you explain it to me, please?

Thank you for your support and attention.

how to Compile libssh2 and exemples under windows ?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
libssh2-devel mailing list
libssh2-devel@...
https://lists.sourceforge.net/lists/listinfo/libssh2-devel

[ libssh2-Bugs-1821872 ] Cannot compile on libssh2 on Solaris 10 Sparc

Bugs item #1821872, was opened at 2007-10-28 17:29
Message generated for change (Comment added) made by nobody
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1821872&group_id=125852

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Cannot compile on libssh2 on Solaris 10 Sparc

Initial Comment:
I have tried downloading and making versions 0.16-0.18.  I receiving the following error trying to compile.

gcc -g -O2 -I/usr/sfw/include -I/usr/local/include -o .libs/ssh2 ssh2.o  -L/usr/sfw/lib
-L/usr/local/lib ../../src/.libs/libssh2.so -lssl -lcrypto -ldl -lz -lnsl -lsocket
-R/usr/local/lib -R/usr/sfw/lib
Undefined                       first referenced
 symbol                             in file
EVP_aes_192_cbc                     ../../src/.libs/libssh2.so
EVP_aes_256_cbc                     ../../src/.libs/libssh2.so
ld: fatal: Symbol referencing errors. No output written to .libs/ssh2
collect2: ld returned 1 exit status
make[3]: *** [ssh2] Error 1
make[3]: Leaving directory `/scratch/source/libssh2-0.18.0-CVS/example/simple'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/scratch/source/libssh2-0.18.0-CVS/example/simple'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/scratch/source/libssh2-0.18.0-CVS/example'
make: *** [all-recursive] Error 1
hal-/scratch/source/libssh2-0.18.0-CVS> env
MANPATH=/opt/SUNWexplo/man:/opt/SUNWvts/man::/usr/man:/usr/share/man:/usr/local/man
LC_MONETARY=en_US.ISO8859-15
TERM=xterm
SHELL=/usr/bin/bash
LC_NUMERIC=en_US.ISO8859-15
OLDPWD=/scratch/source
SSH_TTY=/dev/pts/19
USER=ctouchton
SSH_AUTH_SOCK=/tmp/ssh-POhl9132/agent.9132
PATH=/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/ccs/bin:/usr/platform/bin:/usr/platform/sbin:/usr/sfw/bin:/usr/sfw/sbin:/opt/SUNWspro/bin:/usr/openwin/bin
LC_MESSAGES=C
LC_COLLATE=en_US.ISO8859-15
PWD=/scratch/source/libssh2-0.18.0-CVS
EDITOR=vi
TZ=US/Eastern
SHLVL=1
LC_CTYPE=en_US.ISO8859-15
DISPLAY=localhost:11.0
_=/usr/bin/env
hal-/scratch/source/libssh2-0.18.0-CVS> crle    

Configuration file [version 4]: /var/ld/ld.config  
  Default Library Path (ELF):   /lib:/lib/secure:/usr/lib/secure:/usr/local/lib:/usr/lib:/usr/sfw/lib:/usr/local/ssl/lib/
  Trusted Directories (ELF):    /lib/secure:/usr/lib/secure  (system default)

Command line:
  crle -c /var/ld/ld.config -l /lib:/lib/secure:/usr/lib/secure:/usr/local/lib:/usr/lib:/usr/sfw/lib:/usr/local/ssl/lib/

hal-/scratch/source/libssh2-0.18.0-CVS> 

Thanks.

solarisadmin@...

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2007-11-06 08:23

Message:
Logged In: NO 

Hi

I am facing the same problem

libssh2-0.16/example/simple>make
make  all-am
/bin/bash ../../libtool --tag=CC --mode=link gcc  -g -O2
-I/tmp/pm/openssl-0.9.8f/include -I/usr/include
-L/tmp/pm/openssl-0.9.8f/lib -lcrypto -L/usr/lib -lz -o ssh2  ssh2.o
../../src/libssh2.la -lnsl -lsocket
gcc -g -O2 -I/tmp/pm/openssl-0.9.8f/include -I/usr/include -o .libs/ssh2
ssh2.o  -L/tmp/pm/openssl-0.9.8f/lib -L/usr/lib
../../src/.libs/libssh2.so -lcrypto -lz -lnsl -lsocket
-R/tmp/pm/libssh2-0.16/lib
Undefined                       first referenced
 symbol                             in file
EVP_aes_192_cbc                     ../../src/.libs/libssh2.so
EVP_Cipher                          ../../src/.libs/libssh2.so
EVP_aes_256_cbc                     ../../src/.libs/libssh2.so
ld: fatal: Symbol referencing errors. No output written to .libs/ssh2
collect2: ld returned 1 exit status
*** Error code 1
make: Fatal error: Command failed for target `ssh2'
Current working directory /tmp/pm/libssh2-0.16/example/simple
*** Error code 1
make: Fatal error: Command failed for target `all'

Please do let me know how to resolve this..

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-10-30 11:54

Message:
Logged In: YES 
user_id=236775
Originator: NO

Your guess is probably a good one. This shouldn't be failing in the link
stage otherwise.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2007-10-30 08:18

Message:
Logged In: NO 

Here's my /usr/local/ssl version of openssl.  Is it possible it is looking
at other ssl libraries in /usr/sfw, or somewhere else?

hal-/usr/local/ssl/bin> ./openssl version
OpenSSL 0.9.8f 11 Oct 2007
hal-/usr/local/ssl/bin> 

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2007-10-30 08:17

Message:
Logged In: NO 

Here's my /usr/local/ssl version of openssl.  Is it possible it is looking
at other ssl libraries in /usr/sfw, or somewhere else?

hal-/usr/local/ssl/bin> ./openssl version
OpenSSL 0.9.8f 11 Oct 2007
hal-/usr/local/ssl/bin> 

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-10-28 17:49

Message:
Logged In: YES 
user_id=236775
Originator: NO

This sounds like an old version of OpenSSL. What version are you using?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1821872&group_id=125852

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[ libssh2-Bugs-1826989 ] libssh2_sftp_readdir() hangs forever

Bugs item #1826989, was opened at 2007-11-06 16:50
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1826989&group_id=125852

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SFTP
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: antonio (tonyspoken)
Assigned to: Nobody/Anonymous (nobody)
Summary: libssh2_sftp_readdir() hangs forever

Initial Comment:
An abnormal situation of hanging is noted when doing a remote dir read on a particular directory with the ibssh2_sftp_readdir_ex().
In attach the complete stdout log.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1826989&group_id=125852

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: how to Compile libssh2 and exemples under windows ?

hello,

i suppose also it isn't realy difficult, but il forget probably some think.

i have not file named libssh2.mak in directory Win32 , in subdirectory
or inparent directory
i have only, 
 .
 ..
 .cvsignore
 config.mk
 CVS
 libssh2.dsw
 libssh2_config.h
 libssh2_dll.dsp
 libssh2_lib.dsp
 Makefile
 Makefile.win32
 release
 rules.mk
 test
 tests.dsp

in have Search in libssh2-0.17 libssh2-0.16 et in the daily snapshot.

so if i try the commande line :
--------------------------------------------------------------------
C:\libssh2-0.17\win32> Nmake /f "libssh2.mak" CFG="libssh2 - Win32 Debug"
Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
NMAKE : fatal error U1052: file 'ôlibssh2.makö' not found
Stop.
-----------------------------------------------------------------------
C:\libssh2-0.17\win32> Nmake /f libssh2.mak CFG="libssh2 - Win32 Debug"
Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.
NMAKE : fatal error U1052: file 'libssh2.mak' not found
Stop.
---------------------------------------------------------------------------
C:\libssh2-0.17> Nmake /f NMakefile CFG="libssh2 - Win32 Debug"

Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

NMAKE : fatal error U1073: don't know how to make 'û'
Stop.
---------------------------------------------------------------------------

argument  :CFG="libssh2 - Win32 Debug" give my some error..

------------------------
C:\libssh2-0.17> Nmake /f NMakefile

Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

        for %D in (src) do NMAKE /nologo /f %D/NMakefile BUILD=
SUBDIR=%D all-su
b

C:\libssh2-0.17>NMAKE /nologo /f src/NMakefile BUILD= SUBDIR=src all-sub
        cl -c /nologo /GL /Zi /EHsc /Og /Oi /O2 /Oy /GF /Y- /MD /DNDEBUG
/Iwin32
 /Iinclude /I..\openssl-0.9.8e\inc32 -DLIBSSH2_HAVE_ZLIB=1
/I..\zlib-1.2.3 -DLIB
SSH2_WIN32 /Fo"Release\src\\" src\channel.c src\comp.c src\crypt.c
src\hostkey.c
 src\kex.c src\mac.c src\misc.c src\openssl.c src\packet.c src\pem.c
src\publick
ey.c src\scp.c src\session.c src\sftp.c src\transport.c src\userauth.c
Command line warning D4002 : ignoring unknown option '/GL'
Command line warning D4002 : ignoring unknown option '/Og'
Command line warning D4002 : ignoring unknown option '/Oi'
Command line warning D4002 : ignoring unknown option '/Oy'
Command line warning D4002 : ignoring unknown option '/Y-'
channel.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
comp.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
crypt.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
hostkey.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
kex.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
mac.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
misc.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
openssl.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
packet.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
pem.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
publickey.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
scp.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
session.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
sftp.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
transport.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
userauth.c
src\openssl.h(38) : fatal error C1083: Cannot open include file:
'openssl/openss
lconf.h': No such file or directory
Generating Code...
NMAKE : fatal error U1077: 'cl' : return code '0x2'
Stop.

------------------------------------------------------
so i have copy all files from C:\openssl-0.9.8e\include\openssl into
C:\libssh2-0.17\include\openssl
i have create directory  C:\libssh2-0.17\include\openssl.

so i don't know if it is the good way, but it give.
------------------------------------------------------------------------------------------------------
C:\libssh2-0.17> Nmake /f NMakefile

Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

        for %D in (src) do NMAKE /nologo /f %D/NMakefile BUILD=
SUBDIR=%D all-su
b

C:\libssh2-0.17>NMAKE /nologo /f src/NMakefile BUILD= SUBDIR=src all-sub
        cl -c /nologo /GL /Zi /EHsc /Og /Oi /O2 /Oy /GF /Y- /MD /DNDEBUG
/Iwin32
 /Iinclude /I..\openssl-0.9.8e\inc32 -DLIBSSH2_HAVE_ZLIB=1
/I..\zlib-1.2.3 -DLIB
SSH2_WIN32 /Fo"Release\src\\" src\channel.c src\comp.c src\crypt.c
src\hostkey.c
 src\kex.c src\mac.c src\misc.c src\openssl.c src\packet.c src\pem.c
src\publick
ey.c src\scp.c src\session.c src\sftp.c src\transport.c src\userauth.c
Command line warning D4002 : ignoring unknown option '/GL'
Command line warning D4002 : ignoring unknown option '/Og'
Command line warning D4002 : ignoring unknown option '/Oi'
Command line warning D4002 : ignoring unknown option '/Oy'
Command line warning D4002 : ignoring unknown option '/Y-'
channel.c
comp.c
src\comp.c(40) : fatal error C1083: Cannot open include file: 'zlib.h':
No such
file or directory
crypt.c
hostkey.c
kex.c
mac.c
misc.c
openssl.c
packet.c
pem.c
publickey.c
scp.c
session.c
src\session.c(348) : error C2065: 'socklen_t' : undeclared identifier
src\session.c(348) : error C2146: syntax error : missing ';' before
identifier '
option_len'
src\session.c(348) : error C2065: 'option_len' : undeclared identifier
sftp.c
transport.c
userauth.c
Generating Code...
src\packet.c(1169) : warning C4761: integral size mismatch in argument;
conversi
on supplied
NMAKE : fatal error U1077: 'cl' : return code '0x2'
Stop.

Thank

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/