神华2008能博会（Energy exposition =A9?=

[ libssh2-Bugs-1923038 ] A way to get hostkey is needed

Bugs item #1923038, was opened at 2008-03-22 14:58
Message generated for change (Comment added) made by alamaison
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1923038&group_id=125852

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: API
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: A way to get hostkey is needed

Initial Comment:
Exceprpt from Debian bug #467305.

Client applications may need to obtain server hostkey "as is", in addition to SHA1/MD5 hash returned by libssh2_hostkey_hash.

----------------------------------------------------------------------

Comment By: alamaison (alamaison)
Date: 2008-07-01 11:39

Message:
Logged In: YES 
user_id=1251900
Originator: NO

We really need this to compare the key with the contents of a known_hosts
file.  
I see that _LIBSSH2_SESSION has a server_hostkey field but it is private
and can't be accessed via the API.  Also there is no information on what
format this is in.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1923038&group_id=125852

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

[ libssh2-Bugs-1923038 ] A way to get hostkey is needed

Bugs item #1923038, was opened at 2008-03-22 15:58
Message generated for change (Comment added) made by bagder
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1923038&group_id=125852

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: API
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: A way to get hostkey is needed

Initial Comment:
Exceprpt from Debian bug #467305.

Client applications may need to obtain server hostkey "as is", in addition to SHA1/MD5 hash returned by libssh2_hostkey_hash.

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2008-07-01 12:44

Message:
Logged In: YES 
user_id=1110
Originator: NO

We'll welcome patches to introduce the necessary APIs.

----------------------------------------------------------------------

Comment By: alamaison (alamaison)
Date: 2008-07-01 12:39

Message:
Logged In: YES 
user_id=1251900
Originator: NO

We really need this to compare the key with the contents of a known_hosts
file.  
I see that _LIBSSH2_SESSION has a server_hostkey field but it is private
and can't be accessed via the API.  Also there is no information on what
format this is in.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=703942&aid=1923038&group_id=125852

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

Re: libssh2_sftp_close_handle: how to handle errors

Dan Fandrich <dan@...> writes:
> On Thu, Apr 24, 2008 at 08:35:34PM -0700, J.T. Conklin wrote:
>> libssh2_sftp_close_handle() currently returns -1 on failure.  What is
>> client code supposed to do to handle this?
>
> Good question.  If the failure is due to an out of memory condition, then
> trying again when more memory is available is probably the right thing.
> But in other cases, like if the socket is already closed, then there's
> nothing you can do.  Problem is, some resources aren't freed in the latter
> case, putting you into a catch-22 situation.  That should really be fixed
> (somehow) in the library.

Sorry all, it has taken a while for this to reach the top of my queue
again.

As I said in my original message, having libssh2_sftp_close_handle()
return errors on failures due to a closed socket, etc. results in an
infinate loop in libssh2_sftp_dtor.

For our product, I changed libssh2_sftp_close_handle() to do a "best
effort" close, which ensures all resources are closed and cleaned up
(to the greatest extent possible). This is similar to other protocol
libraries, eg. OpenLDAP's ldap_unbind() function. We've been using
this change for the last few months without issues (in the same
hostile network environment that used to regularly trigger the
failure).

I checked the current CVS version of sftp.c, and it looks like more
effort has been made to make libssh2_sftp_close_handle() return
PACKET_EAGAIN when an underlying libssh_* operation fails with
PACKET_EAGAIN.  I'm not sure what benefit this is supposed to provide.
It seems to push error handling for nonblocking connections up to the
client, so what was a "libssh2_sftp_close_handle(h);" has to be
changed to "while(libssh2_sftp_close_handle(h) != PACKET_EAGAIN);".

I really think this libssh2_sftp_close_handle() needs to be changed 
to be best effort and blocking, like ldap_unbind().  Otherwise sftp
in practice can not be used reliabily.  After looking at the bugs
filed  <at>  SF, I think this is the same issue as reported as 1940276.

    --jtc

FWIW, this is the patch we're using against a CVS snapshot taken on
2007/05/16.

Index: src/sftp.c
===================================================================
--- src/sftp.c	(revision 11094)
+++ src/sftp.c	(working copy)
 <at>  <at>  -1502,13 +1502,15  <at>  <at> 
         unsigned long data_len, retcode, request_id;
         ssize_t packet_len = handle->handle_len + 13; /* packet_len(4) + packet_type(1) + request_id(4) +
handle_len(4) */
         unsigned char *packet, *s, *data;
-		int rc;
+	int rc;
+	int retval = 0;

         _libssh2_debug(session, LIBSSH2_DBG_SFTP, "Closing handle");
         s = packet = LIBSSH2_ALLOC(session, packet_len);
         if (!packet) {
                 libssh2_error(session, LIBSSH2_ERROR_ALLOC, "Unable to allocate memory for FXP_CLOSE packet", 0);
-                return -1;
+                retval = -1;
+		goto error;
         }

         libssh2_htonu32(s, packet_len - 4);                                     s += 4;
 <at>  <at>  -1521,7 +1523,8  <at>  <at> 
         if (packet_len != libssh2_channel_write(channel, (char *)packet, packet_len)) {
                 libssh2_error(session, LIBSSH2_ERROR_SOCKET_SEND, "Unable to send FXP_CLOSE command", 0);
                 LIBSSH2_FREE(session, packet);
-                return -1;
+		retval = -1;
+		goto error;
         }
         LIBSSH2_FREE(session, packet);

 <at>  <at>  -1531,7 +1534,8  <at>  <at> 
 		}
         if (rc) {
                 libssh2_error(session, LIBSSH2_ERROR_SOCKET_TIMEOUT, "Timeout waiting for status message", 0);
-                return -1;
+                retval = -1;
+		goto error;
         }

         retcode = libssh2_ntohu32(data + 5);
 <at>  <at>  -1540,9 +1544,10  <at>  <at> 
         if (retcode != LIBSSH2_FX_OK) {
                 sftp->last_errno = retcode;
                 libssh2_error(session, LIBSSH2_ERROR_SFTP_PROTOCOL, "SFTP Protocol Error", 0);
-                return -1;
+                retval = -1;
         }

+error:
         if (handle == sftp->handles) {
                 sftp->handles = handle->next;
         }
 <at>  <at>  -1558,7 +1563,7  <at>  <at> 
         LIBSSH2_FREE(session, handle->handle);
         LIBSSH2_FREE(session, handle);

-        return 0;
+        return retval;
 }
 /* }}} */

--

-- 
J.T. Conklin

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

Re: libssh2_sftp_close_handle: how to handle errors

On Tue, 1 Jul 2008, J.T. Conklin wrote:

> I checked the current CVS version of sftp.c, and it looks like more effort 
> has been made to make libssh2_sftp_close_handle() return PACKET_EAGAIN when 
> an underlying libssh_* operation fails with PACKET_EAGAIN.  I'm not sure 
> what benefit this is supposed to provide. It seems to push error handling 
> for nonblocking connections up to the client, so what was a 
> "libssh2_sftp_close_handle(h);" has to be changed to 
> "while(libssh2_sftp_close_handle(h) != PACKET_EAGAIN);".
>
> I really think this libssh2_sftp_close_handle() needs to be changed
> to be best effort and blocking

Why so? We've made an effort to bring non-blocking functionality to libssh2, 
why would an application using libssh2 suddenly not care about that when it 
wants to close a SFTP handle?

I don't consider dealing with "blockiness" to be exactly error-handling.

I would however consider your version to be true if the socket is set to 
blocking mode.

--

-- 

  / daniel.haxx.se

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

Re: libssh2_sftp_close_handle: how to handle errors

Daniel Stenberg <daniel@...> writes:
> On Tue, 1 Jul 2008, J.T. Conklin wrote:
>
>> I checked the current CVS version of sftp.c, and it looks like more
>> effort has been made to make libssh2_sftp_close_handle() return
>> PACKET_EAGAIN when an underlying libssh_* operation fails with
>> PACKET_EAGAIN.  I'm not sure what benefit this is supposed to
>> provide. It seems to push error handling for nonblocking
>> connections up to the client, so what was a
>> "libssh2_sftp_close_handle(h);" has to be changed to
>> "while(libssh2_sftp_close_handle(h) != PACKET_EAGAIN);".
>>
>> I really think this libssh2_sftp_close_handle() needs to be changed
>> to be best effort and blocking
>
> Why so? We've made an effort to bring non-blocking functionality to
> libssh2, why would an application using libssh2 suddenly not care
> about that when it wants to close a SFTP handle?

I suggested blocking because it seems conceptually simpler.  After you
close a handle, there's nothing more that can be done with it, as such
it seemed natural to have close() block until it has completed?  Now I
suspect this opinion was formed due to my lack of direct experience
with non-blocking I/O -- I usually use blocking or asynchronous I/O.  

I did a little investigating, and saw that OpenSSL's SSL_shutdown(),
which is roughly equivalent in a handwavy kind of way, also returns a
"partially done" value when the BIO is in nonblocking mode.  It seems
that libssh2's behavior isn't as wierd as I first thought, and I'll
take that suggestion off the table (and we can all pretend I never
made it in the first place .

But if there is a hard failure, as in the case of the underlying
connection being closed, I think libssh2_sftp_close_handle() has to
complete, as no amount of retries are going to complete the close.

    --jtc

--

-- 
J.T. Conklin

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

Re: libssh2_sftp_close_handle: how to handle errors

On Tue, 1 Jul 2008, J.T. Conklin wrote:

>> Why so? We've made an effort to bring non-blocking functionality to 
>> libssh2, why would an application using libssh2 suddenly not care about 
>> that when it wants to close a SFTP handle?
>
> I suggested blocking because it seems conceptually simpler.  After you close 
> a handle, there's nothing more that can be done with it, as such it seemed 
> natural to have close() block until it has completed?  Now I suspect this 
> opinion was formed due to my lack of direct experience with non-blocking I/O 
> -- I usually use blocking or asynchronous I/O.

Yeah, and I am in the opposite camp!  I'm always doing non-blocking 
operations in a single thread. I imagine an application having 10000 SFTP 
connections in a single thread, and when one of them wants to close down the 
connection I don't want that single connection to block the other 9999 ones...

> But if there is a hard failure, as in the case of the underlying connection 
> being closed, I think libssh2_sftp_close_handle() has to complete, as no 
> amount of retries are going to complete the close.

Yes indeed, returning "try this again" is a blatant bug if there's nothing to 
try again. Like if the connection has been closed in the other end already, 
libssh2 should simply cleanup all its own stuff and return fine.

--

-- 

  / daniel.haxx.se

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

EAGAIN but waiting for what?

Hey

I'm pondering about fixing one of the biggest flaws we have in libcurl's use 
of libssh2 for SCP and SFTP transfers, but in order to do this in a 
non-hackish way I think I need to modify the libssh2 API and I thought I'd ask 
for some general advice here.

We use libssh2 entirely non-blocking as we can have N number of simultanoues 
transfers in the same single thread. When LIBSSH2_EAGAIN is returned we go 
back and wait for action on the socket and then we call the same function 
again when action has happened.

But what action should we wait for?

In the download case, transferring data from a SCP/SFTP server libcurl assumes 
that we need to wait for the socket to become readable. This assumption is 
silly since both SCP and SFTP send/get data both ways during transfers. Thus, 
there are times during download when libssh2 returns LIBSSH2_EAGAIN for its 
internal _send_ and since libcurl still waits for read it'll immediately call 
libssh2 again and we are in busy-loop hell (until the socket actually becomes 
writable). And vice versa of course.

To be able to fix this, we need to be able to query libssh2 for what 
particular action it returned LIBSSH2_EAGAIN for, or rather perhaps what 
condition it wants to see fulfilled before I should bother to call it again.

I suggest we introduce a new function:

int libssh2_session_getfd(LIBSSH2_SESSION *session, int *socket, int *action);

This function will simply store the socket this session is using, and a 
bitmask of actions (READ and/or WRITE) it currently waits for. It returns a 
failure if it doesn't know.

Comments?

--

-- 

  / daniel.haxx.se

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

Re: EAGAIN but waiting for what?

On Thu, 3 Jul 2008, J.T. Conklin wrote:

> FWIW, Just yesterday I learned that OpenSSL operations return either 
> SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE with a non-blocking BIO. This 
> seems better to me than having the library return an error code indicating 
> that the operation must be retried and requiring a second call to find out 
> whether an underlying read or write (would have) blocked.

Yes, I agree. But going that route would mean a much larger impact on the API 
and break the ABI which my suggestion doesn't...

--

-- 

  / daniel.haxx.se

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

How to Compile libssh2 and exemples under windows ?

Dear Sirs, 

I am trying to compile the new libcurl version 7.18.1 and libssh2
version 0.18 in order to user the protocol SFTP under Windows. 
But I meet the problem of lacking the include file, for example while
compiling libssh2 version 0.18, I must de-active many 

Included file, such as: 

/* use O_NONBLOCK for non-blocking sockets */ /*#define HAVE_O_NONBLOCK
1*/

/* Define to 1 if you have the `poll' function. */ /*#define HAVE_POLL
1*/

/* Define to 1 if you have the `select' function. */ /*#define
HAVE_SELECT 1*/

/* use SO_NONBLOCK for non-blocking sockets */
/* #undef HAVE_SO_NONBLOCK */

/* Define to 1 if you have the <stdint.h> header file. */
/* #undef HAVE_STDINT_H */

And then I must also deactive these include file: 

/* Define to 1 if you have the <sys/ioctl.h> header file. */ /*#define
HAVE_SYS_IOCTL_H 1*/

/* Define to 1 if you have the <sys/select.h> header file. */ #define
HAVE_SYS_SELECT_H 1

/* Define to 1 if you have the <sys/socket.h> header file. */ /*#define
HAVE_SYS_SOCKET_H 1*/

/* Define to 1 if you have the <sys/stat.h> header file. */ #define
HAVE_SYS_STAT_H 1
/* Define to 1 if you have the <sys/time.h> header file. */

/*#define HAVE_SYS_TIME_H 1*/
/* Define to 1 if you have the <sys/types.h> header file. */ #define
HAVE_SYS_TYPES_H 1

/* Define to 1 if you have the <sys/uio.h> header file. */ /*#define
HAVE_SYS_UIO_H 1*/

/* Define to 1 if you have the <unistd.h> header file. */ /*#define
HAVE_UNISTD_H 1*/

And now I am blocked here by lacking of 'socklen_t' 

 
..\src\session.c(349) : error C2065: 'socklen_t' : undeclared identifier
..\src\session.c(349) : error C2146: syntax error : missing ';' before
identifier 'option_len'
..\src\session.c(349) : error C2065: 'option_len' : undeclared
identifier

NMAKE : fatal error U1077: 'cl.exe' : return code '0x2'

Someone can help me, which procedure I have followed to compile
successful under Windows, I use VC 6. I have exported libssh2_lib.mak
from libssh2_lib.dsw

Thank for your helps 

Best regards

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08