Libevent

Libevent This is an amazing opportunity to the key to success http://www.newsi10web.com
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Libevent

Libevent This is truly an amazing opportunity to success http://www.inewsl0.com
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Libevent

Libevent I was able to put all my kids through college because of this http://wwww.todayl0newsi.com
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Re: Libevent

Re: Avoid potential SSL read spinlocks

Hi,

Why does bev_ssl require deferred callbacks?

Commit fc52dbac (see below) fixed an issue I was seeing where deferred
error_cb(EOF)s were confusing clients which had disabled EV_READs
(notably evhttp). If bev_ssl didn't require deferred cbs, the EOFs
would have stayed queued at the OpenSSL layer (in the same way that
EOFs normally stay queued in the kernel when bev_sock clients disable
EV_READ).

Thanks,
Catalin

diff --git a/bufferevent_openssl.c b/bufferevent_openssl.c
index 86a8619..3843b31 100644
--- a/bufferevent_openssl.c
+++ b/bufferevent_openssl.c
<snip>
-       while ((bev_ssl->bev.bev.enabled & EV_READ) &&
+       if ((bev_ssl->bev.bev.enabled & EV_READ) &&
<snip>
                r = do_read(bev_ssl, n_to_read);
-               if (r <= 0)
-                       break;
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Re: Re: Avoid potential SSL read spinlocks

On Mon, Nov 14, 2011 at 11:03:58AM -0800, Catalin Patulea wrote:
> Hi,
> 
> Why does bev_ssl require deferred callbacks?
> 

As far as I know, it doesn't. But I wrote this patch because if you are
using a deferred bev with bulk incoming data, it never drops back into
base_loop() until SSL_read() says there is nothing to read. This is due
to the fact that your callback for new data is never called until
libevent gets around to processing the deferred queue.

Nick and I discussed why the while loop was needed, and concluded it was
not.

Please read the full explaination here:
https://github.com/nmathewson/Libevent/pull/33

> Commit fc52dbac (see below) fixed an issue I was seeing where deferred
> error_cb(EOF)s were confusing clients which had disabled EV_READs
> (notably evhttp). If bev_ssl didn't require deferred cbs, the EOFs
> would have stayed queued at the OpenSSL layer (in the same way that
> EOFs normally stay queued in the kernel when bev_sock clients disable
> EV_READ).

This does not change any mechanics to openssl bufferevents without
deferred enabled. The logic is this:

SSL data was recv'd
_bufferevent_run_readcb() is called

That function does this:
228         if (p->options & BEV_OPT_DEFER_CALLBACKS) {
229                 p->readcb_pending = 1;
230                 if (!p->deferred.queued)
231                         SCHEDULE_DEFERRED(p);
232         } else {
233                 bufev->readcb(bufev, bufev->cbarg);
234         }

As you can see, if it is not deferred, it's not scheduled, your callback
is immediately called.

You can read why you would use a deferred here:
http://www.wangafu.net/~nickm/libevent-book/Ref6_bufferevent.html

I hope this fixed your problem?
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Re: Re: Avoid potential SSL read spinlocks

On Mon, Nov 14, 2011 at 11:17 AM, Mark Ellzey <mthomas <at> strcpy.net> wrote:
> As far as I know, it doesn't. But I wrote this patch because if you are
> using a deferred bev with bulk incoming data, it never drops back into
> base_loop() until SSL_read() says there is nothing to read. This is due
> to the fact that your callback for new data is never called until
> libevent gets around to processing the deferred queue.
This is actually causing an assert in my case :( The key is to get
SSL_read to return both data (> 0) and EOF (= 0) within the same
iteration of that (former) while loop. Then the flow looks like this:

SSL_read() returns N bytes into an iov from evbuffer_reserve_space -
no run_read_cb yet
SSL_read() returns 0, which leads into conn_closed(), which queues a
error_cb(EOF)
Back in do_read, a read_cb is queued

In the event loop, the read_cb is delivered first, completing the HTTP
request. evhttp disables EV_READs and moves into state
EVCON_DISCONNECTED. ...and then the error_cb(EOF) is delivered,
confusing evhttp and causing assert(req != NULL) in
evhttp_connection_fail.

I've found it hard to point the finger at any particular piece of
code... It depends on whether the bufferevent interface defines an
ordering between EV_READ enable/disables and EOF/ERROR events.

In the case of bev_sock, once disable(EV_READ) has been called, the fd
is not checked for readability anymore, so read() doesn't have a
chance to report any EOF or errors. Effectively, we're leaving these
events queued in the kernel (which seems like a reasonable thing to do
- evhttp no longer cares about that fd).

Your patch indirectly does this - EOF/errors are now queued in OpenSSL
because SSL_read is only called once for every read_cb.

My only concern is that there may be cases where OpenSSL _needs_ us to
call SSL_read repeatedly to make any progress, despite the underlying
fd staying unreadable. Maybe during renegociation or application
payloads that span SSL records? The man page doesn't have anything to
that effect, but that doesn't mean the cases don't exist..

> This does not change any mechanics to openssl bufferevents without
> deferred enabled.
I agree that your patch doesn't affect non-deferred mechanics - but I
haven't been able to get evhttp to stack on top of bev_ssl without
deferred callbacks. It seems to send the request line, but gets stuck
in write_cb's before sending request headers. I haven't investigated
too much because it seems like it would only be a quick fix - sooner
or later someone may want to use deferred cbs with bev_ssl, for the
reasons described in the link you sent.

BTW, sample/le-proxy.c also enabled deferred cbs on its bev_ssl.
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Re: Re: Avoid potential SSL read spinlocks

On Mon, Nov 14, 2011 at 02:30:32PM -0800, Catalin Patulea wrote:
> This is actually causing an assert in my case :( The key is to get
> SSL_read to return both data (> 0) and EOF (= 0) within the same
> iteration of that (former) while loop. Then the flow looks like this:
> 
> SSL_read() returns N bytes into an iov from evbuffer_reserve_space -
> no run_read_cb yet
> SSL_read() returns 0, which leads into conn_closed(), which queues a
> error_cb(EOF)
> Back in do_read, a read_cb is queued

Not entirely true:

                        int err = SSL_get_error(bev_ssl->ssl, r);
                        print_err(err);
                        switch (err) {
                        case SSL_ERROR_WANT_READ:
                                /* Can't read until underlying has more data. */
                                if (bev_ssl->read_blocked_on_write)
                                        if (clear_rbow(bev_ssl) < 0) 
                                                return -1;
                                break;
                        case SSL_ERROR_WANT_WRITE:                                                                                                                                                    
                                /* This read operation requires a write, and the
                                 * underlying is full */
                                if (!bev_ssl->read_blocked_on_write)
                                        if (set_rbow(bev_ssl) < 0) 
                                                return -1;
                                break;
                        default:
                                conn_closed(bev_ssl, err, r);
                                break;

WANT_READ will trigger when a full record hasn't been processed; My
brain isn't working right now, but it looks as if set_rbow() does the
right thing and waits for the whole record to be read. BUT, I think I
see the issue you trying to explain. 

In one sentence - I think this is the problem:

We cannot schedule or call the readcb UNTIL SSL_read() returns > 0 OR an
error occurs on the socket.

Right?
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Re: Re: Avoid potential SSL read spinlocks

On Mon, Nov 14, 2011 at 07:13:39PM -0600, Mark Ellzey wrote:
> 
> We cannot schedule or call the readcb UNTIL SSL_read() returns > 0 OR an
> error occurs on the socket.

I definitely feel confident this is the issue, going into that loop
provided ample time for SSL_read() to process an entire record.  
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Re: Re: Avoid potential SSL read spinlocks

On Mon, Nov 14, 2011 at 07:21:00PM -0600, Mark Ellzey wrote:
> On Mon, Nov 14, 2011 at 07:13:39PM -0600, Mark Ellzey wrote:
> > 
> > We cannot schedule or call the readcb UNTIL SSL_read() returns > 0 OR an
> > error occurs on the socket.
> 
> I definitely feel confident this is the issue, going into that loop
> provided ample time for SSL_read() to process an entire record.  

I retract this. After looking at it more, it does the right thing. And
bevssl does work without deferred. I use it all the time. Is your
application threaded?
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.