liusifan | 1 Jun 07:51 2008
Picon

Integrate Windows I/O Completion Port into Libevent

Integrate Windows I/O Completion Port into Libevent
 
IOCP is true async i/o model, and libevent is event-driven i/o model.
 
1. How to emulate event-driven recv/send ?
It is possible to use IOCP to emulate event-driven mode. When you make read or write calls, pass in a zero-byte buffer (ie, you can manage your i/o buffers using event driven semantics). So if you issue a read passing in a zero-byte length buffer, your will get notified when there is data to read.
 
2. How to emulate event-driven accept ?
The WSAEventSelect API provides an event-driven accept mechanism.
 
3. How to integrate these two mechanism ?
When we issue a WSARecv or WSASend, we need to pass a OVERLAPPED structure. This structure has a hEvent member.
 
http://msdn.microsoft.com/en-us/library/ms742203(VS.85).aspx
If the lpCompletionRoutine parameter is NULL, the hEvent parameter of lpOverlapped is signaled when the overlapped operation completes if it contains a valid event object handle. An application can use WSAWaitForMultipleEvents or WSAGetOverlappedResult to wait or poll on the event object.
 
We could use one event for all WSARecv/WSASend.
 
http://msdn.microsoft.com/en-us/library/ms686211(VS.85).aspx
Setting an event that is already set has no effect.
 
So it is possible to use WSAWaitForMultipleEvents to integrate IOCP and WSAEventSelect.
 
4. Solution
 
/* objects[0] for iocp operations, object[1..63] for accept */
HANDLE objects[64];
struct event * accepts[64];
 
struct win32iocp_event event1;
event1.overlapped.hEvent = objects[0];
WSARecv( ..., &event1.overlapped, ... );
....
 
struct win32iocp_event event2;
event2.overlapped.hEvent = objects[0];
WSASend( ..., &event2.overlapped, ... );
...
 
WSAEventSelect( ev1->ev_fd, objects[1], FD_ACCEPT );
accepts[1] = ev1;
...
 
WSAEventSelect( ev2->ev_fd, objects[2], FD_ACCEPT );
accepts[2] = ev2;
...
 
int index = WSAWaitForMultipleEvents( 64, objects, FALSE, timeout, FALSE );
index = index - WSA_WAIT_EVENT_0;
 
if( index > 0 ) {
 struct event * event = win32iocp_op->accepts[index];
 event_active (event, EV_READ | EV_ACCEPT, 1);
}
 
if( index == 0 ) {
 for( ; ; ) {
  GetQueuedCompletionPort( ...... );
 }
}
 
5. Limitation
It can only support 63 accept fds.
 
6. Source code
diff file : libevent-1.4.4-iocp\libevent-iocp\diff.txt
add file : libevent-1.4.4-iocp\WIN32-Code\win32iocp.cpp
 
 
 
liusifan
2008-06-01
_______________________________________________
Libevent-users mailing list
Libevent-users <at> monkey.org
http://monkeymail.org/mailman/listinfo/libevent-users
liusifan | 1 Jun 15:03 2008
Picon

Re: Integrate Windows I/O Completion Port into Libevent

Found a bug in the source code.
I have forget to get rid of some debug code.

After fix the problem, the echo_iocp program can accept 1000 online connections propriety.

Test result :
Clients : 1000, Messages Per Client : 1000, Byte Per Message : 52
ExecTimes: 34.229569 seconds

client  Send    Recv
total   : 1000000       1000000
average : 29215/s       29215/s

http://spserver.googlecode.com/files/libevent-1.4.4-iocp.zip

>>>
>Integrate Windows I/O Completion Port into Libevent
>
>IOCP is true async i/o model, and libevent is event-driven i/o model.
>
>1. How to emulate event-driven recv/send ?
>It is possible to use IOCP to emulate event-driven mode. When you make read or write calls, pass in a
zero-byte buffer (ie, you can manage your i/o buffers using event driven semantics). So if you issue a read
passing in a zero-byte length buffer, your will get notified when there is data to read.
>
>2. How to emulate event-driven accept ?
>The WSAEventSelect API provides an event-driven accept mechanism.
>
>3. How to integrate these two mechanism ?
>When we issue a WSARecv or WSASend, we need to pass a OVERLAPPED structure. This structure has a hEvent member.
>
>http://msdn.microsoft.com/en-us/library/ms742203(VS.85).aspx
>If the lpCompletionRoutine parameter is NULL, the hEvent parameter of lpOverlapped is signaled when the
overlapped operation completes if it contains a valid event object handle. An application can use
WSAWaitForMultipleEvents or WSAGetOverlappedResult to wait or poll on the event object.
>
>We could use one event for all WSARecv/WSASend.
>
>http://msdn.microsoft.com/en-us/library/ms686211(VS.85).aspx
>Setting an event that is already set has no effect.
>
>So it is possible to use WSAWaitForMultipleEvents to integrate IOCP and WSAEventSelect.
>
>4. Solution
>
>/* objects[0] for iocp operations, object[1..63] for accept */
>HANDLE objects[64];
>struct event * accepts[64];
>
>struct win32iocp_event event1;
>event1.overlapped.hEvent = objects[0];
>WSARecv( ..., &event1.overlapped, ... );
>....
>
>struct win32iocp_event event2;
>event2.overlapped.hEvent = objects[0];
>WSASend( ..., &event2.overlapped, ... );
>...
>
>WSAEventSelect( ev1->ev_fd, objects[1], FD_ACCEPT );
>accepts[1] = ev1;
>...
>
>WSAEventSelect( ev2->ev_fd, objects[2], FD_ACCEPT );
>accepts[2] = ev2;
>...
>
>int index = WSAWaitForMultipleEvents( 64, objects, FALSE, timeout, FALSE );
>index = index - WSA_WAIT_EVENT_0;
>
>if( index > 0 ) {
> struct event * event = win32iocp_op->accepts[index];
> event_active (event, EV_READ | EV_ACCEPT, 1);
>}
>
>if( index == 0 ) {
> for( ; ; ) {
>  GetQueuedCompletionPort( ...... );
> }
>}
>
>5. Limitation
>It can only support 63 accept fds.
>
>6. Source code
>http://spserver.googlecode.com/files/libevent-1.4.4-iocp.zip
>diff file : libevent-1.4.4-iocp\libevent-iocp\diff.txt
>add file : libevent-1.4.4-iocp\WIN32-Code\win32iocp.cpp
>
>
>
>
>
>
>liusifan
>2008-06-01
>
			
Brodie Thiesfield | 2 Jun 01:57 2008

Status win32 build in libevent

Hi all,

I've been trying to build libevent on win32 but am finding a few problems.

1) The VC6 workspace/projects in the 1.4.4 tar file are "corrupted"
because they don't use CRLF line endings (VC6 and VC2003 refuse to
open them, but easy fixed when you know the problem).
2) Once opened, libevent doesn't build in VC6 because of missing
sys/time.h. This appears to be because although config.h is okay, the
file event-config.h defines a heap of stuff that shouldn't be defined.
3) Win32-Code/misc.c is missing

So the 1.4.4 stable release appears quite broken as released.

Getting the latest from subversion I find:
- VC6 has the same problems as 2 & 3 above.
- opening the VC6 workspace/project in VC 2003 gives missing
event2/util.h and event2/event.h errors
- opening the VC2005 project builds libevent but with a number of
serious stylistic warnings (e.g. void functions returning values.
deleting const pointers. missing function definitions. mismatched
signed and unsigned).

Apologies if my problems are because I am using it incorrectly. I
couldn't find a readme.win32 or similar file to let me know how to
build, or what versions of compilers are required. The VC6
workspaces/projects made me assume that VC6 was still supported. I see
from the list that some people at least are still using it.

Would someone please clarify for me the current state of win32 support
in libevent? Is it still beta and not designed to be used?

Regards,
Brodie
Nick Mathewson | 2 Jun 17:35 2008
Picon

Re: Integrate Windows I/O Completion Port into Libevent

On Sun, Jun 01, 2008 at 01:51:53PM +0800, liusifan wrote:
 [...]
> Integrate Windows I/O Completion Port into Libevent
> 
> IOCP is true async i/o model, and libevent is event-driven i/o model.
> 
> 1. How to emulate event-driven recv/send ?
> It is possible to use IOCP to emulate event-driven mode. When you
> make read or write calls, pass in a zero-byte buffer (ie, you can
> manage your i/o buffers using event driven semantics). So if you issue
> a read passing in a zero-byte length buffer, your will get notified
> when there is data to read.

This seems pretty promising, and could wind up being a better
short-term idea than our older plan to get IOCP support by making it a
new backend for bufferevents.

How well tested is this?  Can you get the unit tests to pass, at
least, to the extent that the unit tests in 1.4.4 currently pass on
win32?  I'd like people who use libevent on windows to try this code
out; if it works well, we should integrate it.

A useful but not-immediately-necessary followup benchmark would be to
see how well this does against using IOCP in the "normal" (not
"event-driven") way.  If there's no big difference, we can defer
bufferevents IOCP work for a while; but if there's still a significant
amount of overhead here, we should try to apply this patch *and* get
IOCP working with bufferevents.

 [...]
> 6. Source code
> http://spserver.googlecode.com/files/libevent-1.4.4-iocp.zip
> diff file : libevent-1.4.4-iocp\libevent-iocp\diff.txt

  (For future reference, please use unified diffs (generated with diff -u)
  rather than old-style diffs.  They're shorter and (usually) easier
  to read.  This diff is small enough that there's not much worry
  about this one, though.)

> add file : libevent-1.4.4-iocp\WIN32-Code\win32iocp.cpp
                                                     ^^^^ You mean ".c"?

Hm.  This might be better applied against trunk (which will eventually
become libevent 2.0) than against 1.4.4; the 1.4 series is supposed to
be stable, and Big New Features are usually out there...

...but now that I look at the code, it looks like the only big change
is to add a new backend file, which (as far as I can tell) doesn't
require any big changes elsewhere in the code.  I'd be curious to see
what Niels thinks here, but I wouldn't be opposed to checking it into
the stable branch.

BTW: I haven't read the code fully yet, but it looks relatively clean
and straightforward.  I'd appreciate it if people who know Windows
networking better than I do would check it out and comment.

Finally: Thanks, Stephen!  This looks like great work to me so far.

peace,
--

-- 
Nick
Rolf Vandevaart | 2 Jun 21:04 2008
Picon

Possible configure changes for epoll

Hello:

We embed libevent in the Open MPI project; long ago, we folded 
libevent's configure m4 code into our own configure code.  Over time, we 
tweaked the libevent m4 code for our own purposes as we ran into 
compatibility issues, etc.  We're submitting these changes to you in the 
hopes that they will be useful.  Feel free to use them or disregard them.

We've changed the form of the original m4 so that it's not directly 
[re-]importable to libevent, but perhaps the code can be copied and 
morphed back into your configure.in.  Here's a link to our current 
revision of the libevent-specific m4 in our configure system:

https://svn.open-mpi.org/trac/ompi/browser/trunk/config/ompi_setup_libevent.m4

Here's a list of the changes that we have made that you might care about 
(there are other changes, too, but I doubt you'll care/want them):

1. Checking for epoll_ctl with AC_CHECK_FUNCS is unfortunately not 
sufficient on some older Linux kernels (e.g., whatever was in Fedora 9) 
because although the function exists and is linkable, it's hardwired to 
return ENOSYS.  So if AC_CHECK_FUNCS with epoll reports that the 
function exists, we also run a short program to ensure that epoll 
actually works.  Doesn't work with cross-compiling, of course.

2. The Sun Studio 12 compilers on Linux currently don't properly support 
the "packed" attribute.  As such, (struct epoll_event) generated by Sun 
Studio 12 on 64 bit architectures will not match the same memory layout 
as in the Linux kernel.  Badness ensues.  In conjunction with #1, our 
test  that checks whether epoll works will also fail if the packed-ness 
of (struct epoll_event) doesn't match between the user application and 
the kernel.  Specifically: the test passes with Sun Studio 12 32-bit 
builds, but fails with Sun Studio 12 64-bit builds (exactly as it should).

We extended the #1 and #2 tests into the syscall test for epoll as well.

--> Including tests #2 and #3 would be most helpful to Sun, because it 
   makes libevent compilable by the Sun Studio 12 compilers on Linux.

4. All versions of OS X kqueue up to and including 10.5.3 are broken 
when used with pty's.  We therefore disable kqueue on OS X because Open 
MPI uses libevent with pty's.  I don't think you want this in the 
general case, but perhaps this a useful datapoint for you.

That's it.  Thanks for all your hard work on libevent.

Rolf

--

-- 

=========================
rolf.vandevaart <at> sun.com
781-442-3043
=========================
William Ahern | 2 Jun 21:30 2008

Re: Integrate Windows I/O Completion Port into Libevent

On Mon, Jun 02, 2008 at 11:35:58AM -0400, Nick Mathewson wrote:
> On Sun, Jun 01, 2008 at 01:51:53PM +0800, liusifan wrote:
>  [...]
> > Integrate Windows I/O Completion Port into Libevent
> > 
> > IOCP is true async i/o model, and libevent is event-driven i/o model.
>

I'm curious, how close are IOCP semantics to POSIX AIO? I'm read through
most of the documentation, but not having actually used IOCP before....

I'm writing a kqueue(2) compat library, and AIO support is next on my list.
Aleksandar Lazic | 2 Jun 21:46 2008
Picon

Re: Integrate Windows I/O Completion Port into Libevent

Hi William,

On Mon 02.06.2008 12:30, William Ahern wrote:
>
>I'm writing a kqueue(2) compat library, and AIO support is next on my
>list.

http://software.schmorp.de/pkg/libeio.html

what do you think about this lib, looks quite nice for me?

Cheers

Aleks
lau stephen | 3 Jun 06:40 2008
Picon

Re: Integrate Windows I/O Completion Port into Libevent


2008/6/2, Nick Mathewson <nickm <at> freehaven.net>:
On Sun, Jun 01, 2008 at 01:51:53PM +0800, liusifan wrote:
[...]

This seems pretty promising, and could wind up being a better
short-term idea than our older plan to get IOCP support by making it a
new backend for bufferevents.

How well tested is this?  Can you get the unit tests to pass, at
least, to the extent that the unit tests in 1.4.4 currently pass on
win32?  I'd like people who use libevent on windows to try this code
out; if it works well, we should integrate it.
 
 
Do you mean the uni tests in libevent/test directory ?
I will try to get these unit tests to pass.
 
(For future reference, please use unified diffs (generated with diff -u)
rather than old-style diffs.  They're shorter and (usually) easier
to read.  This diff is small enough that there's not much worry
about this one, though.)
 
OK, will use -u option.
 

> add file : libevent-1.4.4-iocp\WIN32-Code\win32iocp.cpp
                                                    ^^^^ You mean ".c"?
 
Yes. Just a type mistake. It's win32iocp.c .
 

Hm.  This might be better applied against trunk (which will eventually
become libevent 2.0) than against 1.4.4; the 1.4 series is supposed to
be stable, and Big New Features are usually out there...

...but now that I look at the code, it looks like the only big change
is to add a new backend file, which (as far as I can tell) doesn't
require any big changes elsewhere in the code.  I'd be curious to see
what Niels thinks here, but I wouldn't be opposed to checking it into
the stable branch.
 
These some thing i need to explain here.
 
This patch requires to add a EV_ACCEPT macro to indicate the
accept event. This is incompatible with the old code. The old code use
EV_READ to indicate the accept event.
 
Best Regards,
 
Stephen Liu

_______________________________________________
Libevent-users mailing list
Libevent-users <at> monkey.org
http://monkeymail.org/mailman/listinfo/libevent-users
liusifan | 3 Jun 17:40 2008
Picon

Re: Integrate Windows I/O Completion Port into Libevent


Mostly test cases of regress.c are passed.
These are some changes since the libevent-1.4.4-iocp.zip
Please check the diff file as attachment.

The following test cases are failure or cann't be compiled.
I will pay more time to get these test cases to pass.

test_bufferevent_watermarks();   fail, block on WSAWaitForMultipleEvents
test_multiple();                 fail, block on WSAWaitForMultipleEvents
test_persistent();               fail, block on WSAWaitForMultipleEvents

http_suite();                    fail on http.c::name_from_addr , vc6 has't getnameinfo function
rpc_suite();                     it seem so complex to complile, give up
rpc_test();                      same as above
dns_suite();                     cannot compile evdns.c, because vc6's winsock2.h is too old

Test result of regress.c

evutil_stroll: OK
Testing Priorities 1: OK
Testing Priorities 2: OK
Testing Priorities 3: OK
Testing Evbuffer: OK
Testing evbuffer_find 1: OK
Testing evbuffer_find 2: OK
Testing evbuffer_find 3: OK
Bufferevent: OK
Free active base: OK
Event base new: OK
Simple read: OK
Simple write: OK
Combined read/write: OK
Simple timeout: OK
Loop exit: OK
Loop break: OK
Loop Multiple exit: OK
Multiple events for same fd: OK
Want read only once: OK
Testing Tagging:
                encoded 0x00000af0 with 2 bytes
                encoded 0x00001000 with 3 bytes
                encoded 0x00000001 with 1 bytes
                encoded 0xdeadbeef with 5 bytes
                encoded 0x00000000 with 1 bytes
                encoded 0x00bef000 with 4 bytes
        ??: OK
        ??: OK
                encoded 0x00000af0 with 2 bytes
                encoded 0x00001000 with 2 bytes
                encoded 0x00000001 with 1 bytes
                encoded 0xdeadbeef with 5 bytes
                encoded 0x00000000 with 1 bytes
                encoded 0x00bef000 with 4 bytes
        ??: OK
OK
	
Attachment (diff.txt): application/octet-stream, 12 KiB
_______________________________________________
Libevent-users mailing list
Libevent-users <at> monkey.org
http://monkeymail.org/mailman/listinfo/libevent-users
Shiqing Fan | 3 Jun 19:08 2008
Picon

Re: libevent on windows: select() fails

Hi all,

I'm testing Libevent 1.3 with VS2005 on Windows.

The same problem below there happens on my machine, select() always 
returns -1. Does anyone know the reason?  Sorry for polling this old 
mail up, I just found someone has asked the same question that I got, 
but no answer.  Thanks a lot.

Regards,
Shiqing

> But when I run 'event-test', it seems to die at select():
>
>    V:\dev\libevent-svn\WIN32-Prj\event_test\Debug>event_test.exe
>    [msg] libevent using: win32
>    [debug] event_add: event: 002DFF1C, EV_READ   call 00433CE0
>    [debug] win32_insert: adding event for 916
>    [debug] win32_dispatch: select returned -1
>
> Any idea why it would do that, and what I can do to fix it? I would 
> really like to start using libevent on windows.
>
> Cheers,
> Bas Verhoeven
>
> _______________________________________________
> Libevent-users mailing list
> Libevent-users <at> monkey.org
> http://monkeymail.org/mailman/listinfo/libevent-users
>

--

-- 
--------------------------------------------------------------
Shiqing Fan                     
http://www.hlrs.de/people/fan
High Performance Computing            Tel.: +49 711 685 87234
   Center Stuttgart (HLRS)            Fax.: +49 711 685 65832
POSTAL:Nobelstrasse 19                email: fan <at> hlrs.de     

ACTUAL:Allmandring 30
70569 Stuttgart

Gmane