Nate Rosenblum | 27 Jan 22:29 2015

curl_mutli and edge-triggered events

Profiling an application that uses the `curl_multi_socket_action` interface, I noticed that the writeable-socket callback for my event framework (libevent in this case) was being invoked many times when issuing HTTP GET requests. I quickly determined that this stems from configuring libevent for level-triggered & persistent socket events; libcurl registers for INOUT mode while sending the request and libevent will repeatedly raise writeable socket events until write registration is removed with another socket callback with `CURL_POLL_IN` (when waiting for the response). 

One wait to avoid these callbacks would be to set my events to be edge-triggered. I haven't read the multi implementation in sufficient detail to know if this is safe, however. If the underlying socket write returns EAGAIN while writing the the HTTP GET request, will libcurl invoke the socket callback to indicate its interest in writeable-socket notifications? Cursory reading suggests that it will not; it looks to me like determination of when to invoke the callback is based on the phase of the request in `singlesocket()`, and that the `Curl_sh_entry.action` field is not updated outside of that method.

Perhaps somebody with more familiarity can clarify or point out where I've gone wrong? For many concurrent transfers I am burning considerable CPU time servicing useless writeable-socket events and would love to avoid them.

Best,

--nate
Steffen Christgau | 24 Jan 17:33 2015
Picon

requests over HTTPS after server-side close

Hi everybody,

based on code of the https-client sample I use libevent for doing
HTTP(S) requests in one of my applications. It receives data from
another process (via socket using libevent as well), transforms the
data, and issues HTTP requests based on the transformation result.
Between multiple data receptions from the external process there's some
random delay. In case the delay is long enough, the HTTP connection
established by libevent is closed by the server. When using HTTP
libevent automatically reconnects to the server as soon a new request
has been issued. So, everything is fine.

However, when HTTPS is used, the reconnect seems not work and no request
is performed after the server-side disconnect. I was able to reproduce
this behavior with a modified (although somewhat ugly) version of the
https-client example. You can find the source code in the attachment
(don't know if attachments are welcome in this mailing list). The
timeout in http_request_done simulates reception of data from the
external application which causes a new HTTP request to be triggered.
The timeout value should be chosen to let the callback be invoked after
the server closed the connection. You can try it with example.com and a
timeout value of 55 sec. For HTTP you will get a infinite loop of
requests. For HTTPS only one request is performed, the connection
closes, the timer callback is invoked and the application terminates.

I wonder which steps are required to get the code working with HTTPS as
it does with HTTP. From an application programmers view, I would expect
that there should be no difference, except for the bev creation (as in
the example). Do you have any suggestions? I'm using libevent 2.1.5-beta
(although I observed this in 2.1.3 and 2.1.4 as well) with OpenSSL
1.0.1k on Linux (gcc 4.8.3, glibc 2.19).

Kind regards,

Steffen
/*
  This is an example of how to hook up evhttp with bufferevent_ssl

  It just GETs an https URL given on the command-line and prints the response
  body to stdout.

  Actually, it also accepts plain http URLs to make it easy to compare http vs
  https code paths.

  Loosely based on le-proxy.c.
 */

// Get rid of OSX 10.7 and greater deprecation warnings.
#if defined(__APPLE__) && defined(__clang__)
#pragma clang diagnostic ignored "-Wdeprecated-declarations"
#endif

#include <stdio.h>
#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

#ifdef _WIN32
#include <winsock2.h>
#include <ws2tcpip.h>

#define snprintf _snprintf
#define strcasecmp _stricmp
#else
#include <sys/socket.h>
#include <netinet/in.h>
#endif

#include <event2/bufferevent_ssl.h>
#include <event2/bufferevent.h>
#include <event2/buffer.h>
#include <event2/listener.h>
#include <event2/util.h>
#include <event2/http.h>

#include <openssl/ssl.h>
#include <openssl/err.h>
#include <openssl/rand.h>

#include "openssl_hostname_validation.h"

static struct event_base *base;
static int ignore_cert = 0;
struct bufferevent *bev;
static const char* host;
static char* data_file;
static struct evhttp_connection *evcon;
static char uri[256];

static void perform_http_request(void);

static void close_cb(struct evhttp_connection *con, void *arg)
{
	printf("--- connection closed ---\n");
}

static void timer_cb(evutil_socket_t fd, short what, void *arg)
{
	printf("timer expired, issueing next request\n");
	perform_http_request();
}

static void
http_request_done(struct evhttp_request *req, void *ctx)
{
	char buffer[256];
	int nread;

	if (req == NULL) {
		/* If req is NULL, it means an error occurred, but
		 * sadly we are mostly left guessing what the error
		 * might have been.  We'll do our best... */
		struct bufferevent *bev = (struct bufferevent *) ctx;
		unsigned long oslerr;
		int printed_err = 0;
		int errcode = EVUTIL_SOCKET_ERROR();
		fprintf(stderr, "some request failed - no idea which one though!\n");
		/* Print out the OpenSSL error queue that libevent
		 * squirreled away for us, if any. */
		while ((oslerr = bufferevent_get_openssl_error(bev))) {
			ERR_error_string_n(oslerr, buffer, sizeof(buffer));
			fprintf(stderr, "%s\n", buffer);
			printed_err = 1;
		}
		/* If the OpenSSL error queue was empty, maybe it was a
		 * socket error; let's try printing that. */
		if (! printed_err)
			fprintf(stderr, "socket error = %s (%d)\n",
				evutil_socket_error_to_string(errcode),
				errcode);
		return;
	}

	fprintf(stderr, "Response line: %d %s\n",
	    evhttp_request_get_response_code(req),
	    evhttp_request_get_response_code_line(req));

	while ((nread = evbuffer_remove(evhttp_request_get_input_buffer(req),
		    buffer, sizeof(buffer)))
	       > 0) {
		/* These are just arbitrary chunks of 256 bytes.
		 * They are not lines, so we can't treat them as such. */
		fwrite(buffer, nread, 1, stdout);
	}

	{
		struct timeval tv = { 55, 0 };
		evtimer_add(evtimer_new(base, timer_cb, NULL), &tv);
	}

}

static void
syntax(void)
{
	fputs("Syntax:\n", stderr);
	fputs("   https-client -url <https-url> [-data data-file.bin] [-ignore-cert] [-retries num]\n", stderr);
	fputs("Example:\n", stderr);
	fputs("   https-client -url https://ip.appspot.com/\n", stderr);

	exit(1);
}

static void
die(const char *msg)
{
	fputs(msg, stderr);
	exit(1);
}

static void
die_openssl(const char *func)
{
	fprintf (stderr, "%s failed:\n", func);

	/* This is the OpenSSL function that prints the contents of the
	 * error stack to the specified file handle. */
	ERR_print_errors_fp (stderr);

	exit(1);
}

static void perform_http_request(void)
{
	int r;

	struct evhttp_request *req;
	struct evkeyvalq *output_headers;
	struct evbuffer * output_buffer;

	// Fire off the request
	req = evhttp_request_new(http_request_done, bev);
	if (req == NULL) {
		fprintf(stderr, "evhttp_request_new() failed\n");
		return;
	}

	output_headers = evhttp_request_get_output_headers(req);
	evhttp_add_header(output_headers, "Host", host);
//	evhttp_add_header(output_headers, "Connection", "close");

	if (data_file) {
		/* NOTE: In production code, you'd probably want to use
		 * evbuffer_add_file() or evbuffer_add_file_segment(), to
		 * avoid needless copying. */
		FILE * f = fopen(data_file, "rb");
		char buf[1024];
		size_t s;
		size_t bytes = 0;

		if (!f) {
			syntax();
		}

		output_buffer = evhttp_request_get_output_buffer(req);
		while ((s = fread(buf, 1, sizeof(buf), f)) > 0) {
			evbuffer_add(output_buffer, buf, s);
			bytes += s;
		}
		evutil_snprintf(buf, sizeof(buf)-1, "%lu", (unsigned long)bytes);
		evhttp_add_header(output_headers, "Content-Length", buf);
		fclose(f);
	}

	r = evhttp_make_request(evcon, req, data_file ? EVHTTP_REQ_POST : EVHTTP_REQ_GET, uri);
	if (r != 0) {
		fprintf(stderr, "evhttp_make_request() failed\n");
		return;
	}
}

/* See http://archives.seul.org/libevent/users/Jan-2013/msg00039.html */
static int cert_verify_callback(X509_STORE_CTX *x509_ctx, void *arg)
{
	char cert_str[256];
	const char *host = (const char *) arg;
	const char *res_str = "X509_verify_cert failed";
	HostnameValidationResult res = Error;

	/* This is the function that OpenSSL would call if we hadn't called
	 * SSL_CTX_set_cert_verify_callback().  Therefore, we are "wrapping"
	 * the default functionality, rather than replacing it. */
	int ok_so_far = 0;

	X509 *server_cert = NULL;

	if (ignore_cert) {
		return 1;
	}

	ok_so_far = X509_verify_cert(x509_ctx);

	server_cert = X509_STORE_CTX_get_current_cert(x509_ctx);

	if (ok_so_far) {
		res = validate_hostname(host, server_cert);

		switch (res) {
		case MatchFound:
			res_str = "MatchFound";
			break;
		case MatchNotFound:
			res_str = "MatchNotFound";
			break;
		case NoSANPresent:
			res_str = "NoSANPresent";
			break;
		case MalformedCertificate:
			res_str = "MalformedCertificate";
			break;
		case Error:
			res_str = "Error";
			break;
		default:
			res_str = "WTF!";
			break;
		}
	}

	X509_NAME_oneline(X509_get_subject_name (server_cert),
			  cert_str, sizeof (cert_str));

	if (res == MatchFound) {
		printf("https server '%s' has this certificate, "
		       "which looks good to me:\n%s\n",
		       host, cert_str);
		return 1;
	} else {
		printf("Got '%s' for hostname '%s' and certificate:\n%s\n",
		       res_str, host, cert_str);
		return 0;
	}
}

int
main(int argc, char **argv)
{
	int r;

	struct evhttp_uri *http_uri;
	const char *url = NULL;
	const char *scheme, *path, *query;
	int port;
	int retries = 0;

	SSL_CTX *ssl_ctx;
	SSL *ssl;

	int i;

	for (i = 1; i < argc; i++) {
		if (!strcmp("-url", argv[i])) {
			if (i < argc - 1) {
				url = argv[i + 1];
			} else {
				syntax();
			}
		} else if (!strcmp("-ignore-cert", argv[i])) {
			ignore_cert = 1;
		} else if (!strcmp("-data", argv[i])) {
			if (i < argc - 1) {
				data_file = argv[i + 1];
			} else {
				syntax();
			}
		} else if (!strcmp("-retries", argv[i])) {
			if (i < argc - 1) {
				retries = atoi(argv[i + 1]);
			} else {
				syntax();
			}
		} else if (!strcmp("-help", argv[i])) {
			syntax();
		}
	}

	if (!url) {
		syntax();
	}

#ifdef _WIN32
	{
		WORD wVersionRequested;
		WSADATA wsaData;
		int err;

		wVersionRequested = MAKEWORD(2, 2);

		err = WSAStartup(wVersionRequested, &wsaData);
		if (err != 0) {
			printf("WSAStartup failed with error: %d\n", err);
			return 1;
		}
	}
#endif // _WIN32

	http_uri = evhttp_uri_parse(url);
	if (http_uri == NULL) {
		die("malformed url");
	}

	scheme = evhttp_uri_get_scheme(http_uri);
	if (scheme == NULL || (strcasecmp(scheme, "https") != 0 &&
	                       strcasecmp(scheme, "http") != 0)) {
		die("url must be http or https");
	}

	host = evhttp_uri_get_host(http_uri);
	if (host == NULL) {
		die("url must have a host");
	}

	port = evhttp_uri_get_port(http_uri);
	if (port == -1) {
		port = (strcasecmp(scheme, "http") == 0) ? 80 : 443;
	}

	path = evhttp_uri_get_path(http_uri);
	if (path == NULL) {
		path = "/";
	}

	query = evhttp_uri_get_query(http_uri);
	if (query == NULL) {
		snprintf(uri, sizeof(uri) - 1, "%s", path);
	} else {
		snprintf(uri, sizeof(uri) - 1, "%s?%s", path, query);
	}
	uri[sizeof(uri) - 1] = '\0';

	// Initialize OpenSSL
	SSL_library_init();
	ERR_load_crypto_strings();
	SSL_load_error_strings();
	OpenSSL_add_all_algorithms();

	/* This isn't strictly necessary... OpenSSL performs RAND_poll
	 * automatically on first use of random number generator. */
	r = RAND_poll();
	if (r == 0) {
		die_openssl("RAND_poll");
	}

	/* Create a new OpenSSL context */
	ssl_ctx = SSL_CTX_new(SSLv23_method());
	if (!ssl_ctx)
		die_openssl("SSL_CTX_new");

	#ifndef _WIN32
	/* TODO: Add certificate loading on Windows as well */

	/* Attempt to use the system's trusted root certificates.
	 * (This path is only valid for Debian-based systems.) */
	if (1 != SSL_CTX_load_verify_locations(ssl_ctx,
					       "/etc/ssl/certs/ca-certificates.crt",
					       NULL))
		die_openssl("SSL_CTX_load_verify_locations");
	/* Ask OpenSSL to verify the server certificate.  Note that this
	 * does NOT include verifying that the hostname is correct.
	 * So, by itself, this means anyone with any legitimate
	 * CA-issued certificate for any website, can impersonate any
	 * other website in the world.  This is not good.  See "The
	 * Most Dangerous Code in the World" article at
	 * https://crypto.stanford.edu/~dabo/pubs/abstracts/ssl-client-bugs.html
	 */
	SSL_CTX_set_verify(ssl_ctx, SSL_VERIFY_PEER, NULL);
	/* This is how we solve the problem mentioned in the previous
	 * comment.  We "wrap" OpenSSL's validation routine in our
	 * own routine, which also validates the hostname by calling
	 * the code provided by iSECPartners.  Note that even though
	 * the "Everything You've Always Wanted to Know About
	 * Certificate Validation With OpenSSL (But Were Afraid to
	 * Ask)" paper from iSECPartners says very explicitly not to
	 * call SSL_CTX_set_cert_verify_callback (at the bottom of
	 * page 2), what we're doing here is safe because our
	 * cert_verify_callback() calls X509_verify_cert(), which is
	 * OpenSSL's built-in routine which would have been called if
	 * we hadn't set the callback.  Therefore, we're just
	 * "wrapping" OpenSSL's routine, not replacing it. */
	SSL_CTX_set_cert_verify_callback (ssl_ctx, cert_verify_callback,
					  (void *) host);
	#endif // not _WIN32

	// Create event base
	base = event_base_new();
	if (!base) {
		perror("event_base_new()");
		return 1;
	}

	// Create OpenSSL bufferevent and stack evhttp on top of it
	ssl = SSL_new(ssl_ctx);
	if (ssl == NULL) {
		die_openssl("SSL_new()");
	}

	#ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME
	// Set hostname for SNI extension
	SSL_set_tlsext_host_name(ssl, host);
	#endif

	if (strcasecmp(scheme, "http") == 0) {
		bev = bufferevent_socket_new(base, -1, BEV_OPT_CLOSE_ON_FREE);
	} else {
		bev = bufferevent_openssl_socket_new(base, -1, ssl,
			BUFFEREVENT_SSL_CONNECTING,
			BEV_OPT_CLOSE_ON_FREE|BEV_OPT_DEFER_CALLBACKS);
	}

	if (bev == NULL) {
		fprintf(stderr, "bufferevent_openssl_socket_new() failed\n");
		return 1;
	}

	bufferevent_openssl_set_allow_dirty_shutdown(bev, 1);

	// For simplicity, we let DNS resolution block. Everything else should be
	// asynchronous though.
	evcon = evhttp_connection_base_bufferevent_new(base, NULL, bev,
		host, port);
	if (evcon == NULL) {
		fprintf(stderr, "evhttp_connection_base_bufferevent_new() failed\n");
		return 1;
	}

	if (retries > 0) {
		evhttp_connection_set_retries(evcon, retries);
	}

	evhttp_connection_set_closecb(evcon, close_cb, NULL);

	perform_http_request();
	event_base_dispatch(base);

	evhttp_connection_free(evcon);
	event_base_free(base);

#ifdef _WIN32
	WSACleanup();
#endif

	return 0;
}
Andre | 23 Jan 03:35 2015
Picon

Windows and bufferevent_socket_connect

Hello

I have a project that works perfectly in Ubuntu. But I want to port it 
to Windows. So I compiled it with MinGW for Windows and no errors 
occurred while compiling.

Then I tried to run it in Windows. It runs fine until there is an 
unexpected error: I could find out that the return value of 
bufferevent_socket_connect is < 0 which never happened in Ubuntu before.

Do you know why the same function call returns 0 for Ubuntu and -1 for 
Windows? If you don't, do you know a way to get to know what went wrong?

I called WSAStartup and I allowed the program in the Windows built-in 
firewall.

I am using Libevent 2.0.21-stable.

Thank you for your time and best wishes, Andre.
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

slump | 16 Jan 08:42 2015

unsend data in output buffer when close connection

 
I have used bufferevent to send data to socket
the application use variable length message, the first 4 bytes in message is the length
 
all thing is OK but for socket reconnect
 
if the socket is closed by peer and my app reconnect
peer may receive the wrong msg, the message length is  wrong
 
it seems the unsent data in bufferevent's output buffer will be send when the socket reconnect
but the message boundary is broken in output buffer
 
I guess it is because when I write message use bufferevent_write, one message is splited into two chunks in output buffer
one chunk is send to peer before socket closed
and the  next chunk is sent to peer after socket reconnect, so the message is broken
 
if it is true, can I get the data from output buffer and find the next message begin in data
so I can  discard the broken message and send the message follow it
 
but I find I can't get the data from output buffer, the evbuffer_copyout function return -1
 
and I can't find the function to clear the output buffer to discard all message in buffer
 
the only thing I can do is to free the bufferevent and create a new bufferevent when reconnect socket
 
is it the right way to do such thing?
 
Thank you!
 
2015-01-16
slump
Ralph Castain | 9 Jan 19:05 2015

Memory leak in locks?

Hi folks

I’ve trying to make a new library that uses libevent be valgrind clean. This is a threaded library, and so
libevent is thread enabled. I create a new event base, and I call event_base_free before finalizing.

However, I keep seeing reports like this:

==16029== 40 bytes in 1 blocks are still reachable in loss record 1 of 3
==16029==    at 0x4C288FE: malloc (in /home/common/local/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16029==    by 0x52A9EE1: evthread_posix_lock_alloc (evthread_pthread.c:46)
==16029==    by 0x5075572: event_global_setup_locks_ (event.c:2900)
==16029==    by 0x52AA08A: evthread_use_pthreads (evthread_pthread.c:185)

This is with libevent 2.0.22-stable. Is this an expected situation? Or is there something I need to do
beyond event_base_free to get a clean finalize?

Thanks
Ralph

***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Nick Mathewson | 5 Jan 16:32 2015
Picon

[ANN] Libevent 1.4.15-stable is released

Hello, all!

There are three new Libevent releases out today.  One of them is
1.4.15-stable, an updated oldstable release.  (I do not expect to do
any more 1.4 releases after this.)

This release fixes a moderately worrisome security issue in
evbuffers that could affect some programs; see
   http://archives.seul.org/libevent/users/Jan-2015/msg00010.html
for details.

You can get the source code from http://libevent.org or from one of
the git repositories.  If the website hasn't updated yet, you can
get the files from
    https://sourceforge.net/projects/levent/files/libevent/

As usual, make sure to check the GPG signatures on the source
distributions.

================================ Changes in 1.4.15-stable

Changes in 1.4.15-stable (5 January 2015)

 o Avoid integer overflow bugs in evbuffer_add() and related
functions.  See CVE-2014-6272 advisory for more information.
(d49bc0e88b81a5812116074dc007f1db0ca1eecd)

 o Pass flags to fcntl(F_SETFL) as int, not long (b3d0382)
 o Backport and tweak the LICENSE file for 1.4 (8a5ebd3)
 o set close-on-exec bit for filedescriptors created by dns subsystem
(9985231 Ralf Schmitt)
 o Replace unused case of FD_CLOSEONEXEC with a proper null statement. (44f04a2)
 o Fix kqueue correctness test on x84_64 (1c25b07)
 o Avoid deadlock when activating signals. (e0e6958)
 o Backport doc fix for evhttp_bind_socket. (95b71d0 Marco)
 o Fix an issue with forking and signal socketpairs in select/poll
backends (f0ff765)
 o Fix compilation on Visual Studio 2010 (53c47c2 VDm)
 o Defensive programming to prevent (hopefully impossible)
stack-stomping (2d8cf0b)
 o Check for POLLERR, POLLHUP and POLLNVAL for Solaris event ports
(353b4ac Trond Norbye)
 o Fix a bug that could allow dns requests with duplicate tx ids (e50ba5b)
 o Avoid truncating huge values for content-length (1d6e30e)
 o Take generated files out of git; add correct m4 magic for libtool
to auto* files (7cf794b)
 o Prefer autoregen -ivf to manual autogen.sh (823d9be)

================================ Acknowledgments

Thanks to everybody who contributed patches or bug reports or advice
to this release, including but not exclusively those mentioned
above.

Thanks also to everyone mentioned in the CVE-2014-6272 advisory.
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Nick Mathewson | 5 Jan 16:30 2015
Picon

[ANN] Libevent 2.0.22-stable is released

Hello, all!

There are three new Libevent releases out today.  One of them is
2.0.22-stable, an updated stable release.

This release fixes a moderately worrisome security issue in
evbuffers that could affect some programs; see
   http://archives.seul.org/libevent/users/Jan-2015/msg00010.html
for details.

You can get the source code from http://libevent.org or from one of
the git repositories.  If the website hasn't updated yet, you can
get the files from
    https://sourceforge.net/projects/levent/files/libevent/

As usual, make sure to check the GPG signatures on the source
distributions.

================================ Changes in 2.0.22-stable
Changes in version 2.0.22-stable (5 Jan 2015)

SECURITY FIXES (evbuffers)
 o Avoid integer overflow bugs in evbuffer_add() and related
functions.  See CVE-2014-6272 advisory for more information.
(20d6d4458bee5d88bda1511c225c25b2d3198d6c)

BUGFIXES (evhttp)
 o fix #73 and fix http_connection_fail_test to catch it (crash fix)
(b618204 Greg Hazel)
 o Avoid racy bufferevent activation (5eb1788 Nate Rosenblum)

BUGFIXES (compilation and portability)
 o Fix compilation with WIN32_HAVE_CONDITION_VARIABLES enabled (7e45739)
 o Fix missing AC_PROG_SED on older Autoconfs (9ab2b3f Tay Ray Chuan)
 o Backport libevent to vanilla Autoconf 2.59 (as used in RHEL5)
(74d4c44 Kevin Bowling)
 o Use AC_CONFIG_HEADERS in place of AM_CONFIG_HEADERS for autmake
1.13 compat (817ea36)
 o Rename configure.in to configure.ac to appease newer autoconfs (0c79787)
 o Avoid using top_srcdir in TESTS: new automakes do not like this (a55514e)
 o Use windows vsnprintf fixup logic on all windows environments (e826f19)
 o Fix a compiler warning when checking for arc4random_buf linker
breakage. (5cb3865)
 o Fix another arc4random_buf-related warning (e64a2b0)
 o Add -Qunused-arguments for clang on macos (b56611d Trond Norbye)

BUGFIXES (resource leaks/lock errors on error)
 o Avoid leaking fds on evconnlistener with no callback set (69db261)
 o Avoid double-close on getsockname error in evutil_ersatz_socketpair (0a822a6)
 o Fix a locking error in bufferevent_socket_get_dns_error. (0a5eb2e)
 o libevent/win32_dealloc() : fix sizeof(pointer) vs sizeof(*pointer)
(b8f5980 Frank Denis)

BUGFIXES: (other stability)
 o bufferevent_pair: don't call downcast(NULL) (f2428a2)
 o Consistently check for failure from evbuffer_pullup() (60f8f72)
 o Fix race caused by event_active (3c7d6fc vjpai)

BUGFIXES (miscellaneous)
 o Avoid redundant invocations of init_extension_functions for IOCP (3b77d62)
 o Typo fixes from Linus Nordberg (cec62cb, 8cd695b)
 o Add a few files created by "make verify" to .gitignore. (1a8295a
Pierre Phaneuf)
 o regress_buffer: fix 'memcmp' compare size (79800df Maks Naumov)
 o Fix bufferevent setwatermark suspend_read (b34e4ac ufo2243)
 o Fix evbuffer_peek() with len==-1 and start_at non-NULL. (fb7e76a)

BUFGIXES (evdns)
 o Checking request nameserver for NULL, before using it. (5c710c0
Belobrov Andrey)
 o Fix SEGFAULT after evdns_base_resume if no nameservers installed.
(f8d7df8 Azat Khuzhin)
 o Fix a crash in evdns related to shutting down evdns (9f39c88,e8fe749)

BUGFIXES (epoll)
 o Check does arch have the epoll_create and __NR_epoll_wait syscalls.
(dfe1e52 Marcin Juszkiewicz)

BUGFIXES (evutil_secure_random)
 o Avoid other RNG initialization FS reads when urandom file is
specified (9695e9c, bb52471)
 o When we seed from /proc/sys/kernel/random/uuid, count it as success (e35b540)
 o Document that arc4random is not a great cryptographic PRNG. (6e49696)
 o Add evutil_secure_rng_set_urandom_device_file (2bbb5d7)
 o Really remove RNG seeds from the stack (f5ced88)

DOCUMENTATION FIXES
 o Fix a mistake in evbuffer_remove() arguments in example http server
code (c322c20 Gyepi Sam)
 o Fix a typo in a comment in buffer.h. Spotted by Alt_F4 (773b0a5)
 o Clarify event_base_loop exit conditions (031a803)
 o Use FindClose for handle from FindFirstFile in http-server.c (6466e88)
 o Fix a typo in a doxygen comment. Reported by 亦得. (be1aeff)

================================ Acknowledgments

Thanks to everybody who contributed patches or bug reports or advice
to this release, including but not exclusively those mentioned
above.

Thanks also to everyone mentioned in the CVE-2014-6272 advisory.
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Nick Mathewson | 5 Jan 16:29 2015
Picon

[ANN] Libevent 2.1.5-beta is released

Hello, all!

There are three new Libevent releases out today.  One of them is
2.1.5-beta, the latest release in the development series.

This release fixes a moderately worrisome security issue in
evbuffers that could affect some programs; see
   http://archives.seul.org/libevent/users/Jan-2015/msg00010.html
for details.

You can get the source code from http://libevent.org or from one of
the git repositories.  If the website hasn't updated yet, you can
get the files from
    https://sourceforge.net/projects/levent/files/libevent/

As usual, make sure to check the GPG signatures on the source
distributions.

================================ Changes in 2.1.5-beta
Changes in version 2.1.5-beta (5 January 2015)

 Security Fixes (evbuffers)
   o Avoid integer overflow bugs in evbuffer_add() and related
functions.  See CVE-2014-6272 advisory for more information.
(d49bc0e88b81a5812116074dc007f1db0ca1eecd)

 New APIs (evconnlistener)
   o Provide support for SO_REUSEPORT through LEV_OPT_REUSABLE_PORT
(b625361 Maciej Soltysiak)

 Bugfixes (core)
    o Fix use-after-free error in EV_CLOSURE_EVENT callback (3cc0eac John Ohl)
    o Fix race caused by event_active (3c7d6fc vjpai)

 Bugfixes (evbuffer)
   o Fix evbuffer_peek() with len==-1 and start_at non-NULL. (ba59923)
   o Consistently check for failure from evbuffer_pullup() (60f8f72)
   o Fix evbuffer_peek() with len==-1 and start_at non-NULL. (fb7e76a)

 Bugfixes (windows, IOCP)
   o be async: avoid double close() (f133b86 Azat Khuzhin)

 Bugfixes (bufferevents)
   o Fix issue #127, double free for filterevents that use
BEV_OPT_CLOSE_ON_FREE (2c82aa0 John Ohl)
   o make bufferevent_getwatermark api more robust (a21e510 ufo2243)
   o [Bugfix] fix bufferevent setwatermark suspend_read (b34e4ac ufo2243)
   o bufferevent_openssl: reset fd_is_set when setfd with -1 is called
(3da84c2 Azat Khuzhin)
   o Fix compilation for older OpenSSL versions. (5c7282f Joakim Soderberg)

 New APIs (evhttp)
   o Add evhttp_connection_set_family() to set addrinfo->family for
DNS requests (12c29b0 Azat Khuzhin)
   o Implement interface that provides the ability to have an outbound
evhttp_connection free itself once all requests have completed
(2b9ec4c,10fe4f John Ohl)

 New APIs (core)
   o Implement new/free for struct evutil_monotonic_timer and export
monotonic time functions (f2645f8 Andrea Shepard)

 Bugfixes (evdns)
   o Load hosts file on Windows. (a0b247c Vilmos Nebehaj)
   o Don't truncate hosts file path on Windows. (d0dc861 Vilmos Nebehaj)
   o Fix a crash in evdns related to shutting down evdns (9f39c88,e8fe749)
   o evdns: avoid read-after-free in evdns_request_timeout_callback()
(61262a0 Azat Khuzhin)
   o Correctly handle allocation failures in evdns_getaddrinfo (6a53d15)
   o evdns: fix EVDNS_BASE_DISABLE_WHEN_INACTIVE in case
retransmit/retry (74d0eee Azat Khuzhin)
   o evdns: add retry/reissue tests for
EVDNS_BASE_DISABLE_WHEN_INACTIVE (3ca9d43 Azat Khuzhin)
   o evdns: fail ns after we are failing/retrasmitting request
(97c750d Azat Khuzhin)

 Bugfixes (evhttp)
   o http: reset connection before installing retry timer (fix http
retries handling) (bc79cc5 Azat Khuzhin)

 Testing
   o regress_dns: fix leaks in getaddrinfo_async{,_cancel_stress}
tests (2fdc5f2 Azat Khuzhin)
   o test: add family argument for http_connection_test_() (177b8a7
Azat Khuzhin)
   o test: add regress for evhttp_connection_set_family() with AF_INET
and AF_UNSPEC (42aefeb Azat Khuzhin)
   o test/http: add regress test for set family to AF_INET6 (3fbf3cc
Azat Khuzhin)
   o Update to a more recent tinytest_macros. (8da5a18)
   o test/regress: add simplestsignal: to track reorder bugs
separately (b897bef Azat Khuzhin)
   o test/evbuffer_peek: add regress in case we have first buffer
greater (e2d139d Azat Khuzhin)
   o More evbuffer_peek() test cases (154006a)
   o use correct tt macro for pointer compare (08c88ea)
   o regress_buffer: fix 'memcmp' compare size (79800df Maks Naumov)
   o Fix a use-after-free in unit tests. CID 752027 (3739057)
   o Fix a dead-code warning in unit tests. CID 1193548 (c119f24)
   o Use evutil_weakrand() in unit tests. (a677b72, 364c110)
   o Use a more precise calculation for max in time-ratelim.c (ca5b5c7)
   o Make a buffer larger in the tests to avoid a scary
evbuffer_copyout_from() (fb57b8b)
   o Fix several memory leaks in the unit tests. (89c1a3b)
   o Add test for evhttp_connection_free_on_completion (b0e9924 John Ohl)
   o Fix annoying heisenbug in test-time.c (cb73704)

 Sample code
   o Make http-server.c output into good html5 (6d72bdc)
   o Use FindClose for handle from FindFirstFile in http-server.c (6466e88)
   o https-client: add -retries argument, for connection retries
(d9da844 Azat Khuzhin)

 Bugfixes (build)
   o Add missing headerfile for cmake (15d90cc Trond Norbye)
   o ignore one more test binary (b6593aa Michael Richardson)
   o ignore config.cache/test-driver files (c83f333 Mike Frysinger)
   o add a --disable-samples configure flag (0c492b3 Mike Frysinger)
   o Add a few files created by "make verify" to .gitignore. (1a8295a
Pierre Phaneuf)
   o updates in cmake build (27bd9fa Sergey Nikulov)
   o Fix cmake error when the Module path has more than one entry.
(befbd13 Acer Yang)
   o Fix CMake shared library build (e69d910 Nobuaki Sukegawa)
   o Fix warnings when compiling with clang 3.5 (f5b4765 John Ohl)
   o Fix mixed declarations and code (forbidden by ISO C90) (8afbdbc
Thomas Bernard)

 Bugfixes (miscellaneous)
   o tree.h: drop duplicated content of tree.h (6193187 Azat Khuzhin)
   o evdns: disable probing with EVDNS_BASE_DISABLE_WHEN_INACTIVE
(610410b,ad0493e,fea86a6,d83b337,5ca9e97 Azat Khuzhin)
   o [Bugfix] fix grammer error (3a4d249 ufo2243)
   o Change return type of evutil_load_windows_system_library_ to
HMODULE (f691389)
   o Fix a c90 warning (76643dd)
   o Fix a typo in a doxygen comment. Reported by 亦得. (be1aeff)
   o remove trailing comma from enum (b361b8a Jean-Philippe Ouellet)

 Bugfixes (FreeBSD)
   o Handle ENOTCAPABLE from FreeBSD - this is returned if an event in
the changelist is for an FD that has been closed. (6fd7394 Adrian
Chadd)

================================ Acknowledgments

Thanks to everybody who contributed patches or bug reports or advice
to this release, including but not exclusively those mentioned
above.

Thanks also to everyone mentioned in the CVE-2014-6272 advisory.
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Nick Mathewson | 5 Jan 16:27 2015
Picon

Advisory: integer overflow in evbuffers for Libevent <= 1.4.14b,2.0.21,2.1.4-alpha [CVE-2014-6272]

(If my email mangles this message, you can get the original signed
version at http://www.wangafu.net/~nickm/volatile/advisory.txt.asc )

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

CVE-2014-6272

SUMMARY:

    A defect in the Libevent evbuffer API leaves some programs
    that pass insanely large inputs to evbuffers open to a
    possible heap overflow or infinite loop.

    Most programs will not be affected, but just in case, we're
    recommending that you patch or upgrade your Libevent.

    Thanks to Andrew Bartlett of Catalyst (catalyst.net.nz) for
    reporting this issue.

WHICH PROGRAMS ARE AFFECTED:
    Any program that does *not* use the evbuffer API is *not*
    affected.  (A program uses the evbuffer API if it calls any
    functions that begin with evbuffer_, bufferevent_, evhttp_, or
    evrpc_).

    A program _may_ be affected if it uses Libevent 1.4 and one of these
    functions, or a function that calls it:
         * evbuffer_add()
         * evbuffer_expand()
         * bufferevent_write()

    Not all such programs are vulnerable!  The attacker
    additionally needs to be able to find a way to provoke the
    program into trying to make a buffer larger than will fit into
    a single size_t.  (For example, if the buffer's current size
    is X, the attacker needs to provoke the program into calling
    evbuffer_add() or evbuffer_expand() with an argument size
    above SIZE_MAX-X. Most programs don't let an attacker do
    this.)

    A program _may_ be affected if it uses Libevent 2.0 or later
    and one of these functions, or a function that calls it:
         * evbuffer_add()
         * evbuffer_prepend()
         * evbuffer_expand()
         * exbuffer_reserve_space()
         * evbuffer_read()

    Not all such programs are vulnerable! The attacker
    additionally needs to be able to find a way to provoke the
    program into trying to make a buffer chunk larger than will
    fit into a single size_t or off_t. (For example, if the last
    chunk of the buffer's current size is X, the attacker needs to
    provoke the program into calling evbuffer_add() or another
    function with an argument size above SIZE_MAX-X or
    OFF_MAX-X. Most programs don't let an attacker do this.)

    I've used some tools to search for programs like this, and
    didn't find any glaring examples, but my exploit-generation
    skills are not the greatest, and I well could have missed
    something. My colleagues have looked some too, and didn't find
    anything that was actually exploitable to cause a heap
    overflow.

    Still, you should probably just upgrade Libevent if you're using
    the evbuffer interface.

WHAT TO DO:

  - Upgrade to the latest versions of Libevent.  They are Libevent
    1.4.15-stable, Libevent 2.0.22-stable, and Libevent
    2.1.5-beta.  They should be on the website soon.  Until then,
    you can go directly to
       https://sourceforge.net/projects/levent/files/libevent/

  - Alternatively, if you cannot upgrade, apply one of these
    commits to your older version of Libevent:
        7b21c4eabf1f3946d3f63cce1319c490caab8ecf for 1.4
        20d6d4458bee5d88bda1511c225c25b2d3198d6c for 2.0
        841ecbd96105c84ac2e7c9594aeadbcc6fb38bc4 for 2.1

  - Alternatively, if you use your operating system's package for
    Libevent, wait for your distribution to upgrade.

NOTES FOR LIBEVENT PROGRAMMERS:

  - Some non-security bugs related to unsigned integer overflow
    remain; they'll get fixed in the 2.1 series.

ACKNOWLEDGMENTS:

  Thanks to Andrew Bartlett of Catalyst (catalyst.net.nz) for
  reporting this issue.

  Thanks to Yawning, Peter Palfrader, Mark Ellzey, Thomas Hoger,
  and Adam Langley for advice and analysis.  Thanks to the Debian
  Security Team for getting me a CVE number.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQGVAwUBVKqs8ZEDl9iNKTGaAQK4ggv/RKwBNlP5993cZ9YPuo8e9ehJB8PeiO7c
z+jQNyuGK/kcKk0YLny3bbEOQ3Hkajm0SixLxeJw86NXfltcBCinJzjSSRZOElzv
+q4J4jUEAfpEn4bL1Y6nfLkzw0pJQVVcIrHIfOZ9hFK/FcBmHeYtX+nwy5dfQdJX
N74UJidTQMQBmUsZvyqVUupNd7bDnnTgkqzEUl+CeaLvdbi/iRZ1gACP4Zpuwlu+
j+dsC2K8k0mKqecQA9yUZUJuUWG1wZ5UHTzFC73lXpyQxzbVyK3N2G5KnrVQnvMi
S+dTcoR4yx4va4vOJJkghdakY1S1WSf2D2d7TTUZVGImtsUYooKNGXOj47E9N9ac
e2lMbP9iLZoFmhEuu7JTRDLcwx33fyBJsUSAzxgCSDGGkb6qUdUeMIjKxsHjTOI7
4I9e3ofHnBnjQuu05+AUgilHb7iXD6xmv8qovx6/BzmzpGJ9lycJbgC1KSn808Rl
G4sZJqt9NZ6M9eAuCLZDgF2iHmcJpfyR
=o6X/
-----END PGP SIGNATURE-----
***********************************************************************
To unsubscribe, send an e-mail to majordomo <at> freehaven.net with
unsubscribe libevent-users    in the body.

Roman Gershman | 1 Jan 07:46 2015

deadlock in bufferevent_disable

Hi,

I found the following deadlock bug in libevent code (2.0.21). It looks that the problem is in 
bufferevent_disable that locks bufev and transitively calls event_del_internal which waits for read callback to finish when it's not in the main thread. Meanwhile the main thread calls bufferevent_readcb which deadlocks on bufev.

Is it a known issue? Are there workarounds for this?

libevent thread:

#1  0x00007ffff794e672 in _L_lock_953 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff794e4da in __GI___pthread_mutex_lock (mutex=0x1031c80) at ../nptl/pthread_mutex_lock.c:114
#3  0x00007ffff771723c in _bufferevent_incref_and_lock (bufev=bufev <at> entry=0x1188c00) at bufferevent.c:582
#4  0x00007ffff771842a in bufferevent_readcb (fd=11, event=<optimized out>, arg=0x1188c00) at bufferevent_sock.c:133
#5  0x00007ffff770df24 in event_process_active_single_queue (activeq=0x1024640, base=0x1090000) at event.c:1350


user thread:
#0  pthread_cond_wait <at> <at> GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffff74fdfe5 in evthread_posix_cond_wait (_cond=0x1032430, _lock=0x1032460, tv=0x0) at evthread_pthread.c:156
#2  0x00007ffff770b9ed in event_del_internal (ev=<optimized out>) at event.c:2220
#3  event_del (ev=ev <at> entry=0x1188c10) at event.c:2188
#4  0x00007ffff7718645 in be_socket_disable (bufev=0x1188c00, event=<optimized out>) at bufferevent_sock.c:570
#5  0x00007ffff771702b in bufferevent_disable (bufev=0x1188c00, event=<optimized out>) at bufferevent.c:505
Gerry Sweeney | 28 Dec 14:30 2014
Picon

Help with Libevent multi-threaded HTTP server

Hi,

I am new to lib event so please forgive me if this is a noob question. I ran into a problem usinglib event and wonder if anyone can help

My question I hope is very simple: -

I am trying to create a simple multi-threaded HTTP server.  

I create and bind a non-blocking listening socket on port 8080, then for each thread I want to create to process HTTP requests I call the following

event_base_new()
evhttp_new()
evhttp_accept_socket_with_handle()
evhttp_set_gencb()
pthread_create(my_http_worker_thread_fn, base);

Then in the thread function for each worker thread I call event_base_dispatch((struct event_base*)arg);

Everything works as expected and I can process HTTP requests. The problem I am having is shutting down cleanly. 

After my main thread creates the worker threads I wait for a return key press on getchar() so I can press return to shutdown, after which I call event_base_loopbreak() on each event base I created, then I call pthread_join() from my main thread on each worker thread handle to wait for all threads to complete. 

The problem is, after calling event_base_loopbreak() I am expecting event_base_dispatch() to return but it never does, looking at the call stack for each thread they all appear to be waiting for an event. So I end up stuck waiting for threads to end which never do.

I am on OS X

Can anyone tell me what I am doing wrong/should be doing? Any help would be very much appreciated. 

Thanks,



Gmane