Bug 44584 - ab.c: failure after apr_socket_connect() returns 37/EALREADY
Summary: ab.c: failure after apr_socket_connect() returns 37/EALREADY
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: support (show other bugs)
Version: 2.2.8
Hardware: PC FreeBSD
: P2 major with 29 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-03-11 13:45 UTC by Ivan Voras
Modified: 2009-03-02 13:23 UTC (History)
1 user (show)



Attachments
Patch for ab.c to check for EINTR (431 bytes, patch)
2008-03-11 13:45 UTC, Ivan Voras
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ivan Voras 2008-03-11 13:45:57 UTC
Created attachment 21654 [details]
Patch for ab.c to check for EINTR

The ab (apache benchmark) utility creates many connections to the target web server, but ignores the case of EINTR that can be returned by the OS's connect() call. Because of that, it's almost unusable for generating large loads on OSes that actually generate EINTR, like FreeBSD.

From connect(2) manual:

     [EINTR]            The connection attempt was interrupted by the delivery
                        of a signal.  The connection will be established in
                        the background, as in the case of EINPROGRESS.

The attached patch simply adds the check for EINTR. This has been verified to work.
Comment 1 Roy T. Fielding 2008-05-12 14:45:17 UTC
EINTR is checked by apr_socket_connect while making the actual
system call for connect.

If this isn't working, perhaps a different system call is not
being restarted.  Please grab the latest version of ab.c
from trunk and see if that fixes your issues.

http://svn.apache.org/repos/asf/httpd/httpd/trunk/support/ab.c

Comment 2 Ivan Voras 2008-05-12 15:40:12 UTC
(In reply to comment #1)
> EINTR is checked by apr_socket_connect while making the actual
> system call for connect.
> 
> If this isn't working, perhaps a different system call is not
> being restarted.  Please grab the latest version of ab.c
> from trunk and see if that fixes your issues.
> 
> http://svn.apache.org/repos/asf/httpd/httpd/trunk/support/ab.c
> 

The newest version doesn't work:

$ ./ab -c 20 -n 100 http://my.host/
This is ApacheBench, Version 2.3 <$Revision$>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking my.host (be patient)...
Test aborted after 10 failures

apr_socket_connect(): Operation already in progress (37)
Total of 55 requests completed

It seems like errno=37 (EALREADY) also needs to be checked:

     37 EALREADY Operation already in progress.  An operation was attempted on
             a non-blocking object that already had an operation in progress.

There is no APR_STATUS_IS_EALREADY.
Comment 3 Ivan Voras 2008-07-04 04:42:46 UTC
Checked on Apache 2.2.9 release, same error:

# ab -n 1000 -c 4 http://localhost/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests

Test aborted after 10 failures

apr_socket_connect(): Operation already in progress (37)
Total of 560 requests completed
Comment 4 Philp M. Gollucci 2009-01-18 16:19:18 UTC
CC myself on FreeBSD related bugs
Comment 5 Jeff Trawick 2009-01-18 16:51:32 UTC
>apr_socket_connect(): Operation already in progress (37)
>Total of 55 requests completed

This happens on Leopard too and is related to the kqueue usage, so the problem could be in APR.

ab occasionally is awoken from kevent() with a notification for a socket which is in the process of connecting but has not yet completed.  ab thinks the condition must mean connect-complete, calls apr_socket_connect(), gets EALREADY (37), and bombs.

My unproven impression is that ab isn't keeping the pollset minimized (i.e., not removing items from the pollset as soon as it stops needing to be notified for them, but I'm not sure.

FWIW, a hack like this keeps ab going:

             if (rv & APR_POLLOUT) {
                 if (c->state == STATE_CONNECTING) {
                     apr_pollfd_t remove_pollfd;
                     rv = apr_socket_connect(c->aprsock, destsa);
+                    if (rv == 37) {
+                        continue;
+                    }
                     remove_pollfd.desc_type = APR_POLL_SOCKET;
                     remove_pollfd.desc.s = c->aprsock;
                     apr_pollset_remove(readbits, &remove_pollfd);
Comment 6 Jeff Trawick 2009-03-02 13:23:48 UTC
fixed in trunk

Changes were significant, so I have no plans to propose for backport to the 2.2.x branch in the short term.  With some testing/everyday use in trunk on various platforms, it may be appropriate to backport later.