Bug 63296 - mod_event: Connection timeouts with ab in case of keepalive and high concurrency
Summary: mod_event: Connection timeouts with ab in case of keepalive and high concurrency
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mpm_event (show other bugs)
Version: 2.4.6
Hardware: PC Linux
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-03-28 11:56 UTC by Horst Reiterer
Modified: 2019-03-28 13:00 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Horst Reiterer 2019-03-28 11:56:49 UTC
Problem description:

Running ab causes a connection timeout (/ serves an empty HTML file):

$ ab -n 10000 -c 128 -k -A Administrator:Faba8888 http://localhost:17080/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 9901 requests completed

We saw similar issues during load testing of our application with another load testing tool.
Based on strace, it looks like connect(2) times out.

In case of mod_worker, it's clear that a concurrency level of 9+ must lead to delays depending on KeepAliveTimeout because one thread handles one connection throughout the lifetime of the connect. In case of mod_event, however, one thread handles one request from any connection. A level of 9+ works with mod_event, but 128 fails for reasons I cannot explain. Changing KeepAliveTimeout affects mod_event in this scenario too.

In case of a proprietary MPM that handles connections and requests in a manner similar to mod_event, I cannot reproduce the issue. In that implementation, every thread handles new connections and incoming data based on epoll.

Workarounds:

1) Disabling keepalive connections (client or server) works around the issue.
2) Reducing the number of concurrent requests (-c) to 8 works around the issue.

Configuration:

MPM (only one process with a static number of threads):

<IfModule event.c>
  StartServers         1
  ServerLimit          1
  MaxRequestWorkers    8
  MinSpareThreads      8
  MaxSpareThreads      8
  ThreadsPerChild      8
</IfModule>

Keepalive:

  KeepAlive On
  MaxKeepAliveRequests 0
  KeepAliveTimeout 120
Comment 1 Horst Reiterer 2019-03-28 11:58:51 UTC
OS:

CentOS Linux release 7.6.1810 (Core)
3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

RPMs:

httpd-2.4.6-88.el7.centos.x86_64
httpd-tools-2.4.6-88.el7.centos.x86_64
Comment 2 Ruediger Pluem 2019-03-28 13:00:01 UTC
Have a look at http://httpd.apache.org/docs/2.4/en/mod/event.html#asyncrequestworkerfactor