Hi, I have got an important performance drop after moving from apache httpd/2.2 to apache httpd/2.4. In order to tail down the problem, I did the following. I downloaded, built and installed Apache/2.2.34. I did add ssl-support and the following lines to httpd.conv: StartServers 1 ServerLimit 1 ThreadLimit 2500 ThreadsPerChild 2500 ThreadStackSize 1048576 MinSpareThreads 1 MaxSpareThreads 500 MaxClients 2500 MaxRequestsPerChild 0 Here the output using ApacheBench: Apache 2.2: ab -k -n 2000000 -c 60 'https://adnvl005:44300/' This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking adnvl005 (be patient) Completed 200000 requests Completed 400000 requests Completed 600000 requests Completed 800000 requests Completed 1000000 requests Completed 1200000 requests Completed 1400000 requests Completed 1600000 requests Completed 1800000 requests Completed 2000000 requests Finished 2000000 requests Server Software: Apache/2.2.34 Server Hostname: adnvl005 Server Port: 44300 SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256 Document Path: / Document Length: 44 bytes Concurrency Level: 60 Time taken for tests: 192.131 seconds Complete requests: 2000000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 1980237 Total transferred: 731130414 bytes HTML transferred: 88000000 bytes Requests per second: 10409.59 [#/sec] (mean) Time per request: 5.764 [ms] (mean) Time per request: 0.096 [ms] (mean, across all concurrent requests) Transfer rate: 3716.20 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.4 0 87 Processing: 0 6 1.2 6 71 Waiting: 0 6 1.2 5 70 Total: 0 6 1.3 6 98 Percentage of the requests served within a certain time (ms) 50% 6 66% 6 75% 6 80% 6 90% 7 95% 8 98% 9 99% 10 100% 98 (longest request) ---- I did then the same for Apache/2.4.34 (with apr-1.6.3 and apr-util-1.6.1), with the following changes in the httpd.conf (including ssl-support): StartServers 1 ServerLimit 1 ThreadLimit 2500 ThreadsPerChild 2500 ThreadStackSize 1048576 MinSpareThreads 1 MaxSpareThreads 500 MaxRequestWorkers 2500 MaxConnectionsPerChild 0 and here the output of ApacheBench: ab -k -n 2000000 -c 60 'https://adnvl005:44300/' This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking adnvl005 (be patient) Completed 200000 requests Completed 400000 requests Completed 600000 requests Completed 800000 requests Completed 1000000 requests Completed 1200000 requests Completed 1400000 requests Completed 1600000 requests Completed 1800000 requests Completed 2000000 requests Finished 2000000 requests Apache 2.4: Server Software: Apache/2.4.34 Server Hostname: adnvl005 Server Port: 44300 SSL/TLS Protocol: TLSv1/SSLv3,AES256-GCM-SHA384,1024,256 Document Path: / Document Length: 45 bytes Concurrency Level: 60 Time taken for tests: 571.058 seconds Complete requests: 2000000 Failed requests: 0 Write errors: 0 Keep-Alive requests: 1980229 Total transferred: 689130083 bytes HTML transferred: 90000000 bytes Requests per second: 3502.27 [#/sec] (mean) Time per request: 17.132 [ms] (mean) Time per request: 0.286 [ms] (mean, across all concurrent requests) Transfer rate: 1178.48 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 2.1 0 208 Processing: 0 17 20.3 10 285 Waiting: 0 17 20.3 10 285 Total: 0 17 20.4 10 285 Percentage of the requests served within a certain time (ms) 50% 10 66% 16 75% 23 80% 28 90% 44 95% 59 98% 79 99% 94 100% 285 (longest request) ---- As you can see, with apache2.4 it took more as twice as long. I think this is a bug. This performance drop occurs only if ssl is involved. For non-ssl request there is no performance impact. There are no error in neither logs. For technical reasons I can't increase the ServerLimit. Please let me know if you need any other information. Many thanks for your help. Regards Paolo
After some debug-session I found out that the problem are the ERR_clear_error calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the performance is the same like with httpd/2.2. Are those calls really needed in the ssl_io_input_read/ssl_filter_write function? Isn't it enough to have it only in the ssl_io_filter_handshake function. Or what about to call this function only if an error occurred: else /* (rc < 0) */ { int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc); conn_rec *c = (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl); + ERR_clear_error(); if (ssl_err == SSL_ERROR_WANT_READ) { Many thanks for any answer.
(In reply to paolo from comment #1) > After some debug-session I found out that the problem are the > ERR_clear_error calls in ssl_filter_write and ssl_io_input_read. If I remove > those calls the performance is the same like with httpd/2.2. > > Are those calls really needed in the ssl_io_input_read/ssl_filter_write > function? > Isn't it enough to have it only in the ssl_io_filter_handshake function. > > Or what about to call this function only if an error occurred: > > else /* (rc < 0) */ { > int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc); > conn_rec *c = > (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl); > > + ERR_clear_error(); > > if (ssl_err == SSL_ERROR_WANT_READ) { > > > Many thanks for any answer. ref: https://bz.apache.org/bugzilla/show_bug.cgi?id=60223
For 100% clarity, this was observed with the version of ab shipped with 2.2.34, or the version shipped with 2.4.24? It should be obvious that ab has also undergone some enhancements and changes for support of TLS, and the two different versions are expected to produce two different results.
I used for both tests the ab installed on my host: paolo@adnvl005:~ ab -V This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/\ Apart that I get to the same performance issue if using other software, like gatling etc.
Confirmed in dev@ discussion, this flavor of ab dates to the 2.2.x epoch and is consistent for both (observed with 'This is ApacheBench, Version 2.3 <$Revision: 655654 $>'.) Because of the transition of TLS session cache to TLS session tokens, it may be worthwhile to see if this is only true of very ancient clients, and whether this combination of an older ab with an older openssl is in part to blame for the failure to speak efficiently to modern httpd linked to modern openssl. This can be tested using the most recent ab utility against both servers.
> Apart that I get to the same performance issue if using other software, like gatling etc. Thanks Paolo! Presuming the other software is on the same machine, relying on the same openssl client library, what is your system's response to; $ openssl version
Hi William, here the openssl-version which is installed on my host: paolo@adnvl005:~/ openssl version OpenSSL 1.0.1e-fips 11 Feb 2013