Bug 62590

Summary: performance drop after moving from apache 2.2 to apache 2.4
Product: Apache httpd-2 Reporter: paolo <paolo>
Component: CoreAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: NEW ---    
Severity: normal CC: veit
Priority: P2    
Version: 2.4.34   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description paolo 2018-07-31 16:35:00 UTC
Hi,

I have got an important performance drop after moving from apache httpd/2.2 to apache httpd/2.4.

In order to tail down the problem, I did the following.

I downloaded, built and installed Apache/2.2.34. I did add ssl-support and the following lines to httpd.conv:
StartServers         1
ServerLimit          1
ThreadLimit          2500
ThreadsPerChild      2500
ThreadStackSize      1048576 
MinSpareThreads      1
MaxSpareThreads      500
MaxClients           2500
MaxRequestsPerChild  0


Here the output using ApacheBench:

Apache 2.2:
ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking adnvl005 (be patient)
Completed 200000 requests
Completed 400000 requests
Completed 600000 requests
Completed 800000 requests
Completed 1000000 requests
Completed 1200000 requests
Completed 1400000 requests
Completed 1600000 requests
Completed 1800000 requests
Completed 2000000 requests
Finished 2000000 requests


Server Software:        Apache/2.2.34
Server Hostname:        adnvl005
Server Port:            44300
SSL/TLS Protocol:       TLSv1/SSLv3,AES256-GCM-SHA384,1024,256

Document Path:          /
Document Length:        44 bytes

Concurrency Level:      60
Time taken for tests:   192.131 seconds
Complete requests:      2000000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    1980237
Total transferred:      731130414 bytes
HTML transferred:       88000000 bytes
Requests per second:    10409.59 [#/sec] (mean)
Time per request:       5.764 [ms] (mean)
Time per request:       0.096 [ms] (mean, across all concurrent requests)
Transfer rate:          3716.20 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0      87
Processing:     0    6   1.2      6      71
Waiting:        0    6   1.2      5      70
Total:          0    6   1.3      6      98

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      6
  90%      7
  95%      8
  98%      9
  99%     10
 100%     98 (longest request)
 
----

I did then the same for Apache/2.4.34 (with apr-1.6.3 and apr-util-1.6.1), with the following changes in the httpd.conf (including ssl-support):
StartServers         1
ServerLimit          1
ThreadLimit          2500
ThreadsPerChild      2500
ThreadStackSize      1048576 
MinSpareThreads      1
MaxSpareThreads      500
MaxRequestWorkers      2500
MaxConnectionsPerChild  0

 
and here the output of ApacheBench:

ab -k -n 2000000 -c 60 'https://adnvl005:44300/'
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking adnvl005 (be patient)
Completed 200000 requests
Completed 400000 requests
Completed 600000 requests
Completed 800000 requests
Completed 1000000 requests
Completed 1200000 requests
Completed 1400000 requests
Completed 1600000 requests
Completed 1800000 requests
Completed 2000000 requests
Finished 2000000 requests


Apache 2.4:
Server Software:        Apache/2.4.34
Server Hostname:        adnvl005
Server Port:            44300
SSL/TLS Protocol:       TLSv1/SSLv3,AES256-GCM-SHA384,1024,256

Document Path:          /
Document Length:        45 bytes

Concurrency Level:      60
Time taken for tests:   571.058 seconds
Complete requests:      2000000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    1980229
Total transferred:      689130083 bytes
HTML transferred:       90000000 bytes
Requests per second:    3502.27 [#/sec] (mean)
Time per request:       17.132 [ms] (mean)
Time per request:       0.286 [ms] (mean, across all concurrent requests)
Transfer rate:          1178.48 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.1      0     208
Processing:     0   17  20.3     10     285
Waiting:        0   17  20.3     10     285
Total:          0   17  20.4     10     285

Percentage of the requests served within a certain time (ms)
  50%     10
  66%     16
  75%     23
  80%     28
  90%     44
  95%     59
  98%     79
  99%     94
 100%    285 (longest request)
 
----

As you can see, with apache2.4 it took more as twice as long. I think this is a bug.
This performance drop occurs only if ssl is involved. For non-ssl request there is no performance impact.
There are no error in neither logs.
 
For technical reasons I can't increase the ServerLimit.
 
Please let me know if you need any other information.
 
Many thanks for your help.
 
Regards
 	Paolo
Comment 1 paolo 2018-08-27 14:11:06 UTC
After some debug-session I found out that  the problem are the ERR_clear_error calls in ssl_filter_write and ssl_io_input_read. If I remove those calls the performance is the same like with httpd/2.2.

Are those calls really needed in the ssl_io_input_read/ssl_filter_write function?
Isn't it enough to have it only in the ssl_io_filter_handshake function.

Or what about to call this function only if an error occurred:

        else /* (rc < 0) */ {
            int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
            conn_rec *c = (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);

 +           ERR_clear_error();

            if (ssl_err == SSL_ERROR_WANT_READ) {


Many thanks for any answer.
Comment 2 Eric Covener 2018-08-27 14:13:45 UTC
(In reply to paolo from comment #1)
> After some debug-session I found out that  the problem are the
> ERR_clear_error calls in ssl_filter_write and ssl_io_input_read. If I remove
> those calls the performance is the same like with httpd/2.2.
> 
> Are those calls really needed in the ssl_io_input_read/ssl_filter_write
> function?
> Isn't it enough to have it only in the ssl_io_filter_handshake function.
> 
> Or what about to call this function only if an error occurred:
> 
>         else /* (rc < 0) */ {
>             int ssl_err = SSL_get_error(inctx->filter_ctx->pssl, rc);
>             conn_rec *c =
> (conn_rec*)SSL_get_app_data(inctx->filter_ctx->pssl);
> 
>  +           ERR_clear_error();
> 
>             if (ssl_err == SSL_ERROR_WANT_READ) {
> 
> 
> Many thanks for any answer.

ref: https://bz.apache.org/bugzilla/show_bug.cgi?id=60223
Comment 3 William A. Rowe Jr. 2018-08-28 17:32:50 UTC
For 100% clarity, this was observed with the version of ab shipped with 2.2.34, or the version shipped with 2.4.24? It should be obvious that ab has also undergone some enhancements and changes for support of TLS, and the two different versions are expected to produce two different results.
Comment 4 paolo 2018-08-29 07:36:55 UTC
I used for both tests the ab installed on my host:
paolo@adnvl005:~ ab -V
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/\

Apart that I get to the same performance issue if using other software, like gatling etc.
Comment 5 William A. Rowe Jr. 2018-08-29 17:21:07 UTC
Confirmed in dev@ discussion, this flavor of ab dates to the 2.2.x epoch and is consistent for both (observed with 'This is ApacheBench, Version 2.3 <$Revision: 655654 $>'.)

Because of the transition of TLS session cache to TLS session tokens, it may be worthwhile to see if this is only true of very ancient clients, and whether this combination of an older ab with an older openssl is in part to blame for the failure to speak efficiently to modern httpd linked to modern openssl. This can be tested using the most recent ab utility against both servers.
Comment 6 William A. Rowe Jr. 2018-08-29 17:23:38 UTC
> Apart that I get to the same performance issue if using other software, like gatling etc.

Thanks Paolo!

Presuming the other software is on the same machine, relying on the same openssl client library, what is your system's response to;

 $ openssl version
Comment 7 paolo 2018-08-30 07:26:44 UTC
Hi William,

here the openssl-version which is installed on my host:

paolo@adnvl005:~/ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013