Bug 56034 - CPU consumption due to circular linked list in ap_core_input_filter()
Summary: CPU consumption due to circular linked list in ap_core_input_filter()
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Core (show other bugs)
Version: 2.2.15
Hardware: HP HP-UX
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Keywords: MassUpdate
Depends on:
Reported: 2014-01-20 03:51 UTC by venkatunix02
Modified: 2018-11-07 21:09 UTC (History)
0 users

trace (23.09 KB, text/plain)
2014-01-20 03:55 UTC, venkatunix02
log of top and tusc(syscalls) (10.19 KB, text/plain)
2014-01-20 03:56 UTC, venkatunix02
Mapping between stack trace, source code and itanium assembly code (16.08 KB, text/plain)
2014-01-20 03:57 UTC, venkatunix02

Note You need to log in before you can comment on or make changes to this bug.
Description venkatunix02 2014-01-20 03:51:45 UTC
We run Apache 2.2.15(+ fixes for certain CVEs) on HP Itanium server with HP-UX. 

Once in a while, we observe ~100% CPU consumption by the httpd processes.

1. configuration

MPM worker configuration:
<IfModule mpm_worker_module>
    ServerLimit          80
    StartServers         67
    MaxClients          4000
    MinSpareThreads     3350
    MaxSpareThreads     4000
    ThreadsPerChild     50
    MaxRequestsPerChild  1200

web server: Reverse proxy

3rd party module: Oracle webgate

2. Logs

Please find the attached files:
a) stack trace : bt_all
b) top and syscall trace (top_syscall_trace) 
c) Mapping between stack trace, source code and itanium assembly code(trace_srccode_assemblyCode)

3. Trace identified from the core dump generated in that scenario:

syscall trace shows that main thread waits for the user space thread that run ap_core_input_filter()

Thread 22 (system thread 1947121):
#0  0x40000000000977f0:2 in ap_core_input_filter () at core_filters.c:136
#1  0x40000000000c3f50:0 in ap_get_brigade () at util_filter.c:489
#2  0xc000000000ac76c0:0 in logio_in_filter () at mod_logio.c:129
#3  0x40000000000c3f50:0 in ap_get_brigade () at util_filter.c:489
#4  0x4000000000074980:0 in ap_rgetline_core () at protocol.c:231
#5  0xc000000004a33580:0 in ap_proxygetline () at mod_proxy_http.c:1658
#6  0xc000000004a338c0:0 in ap_proxy_http_process_response () at mod_proxy_http.c:1733
#7  0xc000000004a35ee0:0 in proxy_http_handler () at mod_proxy_http.c:2434
#8  0xc000000004929a10:0 in proxy_run_scheme_handler () at mod_proxy.c:2675
#9  0xc000000004928b60:0 in proxy_handler () at mod_proxy.c:1024
#10 0x400000000009d2d0:0 in ap_run_handler ()
#11 0x400000000009e9e0:0 in ap_invoke_handler () at config.c:381
#12 0x40000000000cdec0:0 in ap_process_request () at http_request.c:282
#13 0x40000000000c5a40:0 in ap_process_http_connection () at http_core.c:190
#14 0x40000000000b8d20:0 in ap_process_connection () at connection.c:189
#15 0x40000000000de4c0:0 in process_socket () at worker.c:590
#16 0x40000000000df860:0 in worker_thread () at worker.c:974
#17 0xc000000006f95200:0 in dummy_worker () at threadproc/unix/thread.c:160
#18 0xc00000000013fb20:0 in __pthread_bound_body () at pthreads/pthread.c:4875

Eventhough it shows Line:136, actual loop happens @ line 141:
   140      /* ### This is bad. */
   141      BRIGADE_NORMALIZE(ctx->b);

Due to compiler optimzation and macro @ line 141, this line no is not shown in trace.

As shown in the trace_srccode_assemblyCode attachment, linked list has become circular because address and content of (ctx->b)->list))->next are same

(gdb) info register r44
   gr44: 0x6000000000777de8

(gdb) x/gx0x6000000000777de8
0x6000000000777de8:     0x6000000000777de8

Prev’s value also contains the same.
(gdb) x/gx 0x6000000000774508
0x6000000000774508:     0x6000000000777de8

Please help to resolve this.
Comment 1 venkatunix02 2014-01-20 03:55:53 UTC
Created attachment 31228 [details]
Comment 2 venkatunix02 2014-01-20 03:56:57 UTC
Created attachment 31229 [details]
log of top and tusc(syscalls)
Comment 3 venkatunix02 2014-01-20 03:57:35 UTC
Created attachment 31230 [details]
Mapping between stack trace, source code and itanium assembly code
Comment 4 Yann Ylavic 2014-01-20 09:29:50 UTC
What compiler and APR versions are you using ?

If the former is gcc, can you compile with option "-fno-strict-aliasing" and see if it helps ?
Comment 5 venkatunix02 2014-01-23 05:42:52 UTC
(In reply to Yann Ylavic from comment #4)
> What compiler and APR versions are you using ?
> If the former is gcc, can you compile with option "-fno-strict-aliasing" and
> see if it helps ?

Thanks for checking this.

We use HP C compiler and APR version is 1.4.
Comment 6 Yann Ylavic 2014-01-23 09:38:28 UTC
The attachment 31230 [details] suggests you use APR <= 1.4.2 (without this fix http://svn.apache.org/viewvc?view=revision&revision=1074877).

struct apr_bucket_brigade {
    apr_pool_t *p;
    struct apr_bucket_list {
            struct apr_bucket *next;
            struct apr_bucket *prev;
    apr_bucket_alloc_t *bucket_alloc;

should be (from APR 1.4.3+) :

struct apr_bucket_brigade {
    apr_pool_t *p;
    struct apr_bucket_list {
            struct apr_bucket * volatile next;
            struct apr_bucket * volatile prev;
    apr_bucket_alloc_t *bucket_alloc;

Can you upgrade to the latest APR 1.4.8 and see if it helps?
Comment 7 Yann Ylavic 2014-01-23 09:58:41 UTC
Possible duplicate of bug 50190.
Comment 8 William A. Rowe Jr. 2018-11-07 21:09:54 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.