Bug 42262

Summary: mod_dav / worker fails to allocate memory and segfaults by for larg property-xml files
Product: Apache httpd-2 Reporter: Roger Bonati <rbonati>
Component: mpm_workerAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED LATER    
Severity: normal CC: prez
Priority: P2 Keywords: MassUpdate
Version: 2.2.6   
Target Milestone: ---   
Hardware: Other   
OS: other   
Attachments: output from gdb for apache core

Description Roger Bonati 2007-04-26 06:54:40 UTC 
Hello All,

when upgrading our httpd from version 1.3 to 2.0.59 we run into the trouble that
by defining a number of ThreadsPerChild > 10 for the worker.c, apache fails to
allocate memory for a webdav-client with a huge property-xml file (say gt 1,4MB).
We handled this problem by reducing the number of ThreadsPerChild to an amount
of 8, but this is not really a practicable solution, just a prima-facie workaround. 

The errorlog gives (e.g):

[Thu Apr 26 09:41:05 2007] [notice] child pid 29127 exit signal Segmentation
fault (11)

A strace on this pids shows something like this:

httpd_strace.7164:mmap2(NULL, 1048576, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)

The problem was reproducible

Could someone see for this problem?

Thanks and greetings

Roger
Comment 1 Ruediger Pluem 2007-04-26 13:07:09 UTC 
Please provide a backtrace of the crashed process (see also
http://httpd.apache.org/dev/debugging.html).
What OS are you using?
Comment 2 Roger Bonati 2007-04-27 02:01:53 UTC 
(In reply to comment #1)
> Please provide a backtrace of the crashed process (see also
> http://httpd.apache.org/dev/debugging.html).
> What OS are you using?

Hello,
uname -a gives:
Linux dfs080 2.6.16.43-MODULES-P4 #1 SMP Tue Mar 6 07:56:07 CET 2007 i686 GNU/Linux

Tanks for the fast reply
Comment 3 Roger Bonati 2007-04-27 03:43:53 UTC 
Hello,

this is the output of gdb -p on a pid that receives an segfault:

(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1229497424 (LWP 24355)]
0xb7e3478b in apr_pool_tag () from /opt/gmx/apache2/lib/libapr-0.so.0

Greetings

Roger
Comment 4 Roger Bonati 2007-04-27 05:40:39 UTC 
Hello Ruediger,

here is the bt of an httpd version with debug symbols, hope this helps:

(gdb) thread apply all bt full

Thread 27 (Thread -1221424208 (LWP 7841)):
#0  0xb7de678b in apr_pool_tag (pool=0x0, tag=0x80a0587 "subrequest")
    at apr_pools.c:1795
No locals.
#1  0x08088e90 in make_sub_request (r=0xa39a750, next_filter=0x0)
    at request.c:1466
        rrp = (apr_pool_t *) 0x0
#2  0x08089b10 in ap_sub_req_method_uri (method=0x8098201 "GET", 
    new_uri=0x8132690 "index.html", r=0xa39a750, next_filter=0x80a0587)
    at request.c:1579
        rnew = (request_rec *) 0xa39a750
        res = 0
        udir = 0x0
#3  0x08089be6 in ap_sub_req_lookup_uri (new_uri=0x0, r=0x0, next_filter=0x0)
    at request.c:1620
No locals.
#4  0xb7ba3bf0 in fixup_dir (r=0xa39a750) at mod_dir.c:183
        name_ptr = 0x80a0587 "subrequest"
        rr = (request_rec *) 0xb7327d08
        d = (dir_config_rec *) 0x810b350
        dummy_ptr = {0xb7ba39c3 "\201Ã\205\a"}
        num_names = 1
        error_notfound = 0
#5  0x080894bf in ap_process_request_internal (r=0xa39a750) at request.c:71
        file_req = 0
        access_status = 135493400
#6  0x08089bb0 in ap_sub_req_method_uri (method=0x8098201 "GET", 
    new_uri=0x818c218 "àÁ\030\b¨`\030\bøa\023\b", r=0x818c218, 
    next_filter=0x80a0587) at request.c:1608
        rnew = (request_rec *) 0xa39a750
        res = 0
        udir = 0x0
#7  0x08089be6 in ap_sub_req_lookup_uri (new_uri=0x0, r=0x0, next_filter=0x0)
    at request.c:1620
No locals.
#8  0xb7bca4b3 in dav_insert_coreprop (propdb=0xa396cf0, propid=134874503, 
    name=0xb7bd1c80 "getcontenttype", what=DAV_PROP_INSERT_VALUE, phdr=0x0, 
    inserted=0xb7327e34) at props.c:326
        locks = (dav_lock *) 0x0
        err = (dav_error *) 0x80a0587
#9  0xb7bca892 in dav_get_allprops (propdb=0xa396cf0, 
    what=DAV_PROP_INSERT_VALUE) at props.c:667
        db_hooks = (const dav_hooks_db *) 0xb7bbb120
        hdr = {first = 0xa397248, last = 0xa397840}
        names_ptr = (char **) 0x81326b8
        hdr_ns = {first = 0xa397538, last = 0xa3975d0}
        result = {propstats = 0x0, xmlns = 0x0}
        found_contenttype = 0
        found_contentlang = 0
        unused_inserted = DAV_PROP_INSERT_NOTDEF
#10 0xb7bc11a1 in dav_propfind_walker (wres=0xb7328664, calltype=2)
    at mod_dav.c:1913
        what = DAV_PROP_INSERT_NOTDEF
        ctx = (dav_walker_ctx *) 0xb7328840
        err = (dav_error *) 0x0
        propdb = (dav_propdb *) 0xa396cf0
        propstats = {propstats = 0x0, xmlns = 0x0}
#11 0xb7bb73a5 in dav_fs_walker (fsctx=0xb7328660, depth=2147483642)
    at repos.c:1394
        params = (const dav_walk_params *) 0xb7328840
        pool = (apr_pool_t *) 0x818c1e0
        err = (dav_error *) 0x0
        isdir = 1
        dirent = {pool = 0x0, valid = 0, protection = 0, filetype = 4096, 
  user = 4096, group = 8, inode = 1177675838, device = 5040432765133127680, 
  nlink = 0, size = 1177676261, csize = 23742000, atime = 0, 
  mtime = -5207084655757261480, ctime = 578156477196186112, fname = 0x0, 
  name = 0xb7bd446c "`\203\001", filehand = 0xb7328704}
        dirp = (apr_dir_t *) 0x1051159
#12 0xb7bb76c2 in dav_fs_walker (fsctx=0xb7328660, depth=2147483643)
    at repos.c:1519
        save_path_len = 74
        save_uri_len = 45
        save_path2_len = 40
        len = 16
        status = 0
        params = (const dav_walk_params *) 0xb7328840
        pool = (apr_pool_t *) 0x818c1e0
        err = (dav_error *) 0x0
        isdir = 2147483643
        dirent = {pool = 0x818c1e0, valid = 33562624, protection = 0, 
  filetype = 4096, user = 4096, group = 8, inode = 97193, 
  device = 5039966567907983360, nlink = 0, size = 1177676263, 
  csize = 863742000, atime = 0, mtime = -5207084655762793400, 
  ctime = 578156477196186112, fname = 0x0, 
  name = 0xa396cd8 "Lehmeier Wilhelm", filehand = 0xb7328704}
        dirp = (apr_dir_t *) 0xa318460
#13 0xb7bb76c2 in dav_fs_walker (fsctx=0xb7328660, depth=2147483644)
    at repos.c:1519
        save_path_len = 72
        save_uri_len = 43
        save_path2_len = 39
        len = 1
        status = 0
        params = (const dav_walk_params *) 0xb7328840
        pool = (apr_pool_t *) 0x818c1e0
        err = (dav_error *) 0x0
        isdir = 2147483644
        dirent = {pool = 0x818c1e0, valid = 33562624, protection = 0, 
  filetype = 4096, user = 4096, group = 8, inode = 2684355147, 
  device = 5052652449601421312, nlink = 0, size = 1177676353, 
  csize = 873742000, atime = 0, mtime = -5207084655762793400, 
  ctime = 578156477196186112, fname = 0x0, name = 0xa3319e0 "L", 
  filehand = 0xb7328704}
        dirp = (apr_dir_t *) 0x9df2988
#14 0xb7bb76c2 in dav_fs_walker (fsctx=0xb7328660, depth=2147483645)
    at repos.c:1519
        save_path_len = 55
        save_uri_len = 26
        save_path2_len = 23
        len = 16
        status = 0
        params = (const dav_walk_params *) 0xb7328840
            err = (dav_error *) 0x0
        isdir = 2147483645
        dirent = {pool = 0x818c1e0, valid = 33562624, protection = 0, 
  filetype = 64, user = 4096, group = 0, inode = 2415919781, 
  device = 5052614984601698304, nlink = 0, size = 1177676377, 
  csize = 963742000, atime = 0, mtime = -5207084655792359104, 
  ctime = 578156477196186112, fname = 0x0, 
  name = 0x9df1e30 "Kunden_6300_6309", filehand = 0xb7328704}
        dirp = (apr_dir_t *) 0x9df1cb0
#15 0xb7bb76c2 in dav_fs_walker (fsctx=0xb7328660, depth=2147483646)
    at repos.c:1519
        save_path_len = 40
        save_uri_len = 11
        save_path2_len = 9
        len = 14
        status = 0
        params = (const dav_walk_params *) 0xb7328840
        pool = (apr_pool_t *) 0x818c1e0
        err = (dav_error *) 0x0
        isdir = 2147483646
        dirent = {pool = 0x818c1e0, valid = 33562624, protection = 135987200, 
  filetype = 109, user = 4096, group = 0, inode = 2148008234, 
  device = 5052838679383375872, nlink = 0, size = 1177676378, 
    pool = (apr_pool_t *) 0x818c1e0
  csize = 923742000, atime = 0, mtime = -5207084655792420664, 
  ctime = 578156477196186112, fname = 0x0, name = 0x9ded150 "Agt 6300 _6309", 
  filehand = 0xb7328704}
        dirp = (apr_dir_t *) 0x81bfe28
#16 0xb7bb76c2 in dav_fs_walker (fsctx=0xb7328660, depth=2147483647)
    at repos.c:1519
        save_path_len = 30
        save_uri_len = 1
        save_path2_len = 0
        len = 9
        status = 0
        params = (const dav_walk_params *) 0xb7328840
        pool = (apr_pool_t *) 0x818c1e0
        err = (dav_error *) 0x0
        isdir = 2147483647
        dirent = {pool = 0x818c1e0, valid = 33562624, protection = 135942200, 
  filetype = 3084155168, user = 3084152668, group = 3084155168, 
  inode = 1621306689, device = 13242957831183959512, nlink = -1210812128, 
  size = 135840200, csize = -1212332948, atime = 7368509188, 
  mtime = -5207083982839970312, ctime = 1099647468000, fname = 0x0, 
  name = 0x81b2260 "Dillingen", filehand = 0xb7328704}
        dirp = (apr_dir_t *) 0x81b0930
#17 0    is_move=0, root_dst=0x0, response=0x80a0587) at repos.c:1737
        fsctx = {params = 0xb7328840, wres = {walk_ctx = 0xb7328840, 
    pool = 0x818c1e0, resource = 0xb7328674, response = 0x0}, res1 = {
    type = DAV_RESOURCE_TYPE_REGULAR, exists = 1, collection = 1, 
    versioned = 0, baselined = 0, working = 0, 
    uri = 0x818db58 "/Dillingen/Agt 6300 _6309/Kunden_6300_6309/L/Lehmeier
Wilhelm/", info = 0xb732869c, hooks = 0xb7bbb1a0, pool = 0x818c1e0}, info1 = {
    pool = 0x818c1e0, 
    pathname = 0x818da58 "/var/gmx/users/37/93/17109337/Dillingen/Agt 6300
_6309/Kunden_6300_6309/L/Lehmeier Wilhelm", finfo = {pool = 0x818c1e0, 
      valid = 7598448, protection = 1872, filetype = APR_DIR, user = 17109337, 
      group = 17109337, inode = 97193, device = 2049, nlink = 3, size = 4096, 
      csize = 0, atime = 1177675838000000, mtime = 1173567205000000, 
      ctime = 1177676261000000, 
      fname = 0x818da58 "/var/gmx/users/37/93/17109337/Dillingen/Agt 6300
_6309/Kunden_6300_6309/L/Lehmeier Wilhelm", name = 0x0, filehand = 0x0}}, path1 = {
    alloc_len = 256, cur_len = 90, 
    buf = 0x818da58 "/var/gmx/users/37/93/17109337/Dillingen/Agt 6300
_6309/Kunden_6300_6309/L/Lehmeier Wilhelm"}, uri_buf = {alloc_len = 256, cur_len
= 62, 
    buf = 0x818db58 "/Dillingen/Agt 6300 _6309/Kunden_6300_6309/L/Lehmeier
Wilhelm/"}, res2 = {type = DAV_RESOURCE_TYPE_UNKNOWN, exists = 0, collection = 1, 
    versioned = 0, baselined = 0, working = 0, uri = 0x0, info = 0x0, 
    hooks = 0x0, pool = 0x0}, info2 = {pool = 0x0, pathname = 0x0, finfo = {
xb7bb7dfd in dav_fs_internal_walk (params=0x1, depth=2147483647,
       pool = 0x0, valid = 0, protection = 0, filetype = APR_NOFILE, user = 0, 
      group = 0, inode = 0, device = 0, nlink = 0, size = 0, csize = 0, 
      atime = 0, mtime = 0, ctime = 0, fname = 0x0, name = 0x0, 
      filehand = 0x0}}, path2 = {alloc_len = 0, cur_len = 56, buf = 0x0}, 
  locknull_buf = {alloc_len = 256, cur_len = 0, 
    buf = 0x81b1060 "/var/gmx/users/37/93/17109337/Dillingen/Agt 6300
_6309/Kunden_6300_6309/L/Lang Josef/.DAV/.locknull"}}
        err = (dav_error *) 0x0
        cm_ctx = {is_move = 0, work_buf = {alloc_len = 0, cur_len = 0, 
    buf = 0x0}, res_dst = 0x0, root = 0x0, pool = 0x0}
#18 0xb7bc193b in dav_method_propfind (r=0x818c218) at mod_dav.c:2045
        resource = (dav_resource *) 0x818d8a0
        depth = 2147483647
        err = (dav_error *) 0x0
        result = 0
        doc = (apr_xml_doc *) 0x818d8e0
        child = (const apr_xml_elem *) 0x0
        ctx = {w = {walk_type = 7, func = 0xb7bc0fba <dav_propfind_walker>, 
    walk_ctx = 0xb7328840, pool = 0x818c1e0, root = 0x818d8a0, 
    lockdb = 0x818da38}, bb = 0x818da18, scratchpool = 0x81a4890, 
  r = 0x818c218, doc = 0x818d8e0, propfind_type = 1, propstat_404 = 0x0, 
  if_header = 0x0, locktoken = 0x0, lock = 0x0, skip_root = 0, flags = 0, 
  work_buf = {alloc_len = 0, cur_len = 0, buf = 0x0}}
        multi_status = (dav_response *) 0x810c2f0
#19 0xb7bc8af8 in dav_handler (r=0x818c218) at mod_dav.c:4626
No locals.
#20 0x0806ceff in ap_invoke_handler (r=0x818c218) at config.c:152
        new_handler = 0x0
        p2 = 0x80a0587 "subrequest"
        handler = 0x0
        result = 0
        old_handler = 0xb7bcfb70 "dav-handler"
#21 0x080678fc in ap_process_request (r=0x818c218) at http_request.c:249
        access_status = 0
#22 0x08060f55 in ap_process_http_connection (c=0x81860a8) at http_core.c:253
        r = (request_rec *) 0x818c218
        csd_set = 0
        csd = (apr_socket_t *) 0x0
#23 0x08077aba in ap_process_connection (c=0x81860a8, csd=0x8185fb8)
    at connection.c:43
No locals.
#24 0x0806902e in worker_thread (thd=0x0, dummy=0x0) at worker.c:522
        process_slot = 0
        thread_slot = 0
        csd = (apr_socket_t *) 0x8185fb8
        bucket_alloc = (apr_bucket_alloc_t *) 0x818a1d8
        last_ptrans = (apr_pool_t *) 0x81860a8
        ptrans = (apr_pool_t *) 0x8185f80
        rv = 134874503
        is_idle = 0
#25 0xb7de1421 in dummy_worker (opaque=0x0) at thread.c:105
No locals.
#26 0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#27 0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 26 (Thread -1229816912 (LWP 7842)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb6b27ab4, p=0xb6b27ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 1
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 25 (Thread -1238209616 (LWP 7843)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb6326ab4, p=0xb6326ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 2
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 24 (Thread -1246602320 (LWP 7845)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb5b25ab4, p=0xb5b25ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 3
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 23 (Thread -1254995024 (LWP 7846)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb5324ab4, p=0xb5324ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 4
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 22 (Thread -1263387728 (LWP 7847)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb4b23ab4, p=0xb4b23ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 5
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 21 (Thread -1271780432 (LWP 7848)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb4322ab4, p=0xb4322ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 6
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 20 (Thread -1280173136 (LWP 7849)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb3b21ab4, p=0xb3b21ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 7
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 19 (Thread -1288565840 (LWP 7850)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb3320ab4, p=0xb3320ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 8
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 18 (Thread -1296958544 (LWP 7851)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb2b1fab4, p=0xb2b1fab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 9
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 17 (Thread -1305351248 (LWP 7852)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb231eab4, p=0xb231eab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 10
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 16 (Thread -1313743952 (LWP 7854)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb1b1dab4, p=0xb1b1dab8)
    at fdqueue.c:2        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 11
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 15 (Thread -1322136656 (LWP 7856)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()

   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb131cab4, p=0xb131cab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 12
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 14 (Thread -1330529360 (LWP 7858)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb0b1bab4, p=0xb0b1bab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 13
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 13 (Thread -1338922064 (LWP 7859)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xb031aab4, p=0xb031aab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 14
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 12 (Thread -1347314768 (LWP 7860)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xafb19ab4, p=0xafb19ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 15
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 11 (Thread -1355707472 (LWP 7861)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xaf318ab4, p=0xaf318ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 16
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 10 (Thread -1364100176 (LWP 7862)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xaeb17ab4, p=0xaeb17ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 17
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 9 (Thread -1372492880 (LWP 7863)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xae316ab4, p=0xae316ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 18
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 8 (Thread -1380885584 (LWP 7865)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xadb15ab4, p=0xadb15ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 19
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 7 (Thread -1389278288 (LWP 7887)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xad314ab4, p=0xad314ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 20
 in dummy_worker (opaque=0xfffffffc) at thread.c:105
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 6 (Thread -1397670992 (LWP 7888)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xacb13ab4, p=0xacb13ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 21
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 5 (Thread -1406063696 (LWP 7889)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xac312ab4, p=0xac312ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 22
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 4 (Thread -1414456400 (LWP 7890)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xabb11ab4, p=0xabb11ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 23
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 3 (Thread -1422849104 (LWP 7891)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d5440e in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7de4214 in apr_thread_cond_wait (cond=0xfffffffc, mutex=0x814efb8)
    at thread_cond.c:80
        rv = -4
#3  0x0806c12f in ap_queue_pop (queue=0x814efa0, sd=0xab310ab4, p=0xab310ab8)
    at fdqueue.c:258
        elem = (fd_queue_elem_t *) 0xfffffffc
        rv = -4
#4  0x08068ee7 in worker_thread (thd=0xfffffffc, dummy=0xfffffffc)
    at worker.c:809
        process_slot = 0
        thread_slot = 24
        csd = (apr_socket_t *) 0x0
        bucket_alloc = (apr_bucket_alloc_t *) 0x0
        last_ptrans = (apr_pool_t *) 0x0
        ptrans = (apr_pool_t *) 0x0
        rv = 0
        is_idle = 1
#5  0xb7de1421 in dummy_worker (opaque=0xfffffffc) at thread.c:105
No locals.
#6  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#7  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 2 (Thread -1431241808 (LWP 7892)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d56cd8 in accept () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0xb7ddf087 in apr_socket_accept (new=0xaab0f9e0, sock=0x80ad3e8, 
    connection_context=0x81881d0) at sockets.c:164
No locals.
#3  0xb7ddf89f in apr_accept (new=0xfffffe00, sock=0xfffffe00, 
    connection_context=0xfffffe00) at sockets.c:415
No locals.
#4  0x0808d687 in unixd_accept (accepted=0xaab0fa18, lr=0xfffffe00, 
    ptrans=0xfffffe00) at unixd.c:468
        csd = (apr_socket_t *) 0x8188208
        status = -1210142208
#5  0x080689e7 in listener_thread (thd=0x814f5a0, dummy=0x80ad380)
    at worker.c:698
        process_slot = 0
        tpool = (apr_pool_t *) 0xaab0fa18
        csd = (void *) 0x0
        ptrans = (apr_pool_t *) 0x81881d0
        n = 0
        pollset = (apr_pollfd_t *) 0x8105058
        rv = -1210142208
        lr = (ap_listen_rec *) 0x80ad380
        last_lr = (ap_listen_rec *) 0x80ad380
        have_idle_worker = 1
#6  0xb7de1421 in dummy_woNo locals.
#7  0xb7d51ca3 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#8  0xb7cecf5a in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.

Thread 1 (Thread -1212069088 (LWP 7837)):
#0  0xffffe410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d56abb in __read_nocancel () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0x0806c293 in ap_mpm_pod_check (pod=0xfffffe00) at pod.c:54
        c = -65 '¿'
        fd = 4
        rc = -512
#3  0x08069ede in child_main (child_num_arg=-1079495184) at worker.c:1195
        threads = (apr_thread_t **) 0x817f0d0
        ts = (thread_starter *) 0x814ef40
        thread_attr = (apr_threadattr_t *) 0x814ef50
        start_thread_id = (apr_thread_t *) 0x814ef80
#4  0x0806a1c4 in make_child (s=0x80b0e88, slot=0) at worker.c:1278
        pid = 0
#5  0x0806a3ef in ap_mpm_run (_pconf=0x2, plog=0x80db158, s=0x0)
 orker (opaque=0xfffffe00) at thread.c:105
    at worker.c:1312
        remaining_children_to_start = 0
        rv = -512
#6  0x0807286e in main (argc=3, argv=0xbfa83904) at main.c:623
        exit_status = 0
        c = 0 '\0'
        configtestonly = 0
        confname = 0x809b290 "conf/httpd.conf"
        def_server_root = 0x809c4c3 "/opt/gmx/apache2"
        temp_error_log = 0x0
        server_conf = (server_rec *) 0x80db158
        pglobal = (apr_pool_t *) 0x80ad0a0
        pconf = (apr_pool_t *) 0x80af0a8
        plog = (apr_pool_t *) 0x80db158
        ptemp = (apr_pool_t *) 0x80f41a0
        pcommands = (apr_pool_t *) 0x80b10b0
        opt = (apr_getopt_t *) 0x80b1148
        rv = -512
        mod = (module **) 0x80db158
        optarg = 0x8097b2b "\215\223¸þÿÿ\215\203¸þÿÿ)ÂÁú\0029Ös\034\211×\215´&"
        signal_server = (apr_OFN_ap_signal_server_t *) 0x1
0xb7de678b      1795    in apr_pools.c
Comment 5 Roger Bonati 2007-06-25 06:10:10 UTC 
Hello,

can someone see for this bug?

Thanks
Comment 6 Ruediger Pluem 2007-06-25 06:50:22 UTC 
(In reply to comment #5)
> Hello,
> 
> can someone see for this bug?
> 
> Thanks

Sorry for the late response. My findings so far do not offer anything real new
to you: You run out of memory. The question is: Why. Given Linux optimistic
memory allocation model I doubt that you really hit the wall here. Nevertheless
have you checked the memory consumption on this box during the crash? Did you
still have free physical memory?
Another option that seems to be more likely to me is that you have set an memory
limit via ulimit either for the httpd parent process or for the user that is
used to run the httpd children. It may be wise to also check
/etc/security/limits.conf.
Comment 7 Roger Bonati 2007-06-25 07:22:45 UTC 
Hello Ruediger,

there are no ulimts defined and the physical memory did not go out of limits
during the crash. 
A test with defining explicitly 'ulimit -a' before starting apache gave the same
result. 
But anyway, if a thread cannot allocate memory, malloc (or whatever function is
used) should be checked whether the operation was succesful. A
segmentation-fault is to me like a hint to a programm error or at least some
sign for "quick & dirty" hacked code.   

Maybe you can find the piece of code where the memory allocation is not checked
against the return value?

Thanks and greetings

Roger
Comment 8 Ruediger Pluem 2007-06-25 08:40:16 UTC 
(In reply to comment #7)
> Hello Ruediger,
> 
> there are no ulimts defined and the physical memory did not go out of limits
> during the crash. 
> A test with defining explicitly 'ulimit -a' before starting apache gave the same
> result. 

Mind to attach a longer part of the strace of a crashing process? That might
be helpful, but to be honest I have to admit that I am out of ideas.

> But anyway, if a thread cannot allocate memory, malloc (or whatever function is
> used) should be checked whether the operation was succesful. A
> segmentation-fault is to me like a hint to a programm error or at least some
> sign for "quick & dirty" hacked code.

There were long and controversial discussions regarding this on the dev lists of
apr and httpd. The final outcome was that it is better to crash in such situation as

1. there is usually nothing useful that can be done in such a case
2. it offers you the benefit of a core dump and a full backtrace.

You may not agree with this, but this is designed by purpose.
Comment 9 Preston A. Elder 2007-09-17 14:03:34 UTC 
I have a similar problem with mod_dav and mpm_worker or mpm_event.

In my case, I am using apache 2.2.6 and subversion 1.4.4.  I am on an amd64 box
(so no 32-bit memory limit) with 8gb RAM which I'm not even close to hitting. 
So I don't think I'm running out of memory.  I have tried adjusting
ThreadsPerChild below 10, no dice.

I will attach the output of "thread apply all bt full".  This does not happen
just by using SVN, but when I have a heavy SVN user (for example FishEye, which
has to retrieve pretty much all changes to create its change cache), it happens
frequently and reliably.
Comment 10 Preston A. Elder 2007-09-17 14:04:18 UTC 
Created attachment 20838 [details]
output from gdb for apache core
Comment 11 Preston A. Elder 2007-09-19 10:04:41 UTC 
This is more than likely a bug in subversion:

http://subversion.tigris.org/issues/show_bug.cgi?id=2935

More specifically, mod_dav_svn not being thread safe.
Comment 12 William A. Rowe Jr. 2018-11-07 21:08:16 UTC 
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.