Bug 41142 - Endless loop in apr_allocator_destroy
Summary: Endless loop in apr_allocator_destroy
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_dbd (show other bugs)
Version: 2.2.3
Hardware: Sun Solaris
: P1 blocker (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: PatchAvailable
Depends on:
Blocks:
 
Reported: 2006-12-10 04:34 UTC by Jens Hamisch
Modified: 2007-10-07 18:12 UTC (History)
2 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jens Hamisch 2006-12-10 04:34:55 UTC
The function apr_allocator_destroy removes pool nodes
from memory. Those nodes are organized as a chain in which 
each node points to the subsequent one using its 'next'
member.

If mod_dbd in conjunction with apr_dbd_pgsql is in use it
happens, that the 'next' pointer of a node points to the
node itsself. This introduces an endless loop consuming 100%
of CPU.

The problem IMHO is introduced into the chain of nodes at
some other location. However as a workaround I've applied the
following patch to the destroy function to become aware of
the problem and treat it correctly:

*** ./srclib/apr/memory/unix/apr_pools.c.fcs    Sun Dec 10 12:44:28 2006
--- ./srclib/apr/memory/unix/apr_pools.c        Sun Dec 10 12:45:52 2006
***************
*** 107,112 ****
--- 107,115 ----
      for (index = 0; index < MAX_INDEX; index++) {
          ref = &allocator->free[index];
          while ((node = *ref) != NULL) {
+           /* jens 2006/12/10 - this should not happen, however it intro-
+              duced endless loops when apr_dbd_pgsql is in use */
+           if( node->next == node) node->next = NULL;
              *ref = node->next;
              free(node);
          }

This problem is a blocking point for Apache utilizing PostgreSQL
as authentication provider.
Comment 1 Ruediger Pluem 2006-12-10 04:46:55 UTC
Reclassify as APR bug
Comment 2 Jens Hamisch 2006-12-10 23:51:59 UTC
The patch proposed is a very quick workaround. The problem is 
introduced somewhere else. The patch prevents from endless loops, but
it may result in memory leaks - if the chain looops in the middle, not
at the end - anyway
Comment 3 Joe Orton 2006-12-11 02:49:39 UTC
That is most likely to be the result of some heap corruption bug or pool abuse;
it's certainly not correct to silently ignore such a loop in APR.

You could try standard tricks to narrow down the cause of the corruption (e.g.
building --enable-pool-debug) if this is not a known issue in mod_dbd.
Comment 4 Jens Hamisch 2006-12-11 04:17:21 UTC
Hi,

this ist a *BUG*!

Apache 2.2.3 is no usable at all if mod_dbd with apr_dbd_pgsql
is configured.

I aggree, that the patch I've provided is a workaround, however
it works for me and I do not have the ressources to play around
with some debugging voodoos this time. It already took me more
that one week to drag it down as far as I did!

I'd offer my help but if you insist in closing this issue without
analyzing or even forward it to people more familiar with mod_dbd
and apr_dbd_pgsql, then leave this buggy apache as it is!

Jens
Comment 5 Joe Orton 2006-12-11 06:50:57 UTC
Lacking any further reproduction information beyond "mod_dbd causes heap
corruption" there is not much point keeping the bug open.  There is certainly no
bug in APR described here.

If the symptoms of your bug match bug 39985 could you try the patch Chris posted
there?

Otherwise could you describe how and when mod_dbd is failing, how it was
configured, how you built the server, etc?
Comment 6 Joe Orton 2006-12-11 07:03:36 UTC
Please report problems with libapreq to apreq-dev@httpd.apache.org; I've
forwarded this message.
Comment 7 Joe Orton 2006-12-11 07:04:23 UTC
Blerg, sorry, wrong bug.
Comment 8 Jens Hamisch 2006-12-12 13:32:29 UTC
Hi,

I tried the patch posted in issue 39985 but it did not help.
I've installed the source tree and the development environment
on one of my public servers. I may offer to drag this down
together ...

Jens
Comment 9 hans-peter 2007-03-13 05:21:02 UTC
I had a simmilar Problem with apr when destroying a pool twice.
(Call apr_pool_destroy twice on the same pool).

This causes the allocators "free" list to be corrupted.

Next time, you call 
apr_pool_create(newpool,NULL)
apr_pool_create(newpool2,newpool) you get 2 identical nodes, where
parent and child seem to be self-references.

in allocator_destroy (called by apr_terminate) this leads to an endless loop.

Maybe it would help to enhance the apr-functions with debug output.
Comment 10 Randy Jones 2007-08-16 16:44:58 UTC
This is a me too on this issue. This is a problem in apache 2.2.3 and 2.2.4.
I have also "fixed" the problem by hacking up apr_allocator_destroy(). However
my hack is ugly compared to Jens' modification. My environment is as
follows:
Solaris 10 (sparc)
Sun Studio 11 compiler
Apache 2.2.4 (also had problem with 2.2.3)
php 5.2.3
mysql 5.0.45
apr 1.2.9 (I replaced the apr in srclib with 1.2.9)
apr_dbd_mysql.c from http://apache.webthing.com/database

apr is configured with:
configure --prefix=/usr/local/apache2/apr

apr-util is configured with:
configure --with-mysql=/opt/mysql/mysql --with-apr=/usr/local/apache2/apr -
-prefix=/usr/local/apache2/apr-util --with-expat=builtin

apache is configured with:
configure --enable-ssl --with-ssl=/usr/sfw --with-apr=/usr/local/apache2/ap
r --with-apr-util=/usr/local/apache2/apr-util --enable-authn-dbd

apr, apr-util, and apache all configure, compile, and install nicely.

However a few minutes ( 1 -5 minutes) after startup of apache the
httpd process are all consuming cpu and are stuck in an endless loop
in apr_allocator_destroy(). With my "fixed" apr_allocator_destroy()
function the problem is "fixed".


Below is my ugly modified apr_allocator_destroy() function.
This one happens to be from apr 1.2.9.

APR_DECLARE(void) apr_allocator_destroy(apr_allocator_t *allocator)
{
    apr_uint32_t index;
    apr_memnode_t *node, **ref;
    apr_memnode_t *old1, *old2;/*rsj 08/25/2006 add this to prevent spinning below*/

    for (index = 0; index < MAX_INDEX; index++) {
        ref = &allocator->free[index];
        while ((node = *ref) != NULL) {
            *ref = node->next;
            /*
                rsj 08/25/2006
                The next 4 lines of code are added by me.
                The purpose is to keep from trying to free the same node
                over and over and over...
                This is what was happening when I attached dbx to an httpd
                process that was using 100% cpu.
                If old == *ref then we have the problem. That is why the if
                statement sets node->next to NULL because the original
                node->next points to its self.
            */
            if(old1 == *ref) node->next = NULL;/*rsj 08/25/2006*/
            if(old2 == *ref) node->next = NULL;/*rsj 08/25/2006*/
            old2 = old1;/*rsj 08/25/2006*/
            old1 = *ref;/*rsj 08/25/2006*/

            free(node);
        }
    }

    free(allocator);
}

Comment 11 Nick Kew 2007-09-07 17:52:04 UTC
Do you still have a problem with 2.2.6?
Comment 12 Randy Jones 2007-09-28 14:36:58 UTC
Apache 2.2.6 has fixed this bug for me!!! I have now been running 2.2.6 for
several days now without a problem.

Thanks a bazillion goes to whoever fixed this issue.
Randy Jones
Comment 13 Nick Kew 2007-10-07 18:12:56 UTC
Believed obsoleted.  Reported as fixed.