Bug 7838 - Apache leaves shared memory segments and dies after unclean shutdown
Summary: Apache leaves shared memory segments and dies after unclean shutdown
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: Core (show other bugs)
Version: 2.0.35
Hardware: All All
: P3 major with 4 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Depends on:
Reported: 2002-04-08 14:17 UTC by Stanislav Malyshev
Modified: 2004-11-16 19:05 UTC (History)
2 users (show)


Note You need to log in before you can comment on or make changes to this bug.
Description Stanislav Malyshev 2002-04-08 14:17:58 UTC
If Apache server is killed by signal, it leaves shared memory segment (for me
with ID 0x0103295c) which is not cleaned up.

Also, if Apache is then restarted, it gives the following error:
[Mon Apr 08 17:07:03 2002] [crit] (17)File exists: unable to create scoreboard
(name-based shared memory failure)

and dies with segmentation fault with the following trace:

#0  0x4018d109 in chunk_free (ar_ptr=0x40221cc0, p=0x80f0c78) at malloc.c:3111
#1  0x4018cf9a in __libc_free (mem=0x80f0c80) at malloc.c:3023
#2  0x4003dd8a in apr_allocator_destroy (allocator=0x80a9430)
    at apr_pools.c:135
#3  0x4003e391 in apr_pool_destroy (pool=0x80a94a8) at apr_pools.c:687
#4  0x4003deeb in apr_pool_terminate () at apr_pools.c:513
#5  0x4003a3f5 in apr_terminate () at start.c:112
#6  0x807cb22 in destroy_and_exit_process (process=0x80ad538, 
    process_exit_value=0) at main.c:245
#7  0x807d42c in main (argc=4, argv=0xbffff6a4) at main.c:629
#8  0x4014b9cb in __libc_start_main (main=0x807ce48 <main>, argc=4, 
    argv=0xbffff6a4, init=0x805eaa0 <_init>, fini=0x80920fc <_fini>, 
    rtld_fini=0x4000aea0 <_dl_fini>, stack_end=0xbffff69c)
    at ../sysdeps/generic/libc-start.c:92

Note also:

#1  0x4018cf9a in __libc_free (mem=0x80f0c80) at malloc.c:3023
in malloc.c
#2  0x4003dd8a in apr_allocator_destroy (allocator=0x80a9430)
    at apr_pools.c:135
(gdb) p node
$1 = (apr_memnode_t *) 0x0

I.e., it does free to NULL. 

The source is httpd-2.0.35 release, with prefork MPM.
Comment 1 Erich Prchal 2002-04-08 18:02:41 UTC
I had the same problem, truss is:

from: scoreboard.c, create_namebased_scoreboard:
9935:   unlink("/usr/local/apache2/logs/apache_runtime_status") Err#2 ENOENT

from: create_namebased_scoreboard, 
else path of filename == NULL, #if APR_USE_SHMEM_SHMGET
9935:   open("/usr/local/apache2/logs/apache_runtime_status",
 O_WRONLY|O_CREAT|O_EXCL, 0666) = 29
9935:   stat64("/usr/local/apache2/logs/apache_runtime_status", 0xFFBEF8A0) = 0
9935:   shmget(16778794, 196888, 0600|IPC_CREAT|IPC_EXCL) Err#17 EEXIST

The following change helped:
remove the IPC_EXCL in:
        if ((new_m->shmid = shmget(shmkey, new_m->realsize,
                                   SHM_R | SHM_W | IPC_CREAT | IPC_EXCL )) < 0) 

The scoreboard file was just created with  APR_CREATE | APR_EXCL
so I think this should be safe.

A clean solution would be a function in shm.c which does a cleanup if only the 
is know and use this instead of the apr_file_remove in scoreboard.c

Comment 2 Stanislav Malyshev 2002-04-09 09:29:15 UTC
I guess the solution may be to add shmctl(new_m->shmid, IPC_RMID, NULL) just
after shmat, like all other shmem-using tools do. Thus pre-removing shared
memory segment and making it be released as soon as all processes detach from
it. Am I missing something important here?
Comment 3 Paul J Murphy 2002-04-13 01:03:27 UTC
I have also enountered this bug on Solaris 8/SPARC.  A workaround on that 
platform which allows Apache to be restarted without rebooting is to 
use /usr/bin/ipcs to find the ID of the shared memory and /usr/bin/ipcrm to 
remove it.

Accordingly, I have changed the Platform/OS of this bug from PC/Linux to 
All/All and upped the Severity to Major as this is the sort of bug which might 
cause less experienced sysadmins to reboot when it's not really necessary.
Comment 4 Justin Erenkrantz 2002-04-15 05:41:04 UTC
This is one of the main drawbacks of using System V SHM.  Try using AcceptMutex
pthread if your platform supports it (Solaris certainly does).  You may also try
"flock" or "fcntl" if your OS supports those values.  We can attempt to try to
clean up certain shutdown paths, but I'm not sure we can catch everything.

The reason we do not want to delete the segment on initialization is that doing
so prevents third-party modules (such as mod_perl which relies on this, AIUI)
from attaching to the scoreboard externally.

Also fix typo in bug summary.
Comment 5 Matthew Darwin 2002-04-16 00:13:41 UTC
[adding myself to CC]
Comment 6 Matthew Darwin 2002-04-19 01:09:44 UTC
I am running the following procedure before starting the web server, but
the web server doesn't always start... It still gives an error.  What am 
I doing wrong?  Linux 2.4.18

       [Thu Apr 18 16:03:40 2002] [crit] (17)File exists: unable to create
       (name-based shared memory failure)

sub clean_shared_memory {
        if (! open (PROC, "<", "/proc/sysvipc/shm")) {
                warn "Cannot open proc filesystem: $!\n";

        # key shmid perms size cpid lpid nattch uid gid cuid cgid atime dtime ctime
        while (<PROC>) {
                my ($shmid, $uid) = (split (/\s+/))[2,8];
                next if ($uid ne "405");
                if (! shmctl($shmid, IPC_RMID, 0)) {
                        warn "Cannot remove shared memory segment=<$shmid>: $!\n";
                print "Removed shared memory segment=<$shmid>\n";
        close (PROC);
Comment 7 Aaron Bannert 2002-04-25 15:38:08 UTC
A few very good proposals have been made here, but unfortunately they are

1) calling shmctl(new_m->shmid, IPC_RMID, NULL) right after calling shmat()
  -- This won't work because segments that are marked IPC_RMID can not be
     attached to in child processes.

2) calling shmctl($shmid, IPC_RMID, 0) before creating a new segment
  -- This is also problematic because this would allow a second instance
     of apache to remove the segment of another already-running apache.

It is my understanding that the normal shutdown signals will call the
cleanup routines and properly detach and remove the segment. If you are
killing apache with SIGKILL then you're going to have to use ipcs/ipcrm
to clean up those segments. For the above reasons I'm marking this bug as
invalid, if you disagree please comment.
Comment 8 Matthew Darwin 2002-04-26 01:43:48 UTC
What is the proper way to kill apache, having it NOT finish servicing any
requests, but die immediately and exit cleanly?

This is what I do now, which apparently leaves shared memory around

if (-e /var/run/httpd.pid) {
        kill -TERM `cat /var/run/httpd.pid`
        rm /var/run/httpd.pid

sleep 2

killall -9 httpd
Comment 9 André Malo 2003-02-26 01:05:29 UTC
The proper way is to use httpd -k stop or apachectl -k stop. This should run all
the cleanup stuff and shutdown gently.