Bug 39834

Summary: JkLogLevel debug causes Segmentation fault
Product: Tomcat Connectors Reporter: Ulf Moeller <mail>
Component: CommonAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: normal CC: sca
Priority: P3    
Version: unspecified   
Target Milestone: ---   
Hardware: Other   
OS: Linux   
Bug Depends on:    
Bug Blocks: 40316    

Description Ulf Moeller 2006-06-19 12:06:53 UTC
I am using mod_jk 1.2.15 with Apache 2.0.58 on x64 Linux.

When JkLogLevel is set to debug, "apachectl graceful" reproducably causes 
segmentation faults. Occasionally they occur at other times as well.

[Mon Jun 19 11:47:19 2006] [error] (43)Identifier removed: apr_global_mutex_lock
(jk_log_lock) failed
[Mon Jun 19 11:47:19 2006] [notice] child pid 17568 exit signal Segmentation 
fault (11)

The eror occurs in ap_log_rerror (), called from jk_log_to_file ().
Comment 1 Rainer Jung 2006-06-19 17:34:52 UTC
I fixed a similar problem in the forthcoming 1.2.16.

Could you please enable core dumping and get a backtrace via gdb, so that we can
check which jk method called jk_log_to_file? From your report I assume you
alreyd have a full stack?

Also: if you are able to build from SVN trunk, please do so and retry!

Does the problem occur wuth "apachectl restart" also?

Finally, could you please give some details on your Linux (kernel version) and
GCC version.
Comment 2 Ulf Moeller 2006-06-20 10:39:20 UTC
I tried the version from SVN. The problem still occurs:

#0  0x080a085e in ap_log_rerror ()
#1  0x558eeacc in jk_log_to_file (l=0x814bc58, level=1, 
    what=0xffff9390 "[Tue Jun 20 08:55:56 2006] [4415:24352] [debug] 
jk_cleanup_shmem::mod_jk.c (1758): Shmem cleanup\n") at mod_jk.c:2199
#2  0x558f6dda in jk_log (l=0x814bc58, file=0x5590db20 "mod_jk.c", line=1758, 
funcname=0x5590de7e "jk_cleanup_shmem", level=1, fmt=0x5590de70 "Shmem cleanup")
    at jk_util.c:372
#3  0x558ee9c9 in jk_cleanup_shmem (data=0x814bc58) at mod_jk.c:1758
#4  0x5574269f in run_cleanups () from /abcdef/lib/libapr-0.so.0
#5  0x55741bdc in apr_pool_destroy () from /abcdef/lib/libapr-0.so.0
#6  0x08096a9b in clean_child_exit ()
#7  0x080984c6 in child_main ()
#8  0x080985dc in make_child ()
#9  0x08098bc4 in perform_idle_server_maintenance ()
#10 0x08098e30 in server_main_loop ()
#11 0x080991dd in ap_mpm_run ()
#12 0x080a28c3 in main ()

"apachectl restart" doesn't cause the error.

GCC is version 3.3.3, Linux kernel 2.6.5.
Comment 3 Mladen Turk 2006-06-24 10:47:38 UTC
Fixed in the SVN.
The problem was caused by the ordering of apr_pool_t cleanup
calls. The simples solution is to remove the DEBUG logging
on shmem cleanup hook.
Comment 4 Rainer Jung 2006-06-27 20:56:03 UTC
*** Bug 39914 has been marked as a duplicate of this bug. ***
Comment 5 Yutaka Tanaka 2006-08-22 10:09:29 UTC
hi

We are now facing the same problem.
We are using mod_jk 1.2.18 with Apache 2.0.58 in RHELv3.

When JkLogLevel is set to error (not only debug), "apachectl graceful" causes 
segmentation faults. 
Did the problem be fixed already?

Could you give us any idea to solve this problem?

- Reproduction steps
 1. making a rush with 10 multiple users
 2. # mv access_log access_log.1
    (we'd like to rotate the log file when we reboot Apache(graceful).)
 3. # apachectl graceful


- error_log
[Tue Aug 22 15:35:06 2006] [notice] Graceful restart requested, doing restart  
httpd: Could not determine the server's fully qualified domain name, using 127.
.0.1 for ServerName                                                            
[Tue Aug 22 15:35:06 2006] [notice] Apache configured -- resuming normal operat
ons                                                                            
[Tue Aug 22 15:35:09 2006] [error] (43)Identifier removed: apr_global_mutex_loc
(jk_log_lock) failed                                                           
[Tue Aug 22 15:35:09 2006] [error] (43)Identifier removed: apr_global_mutex_loc
(jk_log_lock) failed                                                           
[Tue Aug 22 15:35:09 2006] [error] (43)Identifier removed: apr_global_mutex_loc
(jk_log_lock) failed                                                           
[Tue Aug 22 15:35:09 2006] [error] (43)Identifier removed: apr_global_mutex_loc
(jk_log_lock) failed                                                           
[Tue Aug 22 15:35:10 2006] [notice] child pid 1958 exit signal Segmentation fau
t (11)                                                                         
[Tue Aug 22 15:35:10 2006] [notice] child pid 1960 exit signal Segmentation fau
t (11)                                                                         
[Tue Aug 22 15:35:10 2006] [notice] child pid 1962 exit signal Segmentation fau
t (11)                                                                         
[Tue Aug 22 15:35:10 2006] [notice] child pid 1963 exit signal Segmentation fau
t (11)                                                                         
Comment 6 Rainer Jung 2006-08-22 20:53:18 UTC
I could not reproduce the segfaults with mod_jk 1.2.18/httpd 2.0.58:

Setting JkLogLevel to error and stressing apache with requests going to tomcat 
and sleeping there for 50 ms with a parallelity of 20 did not produce any 
problem during "apachectl graceful".

When I increased JkLogLevel to debug, I could reproduce the apr_global_mutex_* 
messages, but still *not* the segfaults.

Platform was Linux x86_64 (SLES 9 SP2).

Do you use piped logs for mod_jk? Is it possible, that the reason for the 
segfaults is another module? Are you loading any unusual modules?

In case you want to discuss this further, please open another bugzilla.
Comment 7 Takayoshi Kimura 2006-08-23 01:41:36 UTC
I have tried to reproduce the graceful segv, but cannot reproduce too.
Platform is: RHEL4U3, kernel-2.6.9-34.EL, httpd-2.0.52-22.ent, mod_jk 1.2.18.
Comment 8 Yutaka Tanaka 2006-08-23 10:00:45 UTC
Thank you for your quick reply.

> In case you want to discuss this further, please open another bugzilla.
I'll open another bugzilla if i can't resolve this problem after some our tests.

thansks,
Comment 9 Daniel Rall 2006-11-09 18:18:17 UTC
r417586 and r416898 seem to apply to this issue.