Bug 59184

Summary: mod_jk segmentation fault at jk_shm.c:875
Product: Tomcat Connectors Reporter: erik.hermans
Component: mod_jkAssignee: Tomcat Developers Mailing List <dev>
Status: RESOLVED FIXED    
Severity: normal    
Priority: P2    
Version: 1.2.41   
Target Milestone: ---   
Hardware: Other   
OS: Linux   
Attachments: error_log
httpd.conf
jk.log
mod_jk.conf
workers.properties
Fall back to process local memory to avoid crash when shm init failed

Description erik.hermans 2016-03-15 15:07:02 UTC 
Tried to install mod_jk 1.2.41 on Red Hat Enterprise Linux Server release 6.7 (Santiago)

Installation was done as follow

./configure --with-apxs=/usr/sbin/apxs
LDFLAGS=-lc ./configure -with-apxs=/usr/sbin/apxs
make
make install


When accessing the URL that should connect through mod_jk with tomcat

mod_jk give Segmentation 

Inspecting the core with gdb

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f66b841ac1d in jk_shm_check_maintain (trigger=1458052643) at jk_shm.c:875
875         int maintain_checking = JK_ATOMIC_INCREMENT(&(jk_shmem.hdr->h.data.maintain_checking));
Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-47.el6_7.1.x86_64


The hdr element ik jk_shmem is null

(gdb) p jk_shmem.hdr
$1 = (jk_shm_header_t *) 0x0


(gdb) bt
#0  0x00007f66b841ac1d in jk_shm_check_maintain (trigger=1458052643) at jk_shm.c:875
#1  0x00007f66b8409a22 in wc_maintain (l=0x7f66c635b8f8) at jk_worker.c:348
#2  0x00007f66b83fa36b in jk_handler (r=0x7f66c65a2c28) at mod_jk.c:2880
#3  0x00007f66c54c4e50 in ap_run_handler ()
#4  0x00007f66c54c870e in ap_invoke_handler ()
#5  0x00007f66c54d3e00 in ap_process_request ()
#6  0x00007f66c54d0c48 in ?? ()
#7  0x00007f66c54cc958 in ap_run_process_connection ()
#8  0x00007f66c54d8b97 in ?? ()
#9  0x00007f66c54d8eaa in ?? ()
#10 0x00007f66c54d91db in ap_mpm_run ()
#11 0x00007f66c54b0a30 in main ()
Comment 1 Rainer Jung 2016-03-15 16:34:17 UTC 
Hi,

thanks for the report and looking at the core using the debugger.

Is this reproducible? Are you starting under load or does it also happen when starting during an idle time? Any warnings or errors in the mod_jk log file during atartup or between startup and the crash?

Would it be possible to reproduce with "JkLogLevel debug" and provide access to the logs and you config (Jk dircetives from httpd and workers.properties, plus info about vhost structure)? That will produce loads of log output though, so one would typically only do it on a test system.

If some of that info is not supposed to be available in the open, you might send that via private mail to me.

Regards,

Rainer
Comment 2 erik.hermans 2016-03-15 16:56:12 UTC 
Created attachment 33676 [details]
error_log
Comment 3 erik.hermans 2016-03-15 16:56:30 UTC 
Created attachment 33677 [details]
httpd.conf
Comment 4 erik.hermans 2016-03-15 16:56:58 UTC 
Created attachment 33678 [details]
jk.log
Comment 5 erik.hermans 2016-03-15 16:57:30 UTC 
Created attachment 33679 [details]
mod_jk.conf
Comment 6 erik.hermans 2016-03-15 16:57:54 UTC 
Created attachment 33680 [details]
workers.properties
Comment 7 erik.hermans 2016-03-15 16:59:55 UTC 
The problem is reproducable.

It also happens when starting during an idle time

JkLogLevel debug was already set.

You will find conf files and logs attached
Comment 8 Rainer Jung 2016-03-15 18:13:05 UTC 
There's an error during initializing shm:

[Tue Mar 15 15:38:00.076 2016] [11496:140079373203424] [error] init_jk::mod_jk.c (3574): Initializing shm:/etc/httpd/logs/jk-runtime-status.11496 errno=13. Load balancing workers will not function properly.

errno 13 on Red Hat is "Permission denied".

Since you have not configure a JkShmFile, mod_jk had chosen a path, which in your case was a file in the directory /etc/httpd/logs/. Most likely /etc/httpd is your Apache ServerRoot. mod_jk appends "logs" and tries to open the shared memory in that directory. If that isn't what fits your installation, you should explicitely choose a JkShmFile.

See also http://tomcat.apache.org/connectors-doc/reference/apache.html.

I'll keep that ticket open, because we should fail more gracefully than producing a core dump. But your solution would be to configure a correct JkShmFile.

Please let us know if that fixes your crashes.
Comment 9 erik.hermans 2016-03-17 17:03:26 UTC 
I configured  JkShmFile as you suggested, and now it works.

Would indeed be nice if mod_jk would fail more gracefully.
Comment 10 Rainer Jung 2016-03-20 11:20:34 UTC 
Created attachment 33685 [details]
Fall back to process local memory to avoid crash when shm init failed

This patch should prevent the crash, when shared memory initialization fails. Since IMHO it is not a good idea to use the module in this case, I wonder whether it would actually be better to fail during startup in this case and not let users proceed.

Currently the failure only affects setups using a load balancer worker. But this could change any time and even configs without farming should always add an lb worker above their ajp workers.

Anyone having an opinion, please respond here. If I hear no objections, I will likely choose to let the module fail the web server startup if it can't init its shared memory (and not apply the patch).
Comment 11 Felix Schumacher 2016-03-20 11:26:18 UTC 
+1 for fail early (with a proper error message, of course)
Comment 12 Mark Thomas 2016-09-19 10:24:17 UTC 
This has been fixed in trunk and will be included in 1.2.42 onwards.