Bug 58626 - Tomcat does not start at boot time due to SIGHUP
Summary: Tomcat does not start at boot time due to SIGHUP
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 6
Classification: Unclassified
Component: Native:Integration (show other bugs)
Version: 6.0.44
Hardware: HP HP-UX
: P2 enhancement (vote)
Target Milestone: default
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-11-19 15:22 UTC by Michael Osipov
Modified: 2016-06-17 10:33 UTC (History)
0 users



Attachments
Proposed patch for Tomcat 9.0.x, version 1 (2.27 KB, patch)
2016-05-24 19:47 UTC, Mark Thomas
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Osipov 2015-11-19 15:22:38 UTC
When hooking Tomcat into the init.d of HP-UX, Tomcat does not come up at system start time. After an in-depth analysis of the startup scripts and HP-provided packages, we have noticed that the HP-UX init sends a SIGHUP to the startup script and child processes. Java does exit immediately.

One simple solution to this is to use nohup(1). Therefore, I'd like to propose to change the catalina.sh for "start" only which turns:
"$_RUNJAVA" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS
into
$JAVA_WRAPPER "$_RUNJAVA" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS

Not quoting JAVA_WRAPPER is intentional because those who do not need a wrapper, the script won't fail on the empty argument.

Patches can be provided for Tomcat 6 and up.
Comment 1 Michael Osipov 2015-11-19 15:25:21 UTC
A reference for this problem can be found here: https://groups.google.com/forum/#!topic/comp.sys.hp.hpux/HWBQQNEXfEE
Comment 2 Christopher Schultz 2015-11-19 15:57:03 UTC
I'm +0 on this idea, but failing to quote is going to be a problem.

Instead of using an unquoted value, you should use if/else to either run the command with the wrapper, or run it without the wrapper.
Comment 3 Mark Thomas 2015-11-19 16:05:12 UTC
Given the length of time other HP-UX users have been running Tomcat without this issue this looks more like an HP-UX configuration / usage error than something that needs to be fixed in the Tomcat scripts.
Comment 4 Konstantin Kolinko 2015-11-19 16:47:10 UTC
You should be able to patch the script that calls catalina.sh to wrap that call with a nohup. Such a recipe is already mentioned in the FAQ

https://wiki.apache.org/tomcat/TomcatOnSolaris10
https://wiki.apache.org/tomcat/FAQ/Linux_Unix#Q3


It is also possible to consider switching from "catalina.sh start" to "catalina.sh run".


In general, this is interesting. With nohup I have to redirect stdout and stderr, or it will create its own nohup.out.  Here the redirection is already on the command line, so it writes to catalina.out without a need for separate log file.


Note that there is also bug 53930 (allowing to replace catalina.out with a pipe) that also intends to change the launch command. I think it is not committed yet, because it is too tricky.

> which turns:
> "$_RUNJAVA" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS
> into
> $JAVA_WRAPPER "$_RUNJAVA" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS

1. The above line is from Tomcat 6. It does not match catalina.sh in current trunk which uses "eval".

2. There are more that one variant of "start" command. There are different commands with Security Manager enabled ("-security") and without one.
Comment 5 Michael Osipov 2015-11-19 19:04:36 UTC
(In reply to Christopher Schultz from comment #2)
> I'm +0 on this idea, but failing to quote is going to be a problem.
> 
> Instead of using an unquoted value, you should use if/else to either run the
> command with the wrapper, or run it without the wrapper.

Having an if/else around would duplicate 15 lines for nothing. Unnecessary maintenance overhead.

(In reply to Mark Thomas from comment #3)
> Given the length of time other HP-UX users have been running Tomcat without
> this issue this looks more like an HP-UX configuration / usage error than
> something that needs to be fixed in the Tomcat scripts.

I'd be glad to see a better solution but there is not better way -- yet. Even HP does this, so it must be a OS-specific issue. I did not say that the Tomcat scripts are broken. We just need to broaden support. OS X, OS/400 and Cygwin get special treatment too.

(In reply to Konstantin Kolinko from comment #4)
> You should be able to patch the script that calls catalina.sh to wrap that
> call with a nohup. Such a recipe is already mentioned in the FAQ

Not an option because "stop" does not work anymore.
 
> https://wiki.apache.org/tomcat/TomcatOnSolaris10
> https://wiki.apache.org/tomcat/FAQ/Linux_Unix#Q3

Tomcat 5.5? This is ages old. Nothing is worse that outdated information.

> It is also possible to consider switching from "catalina.sh start" to
> "catalina.sh run".

Not an option becuase it does not detach for the shell script and stdout/stderr go not to catalina.out.

> In general, this is interesting. With nohup I have to redirect stdout and
> stderr, or it will create its own nohup.out.  Here the redirection is
> already on the command line, so it writes to catalina.out without a need for
> separate log file.
> 
> 
> Note that there is also bug 53930 (allowing to replace catalina.out with a
> pipe) that also intends to change the launch command. I think it is not
> committed yet, because it is too tricky.
> 
> > which turns:
> > "$_RUNJAVA" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS
> > into
> > $JAVA_WRAPPER "$_RUNJAVA" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS
> 
> 1. The above line is from Tomcat 6. It does not match catalina.sh in current
> trunk which uses "eval".

I am stuck for 6.x current but would be happy to see this fixed in 8.x and onwards.

> 2. There are more that one variant of "start" command. There are different
> commands with Security Manager enabled ("-security") and without one.

My patch suggests patching both codepaths.
Comment 6 Michael Osipov 2015-11-19 19:06:44 UTC
I do not intend to write another wrapper for a wrapper for a wrapper. This ends in a mess. Given that there are already so many shell scripts involved in the start of Tomcat, I'd like to avoid any further customization of standard scripts. That ends in a private fork which I'd like to avoid at any cost. Tomcat runs perfectly fine on HP-UX on the HP JVM for years now.
Comment 7 Michael Osipov 2015-11-25 15:01:26 UTC
No further comments?
Comment 8 Konstantin Kolinko 2015-11-25 19:02:56 UTC
(In reply to Michael Osipov from comment #5)
>  
> > https://wiki.apache.org/tomcat/TomcatOnSolaris10
> > https://wiki.apache.org/tomcat/FAQ/Linux_Unix#Q3
> 
> Tomcat 5.5? This is ages old. Nothing is worse that outdated information.
> 

1. That information is not outdated. It applies to current versions as well. The issue is with O/S, not with Tomcat.

2. You know that the wiki is editable? You can update it yourself.

> Not an option because "stop" does not work anymore.
>

Nobody ever complained about stop. You do not call "stop" during boot time.

> My patch suggests patching both codepaths.
>

You have provided no patch file. I cannot read one's mind.


> I am stuck for 6.x current but would be happy to see this fixed in 8.x
and onwards.
>

Such a new feature has to be implemented and tested with Tomcat 9 first. Then it can be backported to older versions.

In theory, catalina.sh of Tomcat 9 can be used for Tomcat 6 as well. Is it true? (I mean, is it safe to backport all improvements that have not been backported yet, align Tomcat 6 file with trunk one)?

In general: the idea is interesting, but
1) I prefer a solution that is compatible with proposal in bug 53930
2) It needs testing, especially on rare systems
3) Your proposal does not match the current code in trunk, thus my lack of confidence in your words and your testing.

An explicit patch file is needed so that the discussion were more focused.


> When hooking Tomcat into the init.d

Usually there is a wrapper script. You do not hook-in catalina.sh directly. You do not run it with root user.
Comment 9 Christopher Schultz 2015-11-30 15:56:12 UTC
Michael, can you provide a reference that says that HP-UX sends a SIGHUP to processes after init runs? I found a thread on unix.stackexchange that seemed related, but someone commented that sending SIGHUP after init sounded crazy, even for HP-UX.

http://unix.stackexchange.com/a/21323/68318
Comment 10 Michael Osipov 2015-11-30 16:45:54 UTC
(In reply to Christopher Schultz from comment #9)
> Michael, can you provide a reference that says that HP-UX sends a SIGHUP to
> processes after init runs? I found a thread on unix.stackexchange that
> seemed related, but someone commented that sending SIGHUP after init sounded
> crazy, even for HP-UX.
> 
> http://unix.stackexchange.com/a/21323/68318

HP's documentation is lousy though, I can upload the original /sbin/init.d/tomcat script and their modified version of catalina.sh. Is that enough for you? I have provided another reference in comment 1.
Comment 11 Christopher Schultz 2015-11-30 18:10:11 UTC
(In reply to Michael Osipov from comment #10)
> (In reply to Christopher Schultz from comment #9)
> > Michael, can you provide a reference that says that HP-UX sends a SIGHUP to
> > processes after init runs? I found a thread on unix.stackexchange that
> > seemed related, but someone commented that sending SIGHUP after init sounded
> > crazy, even for HP-UX.
> > 
> > http://unix.stackexchange.com/a/21323/68318
> 
> HP's documentation is lousy though, I can upload the original
> /sbin/init.d/tomcat script and their modified version of catalina.sh. Is
> that enough for you? I have provided another reference in comment 1.

How about a diff from the original catalina.sh?
Comment 12 Konstantin Kolinko 2015-11-30 18:12:34 UTC
(In reply to Michael Osipov from comment #10)

What is the license for those scripts?
Comment 13 Michael Osipov 2015-11-30 21:51:00 UTC
(In reply to Christopher Schultz from comment #11)
> (In reply to Michael Osipov from comment #10)
> > (In reply to Christopher Schultz from comment #9)
> > > Michael, can you provide a reference that says that HP-UX sends a SIGHUP to
> > > processes after init runs? I found a thread on unix.stackexchange that
> > > seemed related, but someone commented that sending SIGHUP after init sounded
> > > crazy, even for HP-UX.
> > > 
> > > http://unix.stackexchange.com/a/21323/68318
> > 
> > HP's documentation is lousy though, I can upload the original
> > /sbin/init.d/tomcat script and their modified version of catalina.sh. Is
> > that enough for you? I have provided another reference in comment 1.
> 
> How about a diff from the original catalina.sh?

No problem, you will get one on Wednesday.


(In reply to Konstantin Kolinko from comment #12)
> (In reply to Michael Osipov from comment #10)
> 
> What is the license for those scripts?

Absolutely no idea, this (https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=HPUXWST603201) is the package I used as a template for the modified start scripts. The rest is unusable and too old. I use an Ant script which downloads Tomcat from Nexus, unpack, compile Tomcat Native, download additional libraries etc.
Comment 14 Konstantin Kolinko 2015-11-30 23:01:10 UTC
(In reply to Michael Osipov from comment #13)

If you do not know what the license is, I prefer you not to show us that code.
Comment 15 Michael Osipov 2015-11-30 23:34:17 UTC
(In reply to Konstantin Kolinko from comment #14)
> (In reply to Michael Osipov from comment #13)
> 
> If you do not know what the license is, I prefer you not to show us that
> code.

I can share in private, the relevant spots. Besides, I am not the only ASF member having access to a HP-UX box, Mladen Turk has also (the last time he wrote). Do we want to involve him?
Comment 16 Mark Thomas 2016-05-24 19:47:10 UTC
Created attachment 33883 [details]
Proposed patch for Tomcat 9.0.x, version 1

I don't have an HP-UX box to test with but the described behaviour is odd to say the least.

There are other reasons why one might want to start with nohup so I have put together a slightly more general solution using some of the ideas proposed above. The short version is nohup on start is available for all operating systems and HP-UX will use it by default.

Feedback on the atached patch appreciated - particularly from anyone with access to an HP-UX box.
Comment 17 Michael Osipov 2016-05-24 21:27:50 UTC
(In reply to Mark Thomas from comment #16)
> Created attachment 33883 [details]
> Proposed patch for Tomcat 9.0.x, version 1
> 
> I don't have an HP-UX box to test with but the described behaviour is odd to
> say the least.
> 
> There are other reasons why one might want to start with nohup so I have put
> together a slightly more general solution using some of the ideas proposed
> above. The short version is nohup on start is available for all operating
> systems and HP-UX will use it by default.
> 
> Feedback on the atached patch appreciated - particularly from anyone with
> access to an HP-UX box.

Quite a nice solution. Line 274 has too many spaces in it.
I will test that next week in a HP-UX 11.31 and will let you know. Are you going to backport this to 8? We are currently running and planning to move to 8.0.

For this case, I will download 9.0.0.M6 and apply the patch to it.
Comment 18 Mark Thomas 2016-05-25 06:19:11 UTC
(In reply to Michael Osipov from comment #17)

> Quite a nice solution. Line 274 has too many spaces in it.

Ack. If this works, I'll fix that before committing it.

> I will test that next week in a HP-UX 11.31 and will let you know.

Thanks.

> Are you going to backport this to 8? 

If it works it will be back-ported to all supported versions (i.e. all the way to 6).

> We are currently running and planning to move to 8.0.

Later this year work on 8.0.x is going to stop and the focus (for 8) will switch to 8.5.x. If you are planning a move to 8, I'd strongly recommend targetting 8.5

> For this case, I will download 9.0.0.M6 and apply the patch to it.

Perfect.
Comment 19 Konstantin Kolinko 2016-05-26 11:23:35 UTC
I think this may change the meaning of pid value in $CATALINA_PID file. I think that it will be the process id of nohup process instead of java one.

Known uses of pid value:

- forcibly terminating Tomcat with shutdown.sh -force

  echo "Killing Tomcat with the PID: $PID"
  kill -9 $PID

- asking java for a thread dump

  echo "To aid diagnostics a thread dump has been written to standard out."
  kill -3 `cat "$CATALINA_PID"`
Comment 20 Michael Osipov 2016-05-26 13:08:48 UTC
(In reply to Konstantin Kolinko from comment #19)
> I think this may change the meaning of pid value in $CATALINA_PID file. I
> think that it will be the process id of nohup process instead of java one.
> 
> Known uses of pid value:
> 
> - forcibly terminating Tomcat with shutdown.sh -force
> 
>   echo "Killing Tomcat with the PID: $PID"
>   kill -9 $PID
> 
> - asking java for a thread dump
> 
>   echo "To aid diagnostics a thread dump has been written to standard out."
>   kill -3 `cat "$CATALINA_PID"`

Are you certain about this?

Here is the output of HP-UX:

> root@blnn728x - /
> 243 # ps -fu root | grep nohup
>     root 29012 17788  0 15:06:04 pts/3     0:00 grep nohup
> 
> root@blnn728x - /
> 244 # echo $USER
> root
> 
> root@blnn728x - /
> 245 # ps -fu root | grep nohup
>     root 29088 17788  0 15:07:38 pts/3     0:00 grep nohup
> 
> root@blnn728x - /
> 246 # ps -fu smartld | grep nohup
> 
> root@blnn728x - /
> 247 # ps -fu smartld | grep java
>  smartld 17619     1  0  Mai 22  ?        59:14 /opt/java7/bin/java -Djava.util.logging.config.file=/var/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apa
>  smartld 28268     1  0 12:03:17 pts/2     3:48 /opt/java7/bin/java -Djava.util.logging.config.file=/var/opt/tomcat-services/conf/logging.properties -Djava.util.logging.manage
>
Comment 21 Konstantin Kolinko 2016-05-27 11:50:38 UTC
(In reply to Michael Osipov from comment #20)
> (In reply to Konstantin Kolinko from comment #19)
> > I think this may change the meaning of pid value in $CATALINA_PID file. I
> > think that it will be the process id of nohup process instead of java one.
> > 
> > Known uses of pid value:
> > 
> > - forcibly terminating Tomcat with shutdown.sh -force
> > 
> >   echo "Killing Tomcat with the PID: $PID"
> >   kill -9 $PID
> > 
> > - asking java for a thread dump
> > 
> >   echo "To aid diagnostics a thread dump has been written to standard out."
> >   kill -3 `cat "$CATALINA_PID"`
> 
> Are you certain about this?
> 
> Here is the output of HP-UX:
> 

Thank you for checking! I was just guessing. (I remember users reporting similar problem with wrong value in pid file in some old version of catalina.sh, ~4 years ago.)

Apparently this means that this version of nohup is implemented as replacing the old process with a new one, preserving the pid. A feature of unix exec() function. OK, good.

If you are using CATALINA_PID files, can you also check that they contain the expected value of process id?


BTW, in POSIX.1-2008/2013 there is a recommendation to redirect stdin when invoking nohup. See 'RATIONALE' section in
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/nohup.html

Their example:

nohup make </dev/null &


BTW, I think it will be better to explicitly unset _NOHUP variable by default. So that it is not carried over from a caller. It is an internal variable used by catalina.sh.

unset _NOHUP
Comment 22 Mark Thomas 2016-06-05 18:50:24 UTC
CATALINA_PID works as expected.
Excess spaces removed
unset _NOHUP added

I'll get this committed shortly.
Comment 23 Mark Thomas 2016-06-05 19:12:40 UTC
The patch has been applied to:
- 9.0.x for 9.0.0.M7 onwards
- 8.5.x for 8.5.3 onwards
- 8.0.x for 8.0.36 onwards
- 7.0.x for 7.0.70 onwards
- 6.0.x for 6.0.46 onwards
Comment 24 Michael Osipov 2016-06-05 21:27:12 UTC
Please note that my tests are still pending. Hopefully will be completes Monday or Tuesday.
Comment 25 Michael Osipov 2016-06-06 09:22:33 UTC
I can happily confirm that the patch from trunk works on HP-UX. Tomcat comes up as expected. Regarding the PID file:

> smartld@blnn724x - /var/opt/tomcat/work
> bash $ pwd
> /var/opt/tomcat/work
> 
> smartld@blnn724x - /var/opt/tomcat/work
> bash $ less catalina.pid
> 1961
> 
> smartld@blnn724x - /var/opt/tomcat/work
> bash $ ps -fu smartld | grep /tomcat/conf
>  smartld  1961     1  0 11:13:58 ?         0:49 /opt/java8/bin/java > -Djava.util.logging.config.file=/var/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apa

Thanks a lot for the patch. I can remove the sed patch with the upgrade to 6.0.46.
Comment 26 Christopher Schultz 2016-06-17 10:33:10 UTC
(In reply to Mark Thomas from comment #22)
> CATALINA_PID works as expected

I was skeptical. My quick test:

$ nohup sleep 100 &
[1] 9556

$ echo $!
9556
$ ps aux | grep sleep
chris            9559   0.0  0.0  2457368    880 s001  S+    6:30AM   0:00.00 grep sleep
chris            9556   0.0  0.0  2434824    664 s001  S     6:30AM   0:00.00 sleep 100

$! returns 9556, the pid of the 'sleep' process.

I haven't looked at the code for, say, GNU nohup, but I suspect it's used often enough in this type of situation that use of exec() is almost certainly required to make it useful.