Bug 63625 - Unable to start Tomcat 7.0.96 (stop by 0xc0000005)
Summary: Unable to start Tomcat 7.0.96 (stop by 0xc0000005)
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Packaging (show other bugs)
Version: 7.0.96
Hardware: PC All
: P2 critical (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-08-02 06:26 UTC by Norimasa Yamamoto
Modified: 2019-11-19 14:21 UTC (History)
1 user (show)



Attachments
appcrash_tomcat7.7z (54.70 KB, application/octet-stream)
2019-08-02 09:20 UTC, Norimasa Yamamoto
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Norimasa Yamamoto 2019-08-02 06:26:16 UTC
Unable to start Tomcat 7.0.96 on Windows 7.
0xc0000005 Error by Tomcat7.exe.

Tomcat 7.0.96 changed Tomcat7.exe version 1.2.0.0.
When I replaced Tomcat7.exe to version 1.1.0.0 (used Tomcat 7.0.94),
Tomcat 7.0.96 works fine.
Comment 1 Mark Thomas 2019-08-02 07:15:38 UTC
Please provide full version information for:
- Windows (winver or systeminfo)
- Java (java -version)
Comment 2 Norimasa Yamamoto 2019-08-02 09:20:58 UTC
Created attachment 36699 [details]
appcrash_tomcat7.7z

- Windows
Windows 7 Professional
Microsoft Windows
Version 6.1 (build 7601: Service Pack 1)

Microsoft Windows [Version 6.1.7601]

- java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) Client VM (build 25.151-b12, mixed mode)

- Additional : WER
attached "appcrash_tomcat7.7z"
Comment 3 Michael Osipov 2019-08-02 09:22:32 UTC
(In reply to Norimasa Yamamoto from comment #2)
> Created attachment 36699 [details]
> appcrash_tomcat7.7z
> 
> - Windows
> Windows 7 Professional
> Microsoft Windows
> Version 6.1 (build 7601: Service Pack 1)
> 
> Microsoft Windows [Version 6.1.7601]
> 
> - java -version
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
> Java HotSpot(TM) Client VM (build 25.151-b12, mixed mode)
> 
> - Additional : WER
> attached "appcrash_tomcat7.7z"

You should try a newer Java version first...
Comment 4 Norimasa Yamamoto 2019-08-02 09:58:18 UTC
Also latest Java 8u221 was crashed.

java -version
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) Client VM (build 25.221-b11, mixed mode)
Comment 5 Norimasa Yamamoto 2019-08-02 10:13:29 UTC
Also Adopt OpenJDK hotspot 11.0.4 was crashed.

java -version
openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.4+11)
OpenJDK Client VM AdoptOpenJDK (build 11.0.4+11, mixed mode)
Comment 6 Norimasa Yamamoto 2019-08-02 10:28:30 UTC
Also Adopt OpenJDK hotspot 12.0.2 was crashed.

java -version
openjdk version "12.0.2" 2019-07-16
OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.2+10)
OpenJDK Client VM AdoptOpenJDK (build 12.0.2+10, mixed mode)

No more latest version were released at this moment.
Comment 7 Mark Thomas 2019-08-02 10:58:32 UTC
This works for me on a clean Windows 7 SP1 x64 install with AdoptOpenJDK 1.8u222-b10

You appear to be using a 32-bit JVM. Make sure you are using the 32-bit version of tomcat7.exe

Also, make sure you have applied all currently available updates from Windows update.

At this point it looks like an environmental issue rather than a Tomcat bug.
Comment 8 Norimasa Yamamoto 2019-08-02 22:44:51 UTC
I made a new Windows 7 using Tomcat 7.0.96 x86/x64 service installer
and AdoptOpenJDK x86. It worked fine.

Next, I checked Tomcat7.exe arch.
I download from prunsrv.exe from Apache commons daemon
and to rename/replace them to installed Tomcat7.exe.
using x86 prunsrv.exe : OK
using x64 prunsrv.exe : NG (0xc0000005, like as this issue)

I think the NG machine was installed Tomcat7.exe x64 even if Java x86
installed directory was selected while Tomcat 7.0.96 installing.
But I don't know why it was occured.
I'll more test on next Monday on the NG machine.
Comment 9 Norimasa Yamamoto 2019-08-05 05:15:19 UTC
Currently status on NG machine.
- Tomcat 9.0.94 x86/x64 installer with x86 jvm.dll : OK
- Tomcat 9.0.96 x86/x64 installer with x86 jvm.dll : NG
-- installed Tomcat7.exe was x86 : YES (seems prunsrv.exe 1.2.0.0)
-- replace prunsrv.exe x86 1.2.0.0 as Tomcat7.exe : NG
-- replace prunsrv.exe x86 1.1.0.0 as Tomcat7.exe : OK
- Windows Update to latest : same as above
- Install MSVC10 x86 runtime : same as above
- Install MSVC2015-2019 x86 runtime : same as above
- procmon.exe : jvm.dll loaded, but dead while initializing ucrtbase.dll???
- from full local crash dump :
  Tomcat7.exe passed NULL lpLibFileName to LoadLibraryExW
  (thread #3 call stack)
  ntdll!RtlInitUnicodeStringEx+0x24 : access violation
  KERNELBASE!LoadLibraryExW+0x5e : args (NULL, NULL, LOAD_LIBRARY_AS_DATAFILE)
  Tomcat7+0x2a4d : ?
  Tomcat7+0x2a4d arg : seems having correct jvm.dll path
  :
Comment 10 Norimasa Yamamoto 2019-08-05 06:33:32 UTC
Difference between prunsrv.exe 1.1.0.0 and 1.2.0.0.
prunsrv.exe 1.2.0.0 :
  Try to use ucrtbase.dll!_wputenv at prunsrv!apxSetInprocEnvironment.
  When there, set SQ_ENVIRONMENT to ucrtbase.dll's environment too.

ucrtbase.dll is used from vcruntime140.dll.
When target jvm.dll is with lower than vcruntime140.dll (i.e. msvc100.dll),
prunsrv.exe must load msvcrXXX.dll, instead. Because jvm.dll's environment is
in msvcrXXX.dll (not in ucrtbase.dll).

So I silulated ucrtbase.dll was missing (patch at 0000E0E1: 64 to 7A),
Tomcat7.exe 1.2.0.0 started. However, I don't test it work or not.
Comment 11 Mark Thomas 2019-08-05 08:36:11 UTC
Thanks for the additional investigation. This may turn out to be a Commons Daemon issue. Let me look into the Daemon change that added the dependency on ucrtbase.dll
Comment 12 Konstantin Preißer 2019-08-05 09:07:50 UTC
Hi,

FYI ucrtbase.dll is the Windows Universal C Runtime which will be used when compiling with Visual C++ 2015 and higher [1].
It is included with Windows 10/Windows Server 2016 and higher. For older OSes, it will only be present if e.g. KB3118401 was installed from Windows Update.
However, it is possible to redistribute it e.g. by using local deployment [3].


[1] https://docs.microsoft.com/en-us/cpp/windows/universal-crt-deployment?view=vs-2019
[2] https://support.microsoft.com/en-us/help/3118401/update-for-universal-c-runtime-in-windows
[3] https://docs.microsoft.com/en-us/cpp/windows/universal-crt-deployment?view=vs-2019#local-deployment
Comment 13 Mark Thomas 2019-08-05 11:03:38 UTC
Just adding to the general information. I'm not drawing any conclusions as this point.

I'm currently setting up a clean install of 32-bit Windows 7. I haven't finished my battle with Windows update yet but already one of the important updates has installed (a fairly recent version of) ucrtbase.dll
Comment 14 Norimasa Yamamoto 2019-08-07 10:35:50 UTC
I thought a concept how to fix this issue.
(If prnsrv.exe really need to change jvm.dll's environment block.)

Before look a concept, important thing should be known.

- Where is a environment block (compiled with shared C runtime)?

compiled by MSVS 2015-2019
MODULE(exe, dll, so, ...)
  _wputenv()
  ==> api-ms-win-crt-environment-l1-1-0.dll!_wputenv (forwarder)
  ==> ucrtbase.dll!_wputenv (HERE!)

compiled by MSVC 7 to 12
MODULE
  _wputenv()
  ==> msvcrXXX.dll!_wputenv (HERE!)
(XXX is one of 70, 80, 90, 100, 110, 120 (release)
 or with postfix "d" (debug))

Other compilers...
compiled by MSVC 6, MinGW w32api+GCC
  held in "msvcrt.dll"
compiled by compiled by Enbarcadero/Borland C++ Builder
  held in "ccYYXXXmt.dll"
compiled by Cygwin
  held in "cygwin1.dll"
:

On the other hand, without shared C runtime,
every modules hold in itself.

compiled by any compiler (include MSVS 2015-2019)
MODULE (HERE! (=builtin))
  _wputenv

If MODULE did not export _wputenv or similar function,
it is impossible to change from OUTER MODULEs.
If OUTER MODULEs knew its address, it may can change
but it is very danger way.

-- end of section

Because prunsrv.exe is currently static linked with C runtime,
prunsrv.exe does not depending any shared C runtime.
So jvm.dll was loaded in prunsrv.exe process, prunsrv.exe can guess
jvm.dll's C runtime by GetModuleHandle.
(If prunsrv.exe were compiled with shared C runtime,
 prunsrv.exe may parse jvm.dll's Import Table in PE section,
 use CreateToolhelp32Snapshot related funtions to find loaded snapshot,
 ... From my experience, these are hard.)

(concept code)

- before
hmodUcrt = LoadLibraryExA("ucrtbase.dll", NULL, LOAD_LIBRARY_SEARCH_SYSTEM32);
if (hmodUcrt != NULL) {
  wputenv_ucrt =  (WPUTENV) GetProcAddress(hmodUcrt, "_wputenv");
}

- after
// jvm.dll should be loaded at this point.
// Guess jvm.dll was compiled by known compilers.
putenv_ucrt = NULL;
if (
     ((hmodUcrt = GetModuleHandle("ucrtbase.dll")) != NULL) // MSVC 14.0x-14.2x
  || ((hmodUcrt = GetModuleHandle("msvcr120.dll")) != NULL) // MSVC 12.0
  || ((hmodUcrt = GetModuleHandle("msvcr110.dll")) != NULL) // MSVC 10.0
  || ((hmodUcrt = GetModuleHandle("msvcr100.dll")) != NULL) // MSVC 10.0
  : (...)
  || ((hmodUcrt = GetModuleHandle("msvcrt.dll")) != NULL) // MSVC 6.0, GCC
) {
  putenv_ucrt =  (WPUTENV) GetProcAddress(hmodUcrt, "_wputenv");
} else {
  // jvm.dll was not compiled by known compilers.
}
Comment 15 Mark Thomas 2019-08-16 17:26:58 UTC
I've made some progress. This issue is only observed with 32-bit JREs. I've narrowed this down using the 32-bit and 64-bit versions of AdoptOpenJDK 8.0.222.10 hotspot (both depend on MSVCR120) on Windows 7 64 bit.

The first call to wputenv_ucrt corrupts memory (root cause currently unknown) and that causes subsequent calls to fail.

Still looking for the root cause although it looks like using a 64-bit JVM should work around the issue.
Comment 16 Manuel Conde 2019-08-27 10:42:42 UTC
Hi all.

Not exactly the same bug, but related:

Win XP 32 bit
Java 6 32 bit
Tomcat 7.0.96.exe

Error on install when trying to create Tomcat7 service.

If I try 7.0.94, all works fine.
Comment 17 Mark Thomas 2019-08-28 06:43:12 UTC
Note that Tomcat is not officially supported on Windows XP now that Microsoft have ended support for XP. That said, you should be able to use Tomcat9.exe from 9.0.22 as a workaround.
Comment 18 Norimasa Yamamoto 2019-09-02 09:36:04 UTC
I checked WPUTENV prototype.

In src\native\windows\include\apwin.h
typedef int (__stdcall *WPUTENV) (const wchar_t *env);

But I disassembled from ucrtbase.dll!_wputenv om x86 platform,
it seems to be _cdecl (=calling args should be destroyed by caller).

This mismatch will cause stack corruption after calling wputenv_ucrt
on x86. This should be root cause, I think.
(On x64, first few calling args are passed by registers.)

Changing prototype will fix the error from stack corruption.
However, environment block manipulation is not work on AdoptOpenJDK,
because it is depending on msvcr120.dll.
Comment 19 Mark Thomas 2019-09-03 21:33:30 UTC
Tx. I've fixed this in DAEMON and confirmed that it does fix the issue. I'm leaving this open until Tomcat is updated to use a Daemon release with the fix.
Comment 20 Christopher Schultz 2019-09-03 22:23:26 UTC
(In reply to Mark Thomas from comment #19)
> Tx. I've fixed this in DAEMON and confirmed that it does fix the issue. I'm
> leaving this open until Tomcat is updated to use a Daemon release with the
> fix.

Would you mind posting the commit id for that? I'd like to see the change you made. It's not often you have to override the compiler to specify a calling convention on the CALLER's side...
Comment 21 Rainer Jung 2019-09-04 00:16:05 UTC
Probably this one:

This is an automated email from the ASF dual-hosted git repository.

markt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/commons-daemon.git


The following commit(s) were added to refs/heads/master by this push:
     new 4228b53  Correct regression in DAEMON-401
4228b53 is described below

commit 4228b53ec99ae4e7f1d96242fbc8d4a27316fd57
Author: Mark Thomas <markt@apache.org>
AuthorDate: Tue Sep 3 22:30:37 2019 +0100

    Correct regression in DAEMON-401
    
    An incorrect calling convention (__stdcall rather than __cdecl) caused
    memory corruption when running as a 32-bit process.
---
 src/changes/changes.xml             | 4 ++++
 src/native/windows/include/apxwin.h | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/src/changes/changes.xml b/src/changes/changes.xml
index 17e040e..6ce7717 100644
--- a/src/changes/changes.xml
+++ b/src/changes/changes.xml
@@ -42,6 +42,10 @@
       <action issue="DAEMON-403" type="fix" dev="markt" due-to="Charles">
         jsvc. Correct debug log message that reports change in umask.
       </action>
+      <action issue="DAEMON-401" type="fix" dev="markt" due-to="Norimasa Yamamoto">
+        Procrun. Correct a regression in the previous fix for this issue that
+        caused 32-bit services to crash on start-up.
+      </action> 
     </release>
     <release version="1.2.0" date="2019-06-28" description="Feature and bug fix release">
       <action type="fix" dev="markt" due-to="mturk" >
diff --git a/src/native/windows/include/apxwin.h b/src/native/windows/include/apxwin.h
index cbd4ac9..1d70063 100644
--- a/src/native/windows/include/apxwin.h
+++ b/src/native/windows/include/apxwin.h
@@ -118,7 +118,7 @@ LPSTR       MzWideToAscii(LPCWSTR ws, LPSTR s);
 LPSTR       WideToANSI(LPCWSTR ws);
 LPSTR       MzWideToANSI(LPCWSTR ws);
 
-typedef int (__stdcall *WPUTENV) (const wchar_t *env);
+typedef int (*WPUTENV) (const wchar_t *env);
 
 typedef struct APXMULTISZ APXMULTISZ;
 typedef APXMULTISZ*       LPAPXMULTISZ;



Regards,

Rainer
Comment 22 Christopher Schultz 2019-09-04 16:33:46 UTC
(In reply to Rainer Jung from comment #21)
> Probably this one:

[snip]

> -typedef int (__stdcall *WPUTENV) (const wchar_t *env);
> +typedef int (*WPUTENV) (const wchar_t *env);

OH.

I didn't realize WE were responsible for declaring the prototype of the call. Makes sense, now.
Comment 23 Mark Thomas 2019-09-10 08:25:02 UTC
I've just updated Tomcat to use Commons Daemon 1.2.1 where this is fixed.

Fixed in:
- master for 9.0.25 onwards
- 8.5.x for 8.5.46 onwards
- 7.0.x for 7.0.97 onwards
Comment 24 Tony Yan 2019-11-19 05:01:08 UTC
(In reply to Mark Thomas from comment #23)
> I've just updated Tomcat to use Commons Daemon 1.2.1 where this is fixed.
> 
> Fixed in:
> - master for 9.0.25 onwards
> - 8.5.x for 8.5.46 onwards
> - 7.0.x for 7.0.97 onwards

So, the fix in 7.0.97, how about 7.0.96? Tomcat 7.0.97 is not released yet.
Comment 25 Tony Yan 2019-11-19 05:09:07 UTC
Does it mean that Tomcat 7.0.96 32bit is not working because the Tomcat7.exe has defect?
Comment 26 Mark Thomas 2019-11-19 08:38:41 UTC
(In reply to Tony Yan from comment #25)
> Does it mean that Tomcat 7.0.96 32bit is not working because the Tomcat7.exe
> has defect?

Yes.

You can download Commons Daemon 1.2.1 and rename the 32-bit version of prunsrv.exe to Tomcat7.exe and use that with Tomcat 7.0.96.
Comment 27 Tony Yan 2019-11-19 14:21:57 UTC
Thank you very much for such quick response, Mark. I will download the Commons Daemon 1.2.1. And thanks all the people for the contributions in this fix.