Bug 55708

Summary: Access violation in msvcrt.dll when calling getenv("PATH") under load running on Windows Server
Product: Apache httpd-2 Reporter: Pierre Boudreau <pierre.boudreau>
Component: CoreAssignee: Apache HTTPD Bugs Mailing List <bugs>
Severity: normal Keywords: MassUpdate
Priority: P2    
Version: 2.2.22   
Target Milestone: ---   
Hardware: PC   
OS: Windows Server 2003   
Attachments: Screen shot of Visual Studio debugging the crash dump

Description Pierre Boudreau 2013-10-25 19:42:10 UTC
Created attachment 30972 [details]
Screen shot of Visual Studio debugging the crash dump

-Windows Server 2003 R2 SP2
-Running in a VMWare VM with 4 Intel Xeon CPUs @2.00Ghz with 3.8 GB of RAM
-See attached manifest.txt and appcompat.txt from one of the crash dumps for a listing of the applications and libraries running at the time of the crash.

Error as seen in the Windows Application Event Viewer:
Faulting application httpd.exe, version, faulting module msvcrt.dll, version 7.0.3790.3959, fault address 0x00027d70.

Problem description and analysis:
Under load, we are experiencing access violation errors in msvcrt.dll running under the Apache httpd.exe process.  Analysis of two heap dumps that have been collected from two different servers running the same application shows that the access violation happens in calls to the getenv() function on line 184 of util_script.c.  These are rare occurrences in the order of one in many thousands of calls.  In both the heap dumps that we have collected, the call is exactly the same.

Attached are two screen shots of Visual Studio debugging each of the crash dumps we have analyzed.  The line of code highlighted is the last line of httpd code before it falls into the msvcrt library.   This is where the call to getenv(“PATH”) is made.

The access violation happens two calls deeper inside msvcrt.dll in __getenv_lk() as seen in the call stack in the bottom right of the screen shots.

What we see leads us to believe that there must be some kind of race condition or something that is somehow not thread safe in that function that results in an access violation in some rare cases.  We have reproduced the error in a test environment.  It happened once in a 1.5 hour test at a sustained rate of close to 1000 hits per second on the Apache httpd server.  It has also happened in our production environment at a lower load of around 100 hits per second.  Under that load, it can run for several days before we see an occurrence of it.

In both cases, the request were quite different from one another.  They were both requests that Apache passes through to back end servers via mod_jk.  In one case the backend server is a Tomcat 7 instance and in the other case it is an entirely other application running in JBoss 4.3.  And both those requests are successful most of the time, so it doesn't look like the crash is specific to any particular type of request.  But I wouldn't rule out that mod_jk is somehow involved since it does appear in the call stack of the crash dump.

Some thoughts:
Is the getenv() function being called supposed to be thread safe?
Should the code calling it be locking around it if it isn't thread safe?

I can provide the crash dump files for analysis privately.
Comment 1 William A. Rowe Jr. 2018-11-07 21:08:28 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.