Bug 34664

Summary: MPM NT+PHP? erealloc(): Unable to allocate 98304 bytes
Product: Apache httpd-2 Reporter: Pavel Rubin <pash_ka>
Component: PlatformAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED INVALID    
Severity: critical    
Priority: P1    
Version: 2.2.3   
Target Milestone: ---   
Hardware: PC   
OS: Windows XP   

Description Pavel Rubin 2005-04-28 11:12:24 UTC
Platform: ASUS AP1600R (2 Xeon 2.4Ghz, 2Gb RAM)
Software: Win2000, Apache 2.0.54, PHP 5.0.4, MySQL 4.1.10a, no 
firevals/antivirus
Average server load: 5-10 requests/sec

First, I've noticed 
---
[warn] (OS 64)The specified network name is no longer available.  : 
winnt_accept: Asynchronous AcceptEx failed
---
warnings and very unstable behaviour of Apache: sometimes request had been 
processed in 20 ms and oter times - more than 3000 ms (since i was running 
tests on local mashine network perfomance was not an option, and this strange 
random responce times was tested on plain html-page with no scripting).

The solution I've found in bug 21425:
---
Win32DisableAcceptEx
EnableMMAP Off
EnableSendfile Off 
---
solved both of tis problems but inroduced another one: 
---
FATAL:  erealloc():  Unable to allocate 98304 bytes
---
Errors like this one occured every 5-10 minutes and caused server restart.
The only solution I've found is decreasing ThreadsPerChild setting from 350 to 
170. Since doing this I have not seen this error.

I've also noticed another problem: restarting server with "apache -k restart" 
command and even stoping it with "net stop apache2" failed to close child 
process. And sometimes lines like
---
[crit] [Thu Apr 28 10:54:23 2005] file .\\server\\mpm\\winnt\\child.c, line 
1078, assertion "(rv >= 0) && (rv < threads_created)" failed
---
(bug 11997) ocured in error log.

I've commented out "EnableMMAP Off" and now everything works fine. Or I just 
hadn't seen the problem yet. :)
Comment 1 Pavel Rubin 2005-05-03 09:42:25 UTC
I'm sorry but the problem described in bug 11997 is still here...
And child process fails to exit and stays in memory.
Comment 2 Levi Tedder 2006-02-23 10:52:10 UTC
I'm having the same problem on my Win2k server (fully patched/updated) - 2x
Intel Xeon. I have Apache 2.0.55 (with openSSL 0.9.8a), PHP 5.1.2. 

The log entry:
[Thu Feb 23 09:28:12 2006] [warn] (OS 64)The specified network name is no longer
available.  : winnt_accept: Asynchronous AcceptEx failed.

was not present while we were using a Symantec Enterprise Firewall, but were
introduced when we switched to Cisco Pix 515e Firewall (appliance). I got rid of
such log entries with Win32DisableAcceptEx in the httpd.conf, but this caused
the following errors about every 10-20 minutes:

FATAL:  erealloc():  Unable to allocate 393216 bytes
[Thu Feb 23 10:15:58 2006] [notice] Parent: child process exited with status 1
-- Restarting.

I see that memory consumption by apache is increasing, and number of threads
too, until it restarts itself.

If I remove the Win32DisableAcceptEx, the server is "very" slow in handling
requests and of course the error.log is getting all those entries I quoted first.
But as I said, this started after we installed the Cisco FW, and we have placed
a bug-thing with Cisco as well.
Comment 3 Vincent Lepot 2006-07-04 14:10:17 UTC
I have the same problem on Win2K servers, PHP 4.3.11.

The erealloc/emalloc fatals occurs when I put Win32DisableAcceptEx config
directive in httpd.conf. When I comment it, no more erealloc/emalloc occurs in
the logs, but the warnings described above.

So it seems to be a memory leak when using the old Accept primitive or something...
Comment 4 Levi Tedder 2006-10-07 03:10:41 UTC
(In reply to comment #2)

This problem is "increasing", meaning it cause a big problem. The line:

FATAL:  erealloc():  Unable to allocate 1572864 bytes

with the exact same number of bytes each time, is appearing every 10-15 minutes
now, and after a day or two apache stops responding (web pages will not load). A
restart of the service will not fix it, I need to restart the entire server.

I have the following now:

(win2k server still (2 cpu))

Apache 2.0.59
OpenSSL 0.9.8c
PHP 5.1.6
Comment 5 Phil Thomas 2006-11-16 13:53:08 UTC
Exact same issue here. Apache restarts (stopping all transfers & active
connections) after consuming a lot of RAM memory (up to peaks of 100-200MB).

Apache 2.2.3
WinXP 1GBRAM Pentium M 1,4GHz
Happens when Firewall disabled.

Hope to see a bugfix soon.
Phil
Comment 6 William A. Rowe Jr. 2006-11-16 14:54:36 UTC
FYI this is known-of on 2.0.59 as well, and in the process of researching.

Can you reporters confirm if you are or are not proxying traffic, and if
that's using ssl from the client and/or ssl to the backend?

Or if you are simply using ssl or not for client (non-proxy) traffic?
Comment 7 Levi Tedder 2006-11-16 23:04:46 UTC
(In reply to comment #6)
> FYI this is known-of on 2.0.59 as well, and in the process of researching.
> Can you reporters confirm if you are or are not proxying traffic, and if
> that's using ssl from the client and/or ssl to the backend?
> Or if you are simply using ssl or not for client (non-proxy) traffic?

We are not proxying traffic, and are simply using ssl for client traffic. Note 
that this issue is only valid when Win32DisableAcceptEx is enabled 
(uncommented) in the conf. I have not tried this with PHP 5.2 which has been 
released, to see if that has something to do with it
Comment 8 William A. Rowe Jr. 2006-11-17 01:27:14 UTC
One more note - we -don't- call erealloc from apr 0.9 or httpd 2.0.

So something else is going on here, either in PHP or in the msvcrt, or perhaps
because of crossed up msvc versions.
Comment 9 Levi Tedder 2006-11-17 01:32:26 UTC
FYI there's a bugreport at php.net regarding this, although I don't think it's 
anything new there: http://bugs.php.net/bug.php?id=32002

I will try to install PHP 5.2 after the weekend and will let you know if it 
helps in any way. I understand that there's some changes to memoryhandling or 
something there....
Comment 10 William A. Rowe Jr. 2006-11-17 01:42:40 UTC
Well if you can determine that this happens with PHP compiled with Visual C 6.0,
linked using /MD (multithread dll c runtime) or only Visual C > 6.0, we might be
on to something.

It's a question of if this is a PHP bug, or a memory management flaw resulting
from the c runtimes in use.
Comment 11 Pavel Rubin 2006-11-17 05:46:22 UTC
I have not used SSL or mod_proxy at the time of this report.
It was very simple configuration for a few vhosts with PHP (as module) ans SSI.

Now we've changed the platform to Linux, so I can do no more tests... 
Comment 12 Levi Tedder 2006-12-03 04:51:03 UTC
I have now installed PHP 5.2 on the server (regular downloaded from php.net),
and the issue seems to be gone. No fatal errors for 24 hours, but I will let you
know if it happens again during peek traffic (although it came every 10-15
minutes no matter what traffic we had).
Comment 13 Levi Tedder 2006-12-04 00:28:40 UTC
Well, the fatal is still gone, but Apache seems to leak memory. It starts with a
few MB, but is up in 4-500 MB "shortly". 

From my conf:
<IfModule mpm_winnt.c>
ThreadsPerChild 200
MaxRequestsPerChild  0
</IfModule>
EnableMMAP off
Win32DisableAcceptEx
EnableSendfile off

I have removed Win32DisableAcceptEx, and the memory consumption does not
increase. But then I'm back to "The specified network name is no longer
available.  : winnt_accept: Asynchronous AcceptEx failed." in the log. Last time
I did this it also served pages really slow, but now it seems to do this faster.
So, I think I will leave it like this and just purge the error log once in a
while. Or degrade to PHP 5.1.6 to have apache restart itself :)

Comment 14 Csabo 2006-12-21 11:03:01 UTC
Hello everyone,

I'd like to report that this particular bug is affecting us as well. We have two
identical web servers running (using load balancing):
Dell, Win2003 Web Edition, 2.8 Ghz, 2 GB memory
Apache 2.2.3, OpenSSL 0.9.8d
Tomcat 5.5.17
PHP 5.2, Zend Optimizer 3

We used to run IIS for over 4 years, but we've always had problems with the
IIS->Tomcat integration. This September we switched to Apache and also started
using PHP.

I had to apply the "Win32DisableAcceptEx / EnableMMAP off / EnableSendfile off"
trio, without it, same errors as mentioned above kept appearing in the error
log, and the users got Page Not Found errors. After applying these settings,
everything seemed fine. There's no firewall software running, but there is a
Sonicwall firewall on the network.

After applying Win32DisableAcceptEx, then "FATAL:  erealloc():  Unable to
allocate XXX bytes" lines immediately started appearing. I did a lot of
searching, assuming the problem was with PHP. (5.1.6 did the same thing. We have
some pages that are encoded, so we needed the new Zend Optimizer too before we
could upgrade to 5.2. It's now out, we upgraded, same problem.)

I wrote a small program to monitor and log the memory taken up by httpd.exe, and
I found that it restarted itself about every 4-5 hours, when it reached around
440Mb. Then I finally found this thread, and the suggestion of the original
poster worked. "ThreadsPerChild" was 250 in httpd.conf, I reduced that to 120,
and it was fine. The service was 100% up! However, our sites became slow, we're
getting about 100 requests per second. We tried to increase the number of
threads. With 150 it still died every 4-5 hours. With 135 too. Now it's on 128,
and it's been stable for almost a day. That's way too low however, the servers
should be able to handle a lot more than that.

I understand that this problem will be very hard to track down and fix... I'm
still not even 100% convinced that it's an Apache problem, could be PHP. Still I
hope someone will be able to figure this out and put out a fix.
Comment 15 Csabo 2007-03-05 11:15:44 UTC
Hi all,

Oh, how I hope someone can give us a clue on what to do with this. Here's what
we've been doing to try to isolate the problem.

1) We thought SSL might be the problem, so we took it out of the equation by
setting up a new web server to take all the SSL connection, and reconfigured
Apache on our production servers to only listen on port 80. No noticeable
changes or improvement.

2) We purchased a shiny new web server from Dell. Quad core CPU (2.33 Ghz), 4 GB
RAM. Very fast, and should be more than adequate for a simple web server. Same
configuration as our other servers. Apache (or PHP, we still don't know which)
dies just as happily:

Fatal error: Out of memory (allocated 0) (tried to allocate 2 bytes) in Unknown
on line 0
[Mon Mar 05 02:44:04 2007] [notice] Parent: child process exited with status
4294967295 -- Restarting.

We watched the server closely, and when these errors happen, the CPU is low, the
number of active connections is low, and there's over 2 GB of physical memory
available. It's running out of *SOMETHING*, we just don't know what. It always,
very reliably runs out of this something. The higher the threads, the faster (it
seems).

But, at least we had 3 servers serving up 128 threads each, which was
temporarily acceptable.

3) We are running 7 websites, only one of which uses PHP. So we directed the 6
websites to just one server and we opened up the threads to 512, and that
machine has been running without a hitch ever since. So it's just Apache +
Tomcat. Hooray! Go Apache! This points to the problem definitely being in PHP,
or the integration of PHP + Apache. (Please, don't send us away to go and post
it on the PHP bug list just yet... All they are going to do is mark it as bogus.)

4) We updated to Apache 2.2.4. No noticeable difference unfortunately.

The problem is, the traffic on the one site that uses PHP is steadily
increasing. (It feels weird calling this a problem :-)) We upgraded our staging
server to 2 GB of RAM, so it's the same specs as the others, and now 3
production servers are serving up just that one site. In addition we restart
Apache daily. It worked fine for a while, but now it's getting worse: even with
just serving 128 threads, and restarting the whole service daily (we have the
redundancy for it), the above error is starting to present itself more and more
frequently...

I didn't mention yet that we tried for over a week to reliably reproduce this
problem on our staging server, but we simply cannot. Our conclusion was that
it's because when we send load to it (much bigger loads than what we normally
get on production), it's all coming from the same IP. Whereas normal traffic
comes from all over the place, different IPs, different browsers.

We've looked at this so much already and running out of ideas and options. Help
please?
Comment 16 William A. Rowe Jr. 2007-12-21 22:12:51 UTC
closing this as a php bug.

http://httpd.apache.org/dev/debugging.html

gives information on how to get an ENTIRE backtrace.