|Summary:||MPM NT+PHP? erealloc(): Unable to allocate 98304 bytes|
|Product:||Apache httpd-2||Reporter:||Pavel Rubin <pash_ka>|
|Component:||Platform||Assignee:||Apache HTTPD Bugs Mailing List <bugs>|
Description Pavel Rubin 2005-04-28 11:12:24 UTC
Platform: ASUS AP1600R (2 Xeon 2.4Ghz, 2Gb RAM) Software: Win2000, Apache 2.0.54, PHP 5.0.4, MySQL 4.1.10a, no firevals/antivirus Average server load: 5-10 requests/sec First, I've noticed --- [warn] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed --- warnings and very unstable behaviour of Apache: sometimes request had been processed in 20 ms and oter times - more than 3000 ms (since i was running tests on local mashine network perfomance was not an option, and this strange random responce times was tested on plain html-page with no scripting). The solution I've found in bug 21425: --- Win32DisableAcceptEx EnableMMAP Off EnableSendfile Off --- solved both of tis problems but inroduced another one: --- FATAL: erealloc(): Unable to allocate 98304 bytes --- Errors like this one occured every 5-10 minutes and caused server restart. The only solution I've found is decreasing ThreadsPerChild setting from 350 to 170. Since doing this I have not seen this error. I've also noticed another problem: restarting server with "apache -k restart" command and even stoping it with "net stop apache2" failed to close child process. And sometimes lines like --- [crit] [Thu Apr 28 10:54:23 2005] file .\\server\\mpm\\winnt\\child.c, line 1078, assertion "(rv >= 0) && (rv < threads_created)" failed --- (bug 11997) ocured in error log. I've commented out "EnableMMAP Off" and now everything works fine. Or I just hadn't seen the problem yet. :)
Comment 1 Pavel Rubin 2005-05-03 09:42:25 UTC
I'm sorry but the problem described in bug 11997 is still here... And child process fails to exit and stays in memory.
Comment 2 Levi Tedder 2006-02-23 10:52:10 UTC
I'm having the same problem on my Win2k server (fully patched/updated) - 2x Intel Xeon. I have Apache 2.0.55 (with openSSL 0.9.8a), PHP 5.1.2. The log entry: [Thu Feb 23 09:28:12 2006] [warn] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed. was not present while we were using a Symantec Enterprise Firewall, but were introduced when we switched to Cisco Pix 515e Firewall (appliance). I got rid of such log entries with Win32DisableAcceptEx in the httpd.conf, but this caused the following errors about every 10-20 minutes: FATAL: erealloc(): Unable to allocate 393216 bytes [Thu Feb 23 10:15:58 2006] [notice] Parent: child process exited with status 1 -- Restarting. I see that memory consumption by apache is increasing, and number of threads too, until it restarts itself. If I remove the Win32DisableAcceptEx, the server is "very" slow in handling requests and of course the error.log is getting all those entries I quoted first. But as I said, this started after we installed the Cisco FW, and we have placed a bug-thing with Cisco as well.
Comment 3 Vincent Lepot 2006-07-04 14:10:17 UTC
I have the same problem on Win2K servers, PHP 4.3.11. The erealloc/emalloc fatals occurs when I put Win32DisableAcceptEx config directive in httpd.conf. When I comment it, no more erealloc/emalloc occurs in the logs, but the warnings described above. So it seems to be a memory leak when using the old Accept primitive or something...
Comment 4 Levi Tedder 2006-10-07 03:10:41 UTC
(In reply to comment #2) This problem is "increasing", meaning it cause a big problem. The line: FATAL: erealloc(): Unable to allocate 1572864 bytes with the exact same number of bytes each time, is appearing every 10-15 minutes now, and after a day or two apache stops responding (web pages will not load). A restart of the service will not fix it, I need to restart the entire server. I have the following now: (win2k server still (2 cpu)) Apache 2.0.59 OpenSSL 0.9.8c PHP 5.1.6
Comment 5 Phil Thomas 2006-11-16 13:53:08 UTC
Exact same issue here. Apache restarts (stopping all transfers & active connections) after consuming a lot of RAM memory (up to peaks of 100-200MB). Apache 2.2.3 WinXP 1GBRAM Pentium M 1,4GHz Happens when Firewall disabled. Hope to see a bugfix soon. Phil
Comment 6 William A. Rowe Jr. 2006-11-16 14:54:36 UTC
FYI this is known-of on 2.0.59 as well, and in the process of researching. Can you reporters confirm if you are or are not proxying traffic, and if that's using ssl from the client and/or ssl to the backend? Or if you are simply using ssl or not for client (non-proxy) traffic?
Comment 7 Levi Tedder 2006-11-16 23:04:46 UTC
(In reply to comment #6) > FYI this is known-of on 2.0.59 as well, and in the process of researching. > Can you reporters confirm if you are or are not proxying traffic, and if > that's using ssl from the client and/or ssl to the backend? > Or if you are simply using ssl or not for client (non-proxy) traffic? We are not proxying traffic, and are simply using ssl for client traffic. Note that this issue is only valid when Win32DisableAcceptEx is enabled (uncommented) in the conf. I have not tried this with PHP 5.2 which has been released, to see if that has something to do with it
Comment 8 William A. Rowe Jr. 2006-11-17 01:27:14 UTC
One more note - we -don't- call erealloc from apr 0.9 or httpd 2.0. So something else is going on here, either in PHP or in the msvcrt, or perhaps because of crossed up msvc versions.
Comment 9 Levi Tedder 2006-11-17 01:32:26 UTC
FYI there's a bugreport at php.net regarding this, although I don't think it's anything new there: http://bugs.php.net/bug.php?id=32002 I will try to install PHP 5.2 after the weekend and will let you know if it helps in any way. I understand that there's some changes to memoryhandling or something there....
Comment 10 William A. Rowe Jr. 2006-11-17 01:42:40 UTC
Well if you can determine that this happens with PHP compiled with Visual C 6.0, linked using /MD (multithread dll c runtime) or only Visual C > 6.0, we might be on to something. It's a question of if this is a PHP bug, or a memory management flaw resulting from the c runtimes in use.
Comment 11 Pavel Rubin 2006-11-17 05:46:22 UTC
I have not used SSL or mod_proxy at the time of this report. It was very simple configuration for a few vhosts with PHP (as module) ans SSI. Now we've changed the platform to Linux, so I can do no more tests...
Comment 12 Levi Tedder 2006-12-03 04:51:03 UTC
I have now installed PHP 5.2 on the server (regular downloaded from php.net), and the issue seems to be gone. No fatal errors for 24 hours, but I will let you know if it happens again during peek traffic (although it came every 10-15 minutes no matter what traffic we had).
Comment 13 Levi Tedder 2006-12-04 00:28:40 UTC
Well, the fatal is still gone, but Apache seems to leak memory. It starts with a few MB, but is up in 4-500 MB "shortly". From my conf: <IfModule mpm_winnt.c> ThreadsPerChild 200 MaxRequestsPerChild 0 </IfModule> EnableMMAP off Win32DisableAcceptEx EnableSendfile off I have removed Win32DisableAcceptEx, and the memory consumption does not increase. But then I'm back to "The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed." in the log. Last time I did this it also served pages really slow, but now it seems to do this faster. So, I think I will leave it like this and just purge the error log once in a while. Or degrade to PHP 5.1.6 to have apache restart itself :)
Comment 14 Csabo 2006-12-21 11:03:01 UTC
Hello everyone, I'd like to report that this particular bug is affecting us as well. We have two identical web servers running (using load balancing): Dell, Win2003 Web Edition, 2.8 Ghz, 2 GB memory Apache 2.2.3, OpenSSL 0.9.8d Tomcat 5.5.17 PHP 5.2, Zend Optimizer 3 We used to run IIS for over 4 years, but we've always had problems with the IIS->Tomcat integration. This September we switched to Apache and also started using PHP. I had to apply the "Win32DisableAcceptEx / EnableMMAP off / EnableSendfile off" trio, without it, same errors as mentioned above kept appearing in the error log, and the users got Page Not Found errors. After applying these settings, everything seemed fine. There's no firewall software running, but there is a Sonicwall firewall on the network. After applying Win32DisableAcceptEx, then "FATAL: erealloc(): Unable to allocate XXX bytes" lines immediately started appearing. I did a lot of searching, assuming the problem was with PHP. (5.1.6 did the same thing. We have some pages that are encoded, so we needed the new Zend Optimizer too before we could upgrade to 5.2. It's now out, we upgraded, same problem.) I wrote a small program to monitor and log the memory taken up by httpd.exe, and I found that it restarted itself about every 4-5 hours, when it reached around 440Mb. Then I finally found this thread, and the suggestion of the original poster worked. "ThreadsPerChild" was 250 in httpd.conf, I reduced that to 120, and it was fine. The service was 100% up! However, our sites became slow, we're getting about 100 requests per second. We tried to increase the number of threads. With 150 it still died every 4-5 hours. With 135 too. Now it's on 128, and it's been stable for almost a day. That's way too low however, the servers should be able to handle a lot more than that. I understand that this problem will be very hard to track down and fix... I'm still not even 100% convinced that it's an Apache problem, could be PHP. Still I hope someone will be able to figure this out and put out a fix.
Comment 15 Csabo 2007-03-05 11:15:44 UTC
Hi all, Oh, how I hope someone can give us a clue on what to do with this. Here's what we've been doing to try to isolate the problem. 1) We thought SSL might be the problem, so we took it out of the equation by setting up a new web server to take all the SSL connection, and reconfigured Apache on our production servers to only listen on port 80. No noticeable changes or improvement. 2) We purchased a shiny new web server from Dell. Quad core CPU (2.33 Ghz), 4 GB RAM. Very fast, and should be more than adequate for a simple web server. Same configuration as our other servers. Apache (or PHP, we still don't know which) dies just as happily: Fatal error: Out of memory (allocated 0) (tried to allocate 2 bytes) in Unknown on line 0 [Mon Mar 05 02:44:04 2007] [notice] Parent: child process exited with status 4294967295 -- Restarting. We watched the server closely, and when these errors happen, the CPU is low, the number of active connections is low, and there's over 2 GB of physical memory available. It's running out of *SOMETHING*, we just don't know what. It always, very reliably runs out of this something. The higher the threads, the faster (it seems). But, at least we had 3 servers serving up 128 threads each, which was temporarily acceptable. 3) We are running 7 websites, only one of which uses PHP. So we directed the 6 websites to just one server and we opened up the threads to 512, and that machine has been running without a hitch ever since. So it's just Apache + Tomcat. Hooray! Go Apache! This points to the problem definitely being in PHP, or the integration of PHP + Apache. (Please, don't send us away to go and post it on the PHP bug list just yet... All they are going to do is mark it as bogus.) 4) We updated to Apache 2.2.4. No noticeable difference unfortunately. The problem is, the traffic on the one site that uses PHP is steadily increasing. (It feels weird calling this a problem :-)) We upgraded our staging server to 2 GB of RAM, so it's the same specs as the others, and now 3 production servers are serving up just that one site. In addition we restart Apache daily. It worked fine for a while, but now it's getting worse: even with just serving 128 threads, and restarting the whole service daily (we have the redundancy for it), the above error is starting to present itself more and more frequently... I didn't mention yet that we tried for over a week to reliably reproduce this problem on our staging server, but we simply cannot. Our conclusion was that it's because when we send load to it (much bigger loads than what we normally get on production), it's all coming from the same IP. Whereas normal traffic comes from all over the place, different IPs, different browsers. We've looked at this so much already and running out of ideas and options. Help please?