|Summary:||Apache2: out of memory during heavy webdav files copy using SSL|
|Product:||Apache httpd-2||Reporter:||Alberto Avi <alberto.avi>|
|Component:||mod_ssl||Assignee:||Apache HTTPD Bugs Mailing List <bugs>|
Description Alberto Avi 2007-09-11 05:14:02 UTC
0. I'm using a Gentoo Linux box (2.6.22-gentoo-r5). I experienced the problem using mpm_prefork and mpm_worker. The problem is always reproducible using the following steps. 1.install Apache2 2.2.6(2)(09:34:11 09/10/07) (-debug doc ldap -mpm-event -mpm-itk -mpm-peruser -mpm-prefork mpm-worker -no-suexec -selinux ssl -static-modules -threads) 2.enable webdav support ipaq ~ # grep ^APACHE2_OPTS /etc/conf.d/apache2 APACHE2_OPTS="-D INFO -D LANGUAGE -D MANUAL -D SUEXEC -D PHP5 -D SSL -D DAV -D DAV_FS" and enable a webdav over ssl in the default virtual host config file: Listen 443 <VirtualHost _default_:443> ServerAdmin firstname.lastname@example.org DocumentRoot /var/www/www.mydomain.it/htdocs ServerName www.mydomain.it ErrorLog logs/ssl.err TransferLog logs/ssl.log <Directory "/var/www/www.mydomain.it/htdocs"> Options Indexes FollowSymLinks AllowOverride Limit Order allow,deny Allow from all </Directory> <IfModule mod_dav.c> Alias /dav /var/www/www.mydomain.it/htdocs/dav DavMinTimeout 600 <Location /dav> Options None Indexes Dav On LimitXMLRequestBody 1000000 DavMinTimeout 30 DavDepthInfinity off <Limit GET PUT POST DELETE PROPFIND PROPPATCH MKCOL COPY MOVE LOCK UNLOCK> AuthType Basic AuthName "WebDav Restricted Repository" AuthUserFile /var/www/www.mydomain.it/davpwd/.davpwd Require valid-user </Limit> </Location> </IfModule> </VirtualHost> 3. on your client create some data to be transfered to the webdav server alberto@nabas ~ $ mkdir ~dav alberto@nabas ~ $ cd dav alberto@nabas ~dav $ wget http://downloads.sourceforge.net/filezilla/FileZilla_3.0.0_i586-linux-gnu.tar.bz2 alberto@nabas ~dav $ wget http://heanet.dl.sourceforge.net/sourceforge/htop/htop-0.6.6.tar.gz alberto@nabas ~dav $ wget http://mirror.tomato.it/apache/httpd/httpd-2.2.6.tar.bz2 alberto@nabas ~dav $ wget http://hem.bredband.net/jpgraph/jpgraph-1.21b.tar.gz alberto@nabas ~dav $ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-220.127.116.11.tar.bz2 untar all the files 4. open Konqueror on the client and try to copy dav folder to you webdav server using ssl: webdavs://email@example.com/dav The Linux server after a while experienced an out of memory condition (I have just 384MB RAM and 512MB swap area). In the Apache log file no error was found. The web server for a about 3/4 minutes (on a PIII 700MHZ) is unavailable and then come back online without service restart. In the /var/log/messages I can see a lot of errors as this one: Sep 11 10:18:08 ipaq apache2 invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 Sep 11 10:18:19 ipaq [<c013cecc>] out_of_memory+0x69/0x189 Sep 11 10:18:32 ipaq [<c013e35f>] __alloc_pages+0x203/0x28d Sep 11 10:18:40 ipaq [<c013f866>] __do_page_cache_readahead+0xbd/0x1dd Sep 11 10:18:40 ipaq [<c01aee2e>] ext3_get_block+0x0/0xd0 Sep 11 10:18:40 ipaq [<c040e81a>] io_schedule+0x1d/0x27 Sep 11 10:18:40 ipaq [<c040e9fc>] __wait_on_bit_lock+0x4b/0x52 Sep 11 10:18:40 ipaq [<c013a218>] __lock_page+0x58/0x5e Sep 11 10:18:40 ipaq [<c013c3e4>] filemap_nopage+0x152/0x2a4 Sep 11 10:18:40 ipaq [<c014563d>] __handle_mm_fault+0x15d/0x7f3 Sep 11 10:18:40 ipaq [<c0121ba9>] run_timer_softirq+0x126/0x154 Sep 11 10:18:40 ipaq [<c011d2f4>] do_wait+0x918/0x9ae Sep 11 10:18:40 ipaq [<c0410d7d>] do_page_fault+0x20d/0x52b Sep 11 10:18:40 ipaq [<c015f6a9>] sys_select+0x161/0x187 Sep 11 10:18:40 ipaq [<c0410b70>] do_page_fault+0x0/0x52b Sep 11 10:18:40 ipaq [<c040f9aa>] error_code+0x72/0x78 Sep 11 10:18:40 ipaq [<c0400000>] xprt_connect+0xe/0xfb Sep 11 10:18:40 ipaq ======================= Sep 11 10:18:40 ipaq Mem-info: Sep 11 10:18:40 ipaq DMA per-cpu: Sep 11 10:18:40 ipaq CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Sep 11 10:18:40 ipaq Normal per-cpu: Sep 11 10:18:40 ipaq CPU 0: Hot: hi: 186, btch: 31 usd: 172 Cold: hi: 62, btch: 15 usd: 0 Sep 11 10:18:40 ipaq Active:44644 inactive:44277 dirty:0 writeback:0 unstable:0 Sep 11 10:18:40 ipaq free:995 slab:1904 mapped:8 pagetables:482 bounce:0 Sep 11 10:18:40 ipaq DMA free:1552kB min:104kB low:128kB high:156kB active:4792kB inactive:4348kB present:16256kB pages_scanned:15967 all_unreclaimable? yes Sep 11 10:18:40 ipaq lowmem_reserve: 0 364 364 Sep 11 10:18:40 ipaq Normal free:2428kB min:2384kB low:2980kB high:3576kB active:173784kB inactive:172760kB present:372872kB pages_scanned:624531 all_unreclaimable? yes Sep 11 10:18:40 ipaq lowmem_reserve: 0 0 0 Sep 11 10:18:40 ipaq DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1552kB Sep 11 10:18:40 ipaq Normal: 1*4kB 10*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2404kB Sep 11 10:18:40 ipaq Swap cache: add 132564, delete 132564, find 1783/2473, race 0+6 Sep 11 10:18:40 ipaq Free swap = 0kB Sep 11 10:18:40 ipaq Total swap = 506512kB Sep 11 10:18:40 ipaq Free swap: 0kB Sep 11 10:18:40 ipaq 98048 pages of RAM Sep 11 10:18:40 ipaq 0 pages of HIGHMEM Sep 11 10:18:40 ipaq 2364 reserved pages Sep 11 10:18:40 ipaq 2599 pages shared Sep 11 10:18:40 ipaq 0 pages swap cached Sep 11 10:18:40 ipaq 0 pages dirty Sep 11 10:18:40 ipaq 0 pages writeback Sep 11 10:18:40 ipaq 8 pages mapped Sep 11 10:18:40 ipaq 1904 pages slab Sep 11 10:18:40 ipaq 482 pages pagetables Sep 11 10:18:51 ipaq Out of memory: kill process 8698 (apache2) score 272698 or a child Sep 11 10:19:46 ipaq Killed process 8699 (apache2) Sep 11 10:20:58 ipaq syslog-ng invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 5. open Konqueror on the client and try to copy dav folder to the web server _without_ ssl: webdav://firstname.lastname@example.org/dav The copy go well. No error found.
Comment 1 Joe Orton 2007-11-05 02:15:07 UTC
Is the client running on a different machine to the server? Please follow the instructions below to generate a core dump and backtrace, knowing what httpd was doing at the time of the OOM may help: http://httpd.apache.org/dev/debugging.html#crashes
Comment 2 Alberto Avi 2007-11-06 11:10:39 UTC
Hi Joe, I confirm, the client running on a different machine to the http dav server. I tryed a lot of webdav client: Konqueror, Windows XP (I defined a network disk on webdav) and davfs2 (http://dav.sourceforge.net). All of them give me the same result: they working fine with http dav connection. All of them failed after a few minutes (2 or 3) if I try httpS dav connection. The client give me a generic network error. I followed all the instructions to configure the system and httpd to write the core dump on the file system. I generated again the same httpS dav workload I described in the first post but no core dump was written on disk. I noticed that during the file copy (about 230MB total) the resident memory of httpd server grow immoderately (I used htop to monitor it). When the RAM (384MB) on the server is full allocated the kernel start using the swap space too (512MB)! When the swap space is full allocated the system became unusable but it don't crash. I noticed another detail. During http webdav copy, httpd write progressively in its access log file: 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD /dav/linux-18.104.22.168/sound/oss/emu10k1/ecard.h HTTP/1.1" 404 - 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "PUT /dav/linux-22.214.171.124/sound/oss/emu10k1/ecard.h HTTP/1.1" 201 287 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD /dav/linux-126.96.36.199/sound/oss/emu10k1/ecard.h HTTP/1.1" 200 - 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "UNLOCK /dav/linux-188.8.131.52/sound/oss/emu10k1/ecard.h HTTP/1.1" 204 - 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD /dav/linux-184.108.40.206/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 404 - 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "PUT /dav/linux-220.127.116.11/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 201 290 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD /dav/linux-18.104.22.168/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 200 - 192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "UNLOCK /dav/linux-22.214.171.124/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 204 - During httpS webdav copy httpd dont't log neither one line! I think that during httpS webdav copy httpd put in memory all the files without write them on the disk and without update the access log file. I done another test. I tried succesfully to copy just 1MB of data using httpS webdav but no access log were written and the resident httpd memory grow of about 30MB. I retried a few time this test and finally I needed to restart httpd to free memory. It's a very strange case. Thanks a lot for your attention, Alberto.
Comment 3 Ryan McGuire 2009-10-27 09:14:18 UTC
I have what appears to be this identical issue running apache 2.2.3-22. I have a subversion repository that is accessed using mod_dav_svn. Here is the abbreviated config: <Location /svn> DAV svn SVNPath /stats/svn SSLRequireSSL AuthType Basic AuthName "***" AuthBasicProvider external AuthExternal pwauth Require valid-user </Location> Whenever a file is copied to the server (typically via 'svn import' or 'svn commit' with new file additions) it appears memory is not being freed after the transfer completes. This has never caused a crash since there is 4GB of ram and an equivalent swap partition so I have no core files. Restarting apache when I notice the low memory condition does free the memory. If necessary I can force a core of the httpd pid while in the low memory condition but I would like to avoid if possible since this is a production machine and the issue is easily reproducible. B.T.W. I see nothing strange in the logs.
Comment 4 William A. Rowe Jr. 2018-11-07 21:08:26 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd. As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd. If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question. If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with. Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.