Bug 43352

Summary: Apache2: out of memory during heavy webdav files copy using SSL
Product: Apache httpd-2 Reporter: Alberto Avi <alberto.avi>
Component: mod_sslAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED LATER    
Severity: normal CC: rmcguire
Priority: P2 Keywords: MassUpdate
Version: 2.2.6   
Target Milestone: ---   
Hardware: PC   
OS: Linux   

Description Alberto Avi 2007-09-11 05:14:02 UTC
0. I'm using a Gentoo Linux box (2.6.22-gentoo-r5).
I experienced the problem using mpm_prefork and mpm_worker.
The problem is always reproducible using the following steps.

1.install Apache2 2.2.6(2)(09:34:11 09/10/07) (-debug doc ldap -mpm-event
-mpm-itk -mpm-peruser -mpm-prefork mpm-worker -no-suexec -selinux ssl
-static-modules -threads)

2.enable webdav support
ipaq ~ # grep ^APACHE2_OPTS /etc/conf.d/apache2
APACHE2_OPTS="-D INFO -D LANGUAGE -D MANUAL -D SUEXEC -D PHP5 -D SSL -D DAV -D
DAV_FS"

and enable a webdav over ssl in the default virtual host config file:

Listen 443
<VirtualHost _default_:443>
        ServerAdmin admin@bigip.it
        DocumentRoot /var/www/www.mydomain.it/htdocs
        ServerName www.mydomain.it
        ErrorLog logs/ssl.err
        TransferLog logs/ssl.log

        <Directory "/var/www/www.mydomain.it/htdocs">
                Options Indexes FollowSymLinks
                AllowOverride Limit
                Order allow,deny
                Allow from all
        </Directory>


<IfModule mod_dav.c>
        Alias /dav /var/www/www.mydomain.it/htdocs/dav
        DavMinTimeout 600
        <Location /dav>
                Options None Indexes
                Dav On
                LimitXMLRequestBody 1000000
                DavMinTimeout 30
                DavDepthInfinity off
                <Limit GET PUT POST DELETE PROPFIND PROPPATCH MKCOL COPY MOVE
LOCK UNLOCK>
                        AuthType Basic
                        AuthName "WebDav Restricted Repository"
                        AuthUserFile /var/www/www.mydomain.it/davpwd/.davpwd
                        Require valid-user
                </Limit>
        </Location>
</IfModule>
</VirtualHost>

3. on your client create some data to be transfered to the webdav server
alberto@nabas ~ $ mkdir ~dav
alberto@nabas ~ $ cd dav
alberto@nabas ~dav $ wget
http://downloads.sourceforge.net/filezilla/FileZilla_3.0.0_i586-linux-gnu.tar.bz2
alberto@nabas ~dav $ wget
http://heanet.dl.sourceforge.net/sourceforge/htop/htop-0.6.6.tar.gz
alberto@nabas ~dav $ wget http://mirror.tomato.it/apache/httpd/httpd-2.2.6.tar.bz2
alberto@nabas ~dav $ wget http://hem.bredband.net/jpgraph/jpgraph-1.21b.tar.gz
alberto@nabas ~dav $ wget
http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.6.tar.bz2

untar all the files

4. open Konqueror on the client and try to copy dav folder to you webdav server
using ssl:

webdavs://davusr@www.mydomain.it/dav

The Linux server after a while experienced an out of memory condition (I have
just 384MB RAM and 512MB swap area).
In the Apache log file no error was found. The web server for a about 3/4
minutes (on a PIII 700MHZ) is unavailable and then come back online without
service restart.
In the /var/log/messages I can see a lot of errors as this one:

Sep 11 10:18:08 ipaq apache2 invoked oom-killer: gfp_mask=0x201d2, order=0,
oomkilladj=0
Sep 11 10:18:19 ipaq [<c013cecc>] out_of_memory+0x69/0x189
Sep 11 10:18:32 ipaq [<c013e35f>] __alloc_pages+0x203/0x28d
Sep 11 10:18:40 ipaq [<c013f866>] __do_page_cache_readahead+0xbd/0x1dd
Sep 11 10:18:40 ipaq [<c01aee2e>] ext3_get_block+0x0/0xd0
Sep 11 10:18:40 ipaq [<c040e81a>] io_schedule+0x1d/0x27
Sep 11 10:18:40 ipaq [<c040e9fc>] __wait_on_bit_lock+0x4b/0x52
Sep 11 10:18:40 ipaq [<c013a218>] __lock_page+0x58/0x5e
Sep 11 10:18:40 ipaq [<c013c3e4>] filemap_nopage+0x152/0x2a4
Sep 11 10:18:40 ipaq [<c014563d>] __handle_mm_fault+0x15d/0x7f3
Sep 11 10:18:40 ipaq [<c0121ba9>] run_timer_softirq+0x126/0x154
Sep 11 10:18:40 ipaq [<c011d2f4>] do_wait+0x918/0x9ae
Sep 11 10:18:40 ipaq [<c0410d7d>] do_page_fault+0x20d/0x52b
Sep 11 10:18:40 ipaq [<c015f6a9>] sys_select+0x161/0x187
Sep 11 10:18:40 ipaq [<c0410b70>] do_page_fault+0x0/0x52b
Sep 11 10:18:40 ipaq [<c040f9aa>] error_code+0x72/0x78
Sep 11 10:18:40 ipaq [<c0400000>] xprt_connect+0xe/0xfb
Sep 11 10:18:40 ipaq =======================
Sep 11 10:18:40 ipaq Mem-info:
Sep 11 10:18:40 ipaq DMA per-cpu:
Sep 11 10:18:40 ipaq CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:   
0, btch:   1 usd:   0
Sep 11 10:18:40 ipaq Normal per-cpu:
Sep 11 10:18:40 ipaq CPU    0: Hot: hi:  186, btch:  31 usd: 172   Cold: hi:  
62, btch:  15 usd:   0
Sep 11 10:18:40 ipaq Active:44644 inactive:44277 dirty:0 writeback:0 unstable:0
Sep 11 10:18:40 ipaq free:995 slab:1904 mapped:8 pagetables:482 bounce:0
Sep 11 10:18:40 ipaq DMA free:1552kB min:104kB low:128kB high:156kB
active:4792kB inactive:4348kB present:16256kB pages_scanned:15967
all_unreclaimable? yes
Sep 11 10:18:40 ipaq lowmem_reserve[]: 0 364 364
Sep 11 10:18:40 ipaq Normal free:2428kB min:2384kB low:2980kB high:3576kB
active:173784kB inactive:172760kB present:372872kB pages_scanned:624531
all_unreclaimable? yes
Sep 11 10:18:40 ipaq lowmem_reserve[]: 0 0 0
Sep 11 10:18:40 ipaq DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB
1*512kB 1*1024kB 0*2048kB 0*4096kB = 1552kB
Sep 11 10:18:40 ipaq Normal: 1*4kB 10*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB
0*512kB 0*1024kB 1*2048kB 0*4096kB = 2404kB
Sep 11 10:18:40 ipaq Swap cache: add 132564, delete 132564, find 1783/2473, race 0+6
Sep 11 10:18:40 ipaq Free swap  = 0kB
Sep 11 10:18:40 ipaq Total swap = 506512kB
Sep 11 10:18:40 ipaq Free swap:            0kB
Sep 11 10:18:40 ipaq 98048 pages of RAM
Sep 11 10:18:40 ipaq 0 pages of HIGHMEM
Sep 11 10:18:40 ipaq 2364 reserved pages
Sep 11 10:18:40 ipaq 2599 pages shared
Sep 11 10:18:40 ipaq 0 pages swap cached
Sep 11 10:18:40 ipaq 0 pages dirty
Sep 11 10:18:40 ipaq 0 pages writeback
Sep 11 10:18:40 ipaq 8 pages mapped
Sep 11 10:18:40 ipaq 1904 pages slab
Sep 11 10:18:40 ipaq 482 pages pagetables
Sep 11 10:18:51 ipaq Out of memory: kill process 8698 (apache2) score 272698 or
a child
Sep 11 10:19:46 ipaq Killed process 8699 (apache2)
Sep 11 10:20:58 ipaq syslog-ng invoked oom-killer: gfp_mask=0x201d2, order=0,
oomkilladj=0


5. open Konqueror on the client and try to copy dav folder to the web server
_without_ ssl:

webdav://davusr@www.mydomain.it/dav

The copy go well. No error found.
Comment 1 Joe Orton 2007-11-05 02:15:07 UTC
Is the client running on a different machine to the server?

Please follow the instructions below to generate a core dump and backtrace,
knowing what httpd was doing at the time of the OOM may help:

http://httpd.apache.org/dev/debugging.html#crashes
Comment 2 Alberto Avi 2007-11-06 11:10:39 UTC
Hi Joe,

I confirm, the client running on a different machine to the http dav server.
I tryed a lot of webdav client: Konqueror, Windows XP (I defined a network disk
on webdav) and davfs2 (http://dav.sourceforge.net).
All of them give me the same result: they working fine with http dav connection.
All of them failed after a few minutes (2 or 3) if I try httpS dav connection.
The client give me a generic network error.

I followed all the instructions to configure the system and httpd to write the
core dump on the file system.
I generated again the same httpS dav workload I described in the first post but
no core dump was written on disk. I noticed that during the file copy (about
230MB total) the resident memory of httpd server grow immoderately (I used htop
to monitor it). When the RAM (384MB) on the server is full allocated the kernel
start using the swap space too (512MB)! When the swap space is full allocated
the system became unusable but it don't crash.

I noticed another detail. During http webdav copy, httpd write progressively in
its access log file:

192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD
/dav/linux-2.6.22.6/sound/oss/emu10k1/ecard.h HTTP/1.1" 404 -
192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "PUT
/dav/linux-2.6.22.6/sound/oss/emu10k1/ecard.h HTTP/1.1" 201 287
192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD
/dav/linux-2.6.22.6/sound/oss/emu10k1/ecard.h HTTP/1.1" 200 -
192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "UNLOCK
/dav/linux-2.6.22.6/sound/oss/emu10k1/ecard.h HTTP/1.1" 204 -
192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD
/dav/linux-2.6.22.6/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 404 -
192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "PUT
/dav/linux-2.6.22.6/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 201 290
192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "HEAD
/dav/linux-2.6.22.6/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 200 -
192.168.1.1 - davusr [05/Nov/2007:19:34:38 +0100] "UNLOCK
/dav/linux-2.6.22.6/sound/oss/emu10k1/voicemgr.h HTTP/1.1" 204 -


During httpS webdav copy httpd dont't log neither one line!
I think that during httpS webdav copy httpd put in memory all the files without
write them on the disk and without update the access log file.

I done another test.
I tried succesfully to copy just 1MB of data using httpS webdav but no access
log were written and the resident httpd memory grow of about 30MB. I retried a
few time this test and finally I needed to restart httpd to free memory.

It's a very strange case.

Thanks a lot for your attention,

Alberto.

Comment 3 Ryan McGuire 2009-10-27 09:14:18 UTC
I have what appears to be this identical issue running apache 2.2.3-22. I have a subversion repository that is accessed using mod_dav_svn.

Here is the abbreviated config:

<Location /svn>
    DAV svn
    SVNPath /stats/svn
    SSLRequireSSL
    AuthType Basic
    AuthName "***"
    AuthBasicProvider external
    AuthExternal pwauth
    Require valid-user
</Location>

Whenever a file is copied to the server (typically via 'svn import' or 'svn commit' with new file additions) it appears memory is not being freed after the transfer completes.

This has never caused a crash since there is 4GB of ram and an equivalent swap partition so I have no core files. Restarting apache when I notice the low memory condition does free the memory.

If necessary I can force a core of the httpd pid while in the low memory condition but I would like to avoid if possible since this is a production machine and the issue is easily reproducible.

B.T.W. I see nothing strange in the logs.
Comment 4 William A. Rowe Jr. 2018-11-07 21:08:26 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.