Bug 49437

Summary: apache and mod_auth_basic segmentation fault
Product: Apache httpd-2 Reporter: erno.kovacs
Component: mod_authAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: critical Keywords: FixedInTrunk
Priority: P2    
Version: 2.2.15   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Attachments: Proposed patch

Description erno.kovacs 2010-06-14 14:19:34 UTC
# /usr/local/apache/bin/httpd -v
Server version: Apache/2.2.15 (Unix)
Server built:   Apr  6 2010 11:11:49

# /usr/local/apache/bin/httpd -l
Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_default.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_default.c
  mod_auth_basic.c
  mod_cache.c
  mod_filter.c
  mod_log_config.c
  mod_env.c
  mod_setenvif.c
  mod_version.c
  mod_ssl.c
  worker.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_cgid.c
  mod_cgi.c
  mod_negotiation.c
  mod_dir.c
  mod_actions.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c


# cat /usr/local/apache/conf/httpd-test.conf
############################################################### alapveto adatok begin
ServerRoot "/usr/local/apache"
Listen ip_to_listen_on
User nobody
Group nogroup
ServerAdmin tech@monstermedia.hu
ServerName monstermedia.hu:80
DocumentRoot "/docroot"
DirectoryIndex index.html index.htm index.php
ExtendedStatus on
ServerTokens Prod
ServerSignature Off
DefaultType text/plain
Timeout 30
Keepalive on
MaxKeepAliveRequests 100
KeepAliveTimeout 5
UseCanonicalName Off
AccessFilename .htaccess
HostnameLookups off
CoreDumpDirectory /tmp
############################################################### alapveto adatok end

############################################################### MPM begin
<IfModule mpm_worker_module>
  MaxClients 400
  ServerLimit 16
  StartServers 2
  MinSpareThreads 25
  MaxSpareThreads 75
  ThreadsPerChild 25
  ThreadStackSize 131072
  MaxRequestsPerChild  10000
</IfModule>
############################################################### MPM end

############################################################### access control begin
<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory "/docroot">
    Options -Indexes FollowSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>
############################################################### access control end

################################################################### logging begin
PidFile "logs/httpd-test.pid"
ErrorLog "logs/error_log-test"
LogLevel error
LogFormat "%h %V %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\"" myvcommon
CustomLog "/usr/local/apache/logs/access_log" myvcommon
#################################################################### logging end

#################################################################### alias begin
Alias /icons/ "/usr/local/apache/icons/"
<Directory "/usr/local/apache/icons">
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>
#################################################################### alias end


# ulimit -c unlimited
# /usr/local/apache/bin/httpd -f /usr/local/apache/conf/httpd-test.conf

# cat /docroot/.htaccess
AuthType Basic
AuthName "Password Required"
AuthUserFile /docroot/.htpasswd
Require valid-user

# cat /web/web/host/netlogic.hu/pages/stats/.htpasswd
adminuser:sensitivepasswordhash

then sending a request with a browser causes httpd process to crash.


# cat /usr/local/apache/logs/error_log-test
[Mon Jun 14 20:11:47 2010] [notice] Apache/2.2.15 (Unix) mod_ssl/2.2.15 OpenSSL/0.9.8g configured -- resuming normal operations
[Mon Jun 14 20:11:59 2010] [notice] child pid 669 exit signal Segmentation fault (11), possible coredump in /tmp

access_log-test is empty.



backtrace:

# cd /tmp
# gdb /usr/local/apache/bin/httpd 669
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
Attaching to program: /usr/local/apache/bin/httpd, process 669
ptrace: No such process.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/local/apache/lib/libaprutil-1.so.0...done.
Loaded symbols for /usr/local/apache/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libexpat.so.1...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /usr/local/apache/lib/libapr-1.so.0...done.
Loaded symbols for /usr/local/apache/lib/libapr-1.so.0
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Failed to read a valid object file image from memory.
Core was generated by `./httpd -f /usr/local/apache/conf/httpd-test.conf'.
Program terminated with signal 11, Segmentation fault.
[New process 671]
[New process 698]
[New process 697]
[New process 696]
[New process 695]
[New process 693]
[New process 692]
[New process 691]
[New process 690]
[New process 689]
[New process 688]
[New process 687]
[New process 686]
[New process 685]
[New process 684]
[New process 683]
[New process 682]
[New process 681]
[New process 680]
[New process 679]
[New process 678]
[New process 677]
[New process 676]
[New process 675]
[New process 674]
[New process 673]
[New process 669]
#0  0x00007f729ff3ddd6 in apr_password_validate (passwd=0x1506722 "sensitivepass", hash=0x1507bc8 "sensitivepasswordhash")
    at crypto/apr_md5.c:705
705     crypto/apr_md5.c: No such file or directory.
        in crypto/apr_md5.c
(gdb) bt full
#0  0x00007f729ff3ddd6 in apr_password_validate (passwd=0x1506722 "sensitivepass", hash=0x1507bc8 "sensitivepasswordhash")
    at crypto/apr_md5.c:705
        sample = "\000\000\000\000\000\000\000\000Úá\234 r\177\000\000\005", '\0' <repeats 15 times>, "\001\000\000\000\000\000\000\000P\177=A\000\000\000\000 >ó\237r\177\000\000Č{P\001\000\000\000\000P\237=A\000\000\000\000\"@\235 r\177\000\000\003", '\0' <repeats 15 times>, "\002\000\000\000\000\000\000\000Č{P\001\000\000\000\000\"gP\001\000\000\000"
        crypt_pw = <value optimized out>
#1  0x000000000044ef08 in check_password (r=0x14fdb90, user=0x1506730 "adminuser", password=0x1506722 "sensitivepass")
    at mod_authn_file.c:103
        conf = <value optimized out>
        f = (ap_configfile_t *) 0x1507b58
        l = "adminuser:sensitivepasswordhash", '\0' <repeats 6569 times>, "\220\232=A", '\0' <repeats 16 times>, "\002", '\0' <repeats 227 times>, "\030ŰO\001\000\000\000\000p_P\001\000\000\000\000p_P\001\000\000\000\000h_P\001", '\0' <repeats 12 times>, "8ŤG\001\000\000\000\000H\rŻ\237r\177\000\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000ř_P\001\000\000\000\000 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\030ŰO\001\000\000\000\000ř\\P\001\000\000\000\000\v", '\0' <repeats 15 times>, "Ř\\P\001\000\000\000\000°\234=A\000\000\000\000ĐfP\001\000"...
        status = <value optimized out>
        file_password = 0x1507bc8 "sensitivepasswordhash"
#2  0x0000000000450356 in authenticate_basic_user (r=0x14fdb90) at mod_auth_basic.c:230
        provider = (const authn_provider *) 0x49f7a0
        current_auth = <value optimized out>
        res = <value optimized out>
        auth_result = <value optimized out>
        current_provider = (authn_provider_list *) 0x0
#3  0x000000000043ed43 in ap_run_check_user_id (r=0x14fdb90) at request.c:71
        n = 1
        rv = 2
#4  0x00000000004410f4 in ap_process_request_internal (r=0x14fdb90) at request.c:214
        file_req = 0
        access_status = 0
#5  0x000000000046f748 in ap_process_request (r=0x14fdb90) at http_request.c:280
        access_status = 3
#6  0x000000000046c778 in ap_process_http_connection (c=0x14f77b8) at http_core.c:190
        r = (request_rec *) 0x14fdb90
        csd = (apr_socket_t *) 0x0
#7  0x000000000044b3f3 in ap_run_process_connection (c=0x14f77b8) at connection.c:43
        n = 0
        rv = 2
#8  0x000000000048dd41 in worker_thread (thd=0x14a7410, dummy=<value optimized out>) at worker.c:544
        process_slot = 0
        thread_slot = 0
        csd = (apr_socket_t *) 0x14f75a0
        bucket_alloc = (apr_bucket_alloc_t *) 0x14fbb08
        last_ptrans = <value optimized out>
        ptrans = (apr_pool_t *) 0x14f7528
        rv = <value optimized out>
        is_idle = <value optimized out>
---Type <return> to continue, or q <return> to quit---
#9  0x00007f729f483fc7 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#10 0x00007f729eff559d in clone () from /lib/libc.so.6
No symbol table info available.
#11 0x0000000000000000 in ?? ()
No symbol table info available.
Comment 1 Nick Kew 2010-07-04 22:35:54 UTC
Confused.  The crash comes from a strncmp, and seems to imply that your hash is a dangling pointer and segfaults when dereferenced.  Any chance of de-obfuscating the stack dump (create a non-sensitive dummy user so you can post it literally without revealing anything that matters).

Also your configuration doesn't look as if it should work in 2.2.  But that's a different issue, and a dicky config shouldn't cause segfault!
Comment 2 erno.kovacs 2010-07-05 05:02:14 UTC
(In reply to comment #1)
> Confused.  The crash comes from a strncmp, and seems to imply that your hash is
> a dangling pointer and segfaults when dereferenced.  Any chance of
> de-obfuscating the stack dump (create a non-sensitive dummy user so you can
> post it literally without revealing anything that matters).
> 
> Also your configuration doesn't look as if it should work in 2.2.  But that's a
> different issue, and a dicky config shouldn't cause segfault!

it must be some mpm-worker/threading issue, since i moved to mpm-prefork, the problem doesnt exist anymore.
Comment 3 Alexey Asemov 2010-07-09 12:19:20 UTC
Confirming this issue with Apache 2.2.15. mpm_worker / mod_auth_basic crashes on <successful> authentication (on failing auth, everything goes smooth, mpm_prefork is okay, too).
Comment 4 Alexey Asemov 2010-07-09 14:35:30 UTC
I found the reason and the (probable) solution.

It's related to the ThreadStackSize. I am using reduced TSS (131072 bytes, to be precise) to reduce memory footprint of the frontend. The crypt_data structure is just too large to fit on stack (it is more than 131072 bytes in size).

While increasing TSS is a viable option, I still do think keeping several tens of kilobytes of temporary data on stack sounds nonsense. So, I made a patch to malloc the structure insted (feel free to correct me there if something more effective can be used instead of malloc).

Attaching a patch.
Comment 5 Alexey Asemov 2010-07-09 14:37:11 UTC
Created attachment 25737 [details]
Proposed patch
Comment 6 Alexey Asemov 2010-07-09 14:50:08 UTC
Oh, and the malloc is not always thread safe. While it works for me, the patch surely requires cleanup.
Comment 7 Stefan Fritsch 2010-10-07 13:22:13 UTC
The fact that 128k stack size will crash on Linux is now documented in trunk and 2.2.x: r1005529 and r1005531
Comment 8 Christophe JAILLET 2013-06-02 05:16:30 UTC
This function has been copied from apr_md5.c to apr_passwd.c in r1357772.
This has been fixed later on in r1460243.

This is available in apr-util 1.5.2