I have a page which people POST data too. I have blocked large blocks in .htaccess with "Deny From". They seem to have archived the pages though, or people are turning on proxies to post, but as soon as they POST from a banned IP, I get segfault in the logs and this error "Additionally an error was encounted while..." for the 403 request itself, which is just a plain .html page. This is easy to reproduce, any POST form, surf to it, ban your IP in .htaccess, hit "submit" and you will see a segfault. Thank you
I cannot reproduce this. Please attach a backtrace of your segfault. For hints how to create a proper backtrace please have a look at http://httpd.apache.org/dev/debugging.html.
(In reply to comment #1) > I cannot reproduce this. Please attach a backtrace of your segfault. For hints > how to create a proper backtrace please have a look at > http://httpd.apache.org/dev/debugging.html. I hope this strace is enough, I can't make this drop a core, it's also very hard to attach to the child that's going to die (I suppose I could attach with gdb and start killing children like I did for this one). It only does this on POSTs to forms from IPs that are blocked, not on normal page requests even if the IP is blocked. I am using suexec, chroot()'d, fastcgi, php. But, this is a simple 403 request to a ErrorDocument /403_error.html, so I don't think any of that would interfere, and I'm almost positive it was doing this long before fastcgi and it's suexec wrapper. It looks like it's segfaulting when it writes to the log? At any rate, there is no log entry for the blocked IP. (Also, I mangled the IPs and hostname, prolly not well.. but meh.) Thank you! ------- STRACE FROM THE CHILD WHEN IT DIED ------- 15041 poll([{fd=4, events=POLLIN}, {fd=3, events=POLLIN, revents=POLLIN}], 2, -1) = 1 15041 accept(3, {sa_family=AF_INET, sin_port=htons(43259), sin_addr=inet_addr("21.21.14.20")}, [16]) = 69 15041 semop(60719148, 0x402e7fd6, 1) = 0 15041 gettimeofday({1153764525, 36922}, NULL) = 0 15041 getsockname(69, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("6.3.21.33")}, [16]) = 0 15041 gettimeofday({1153764525, 37085}, NULL) = 0 15041 brk(0) = 0x82f0000 15041 brk(0x82f2000) = 0x82f2000 15041 fcntl64(69, F_GETFL) = 0x2 (flags O_RDWR) 15041 fcntl64(69, F_SETFL, O_RDWR|O_NONBLOCK) = 0 15041 brk(0) = 0x82f2000 15041 brk(0x82f4000) = 0x82f4000 15041 read(69, 0x82f1e88, 8000) = -1 EAGAIN (Resource temporarily unavailable) 15041 poll([{fd=69, events=POLLIN, revents=POLLIN}], 1, 300000) = 1 15041 read(69, "POST /misc/add_review.php?id=51 "..., 8000) = 1116 15041 gettimeofday({1153764525, 39239}, NULL) = 0 15041 semop(60686379, 0x402e7fd0, 1) = 0 15041 semop(60686379, 0x402e7fd6, 1) = 0 15041 semop(60686379, 0x402e7fd0, 1) = 0 15041 semop(60686379, 0x402e7fd6, 1) = 0 15041 gettimeofday({1153764525, 39580}, NULL) = 0 15041 gettimeofday({1153764525, 39610}, NULL) = 0 15041 stat64("/www/n/d/ndhost.net/html/misc/add_review.php", {st_mode=S_IFREG|0755, st_size=14137, ...}) = 0 15041 open("/www/n/d/ndhost.net/html/.htaccess", O_RDONLY) = 71 15041 brk(0) = 0x82f4000 15041 brk(0x82f6000) = 0x82f6000 15041 fstat64(71, {st_mode=S_IFREG|0775, st_size=6550, ...}) = 0 15041 brk(0) = 0x82f6000 15041 brk(0x82f9000) = 0x82f9000 15041 read(71, "#\n# Apache/PHP/site settings:\n#\n"..., 4096) = 4096 15041 brk(0) = 0x82f9000 15041 brk(0x82fc000) = 0x82fc000 15041 read(71, "doctor/(.*)$\n RewriteRule ^(.*)"..., 4096) = 2454 15041 brk(0) = 0x82fc000 15041 brk(0x82fe000) = 0x82fe000 15041 read(71, "", 4096) = 0 15041 brk(0) = 0x82fe000 15041 brk(0x8300000) = 0x8300000 15041 brk(0) = 0x8300000 15041 brk(0x8301000) = 0x8301000 15041 brk(0) = 0x8301000 15041 brk(0x8303000) = 0x8303000 15041 close(71) = 0 15041 open("/www/n/d/ndhost.net/html/misc/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory) 15041 open("/www/n/d/ndhost.net/html/misc/add_review.php/.htaccess", O_RDONLY) = -1 ENOTDIR (Not a directory) 15041 brk(0) = 0x8303000 15041 brk(0x8305000) = 0x8305000 15041 gettimeofday({1153764525, 42933}, NULL) = 0 15041 write(15, "[Mon Jul 24 14:08:45 2006] [erro"..., 187) = 187 15041 semop(60686379, 0x402e7fd0, 1) = 0 15041 semop(60686379, 0x402e7fd6, 1) = 0 15041 semop(60686379, 0x402e7fd0, 1) = 0 15041 semop(60686379, 0x402e7fd6, 1) = 0 15041 stat64("/www/n/d/ndhost.net/html/403_error.html", {st_mode=S_IFREG|0775, st_size=524, ...}) = 0 15041 open("/www/n/d/ndhost.net/html/403_error.html/.htaccess", O_RDONLY) = -1 ENOTDIR (Not a directory) 15041 gettimeofday({1153764525, 43353}, NULL) = 0 15041 write(15, "[Mon Jul 24 14:08:45 2006] [erro"..., 182) = 182 15041 brk(0) = 0x8305000 15041 brk(0x8307000) = 0x8307000 15041 brk(0) = 0x8307000 15041 brk(0x8309000) = 0x8309000 15041 brk(0) = 0x8309000 15041 brk(0x830b000) = 0x830b000 15041 read(69, 0x8308a30, 8000) = -1 EAGAIN (Resource temporarily unavailable) 15041 writev(69, [{"HTTP/1.1 403 Forbidden\r\nDate: Mo"..., 201}, {"<!DOCTYPE HTML PUBLIC \"-//IETF//"..., 402}], 2) = 603 15041 gettimeofday({1153764525, 43948}, NULL) = 0 15041 write(43, "21.21.14.20 - - [24/Jul/2006"..., 228) = 228 15041 --- SIGSEGV (Segmentation fault) @ 0 (0) --- 15041 chdir("/tmp") = 0 15041 rt_sigaction(SIGSEGV, {SIG_DFL}, {SIG_DFL}, 8) = 0 15041 getpid() = 15041 15041 getpid() = 15041 15041 kill(15041, SIGSEGV) = 0 15041 sigreturn() = ? (mask now [RTMIN]) 15041 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
Sorry but the strace does not really help. It seems to me that it tries to write a core dump to the /tmp directory. Please set ulimit -c to unlimited in apachectl. On Linux systems ulimit -c often defaults to 0 which prevents core dumping.
Alright, got it to core, and gdb "bt full" I hope this is more helpful!! ---------------- gdb /usr/local/apache/bin/httpd /tmp/core.21101 GNU gdb Red Hat Linux (5.3post-0.20021129.18rh) Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... Core was generated by `/usr/local/apache/bin/httpd -f /etc/conf/httpd/httpd2.conf'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/ssl/lib/libssl.so.0.9.7...done. Loaded symbols for /usr/local/ssl/lib/libssl.so.0.9.7 Reading symbols from /usr/local/ssl/lib/libcrypto.so.0.9.7...done. Loaded symbols for /usr/local/ssl/lib/libcrypto.so.0.9.7 Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2...done. Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2 Reading symbols from /usr/kerberos/lib/libkrb5.so.3...done. Loaded symbols for /usr/kerberos/lib/libkrb5.so.3 Reading symbols from /usr/kerberos/lib/libcom_err.so.3...done. Loaded symbols for /usr/kerberos/lib/libcom_err.so.3 Reading symbols from /usr/kerberos/lib/libk5crypto.so.3...done. Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/local/apache-2.0.55/lib/libaprutil-0.so.0...done. Loaded symbols for /usr/local/apache-2.0.55/lib/libaprutil-0.so.0 Reading symbols from /usr/lib/libgdbm.so.2...done. Loaded symbols for /usr/lib/libgdbm.so.2 Reading symbols from /lib/libdb-4.0.so...done. Loaded symbols for /lib/libdb-4.0.so Reading symbols from /usr/lib/libexpat.so.0...done. Loaded symbols for /usr/lib/libexpat.so.0 Reading symbols from /usr/local/apache-2.0.55/lib/libapr-0.so.0...done. Loaded symbols for /usr/local/apache-2.0.55/lib/libapr-0.so.0 ---Type <return> to continue, or q <return> to quit--- Reading symbols from /lib/librt.so.1...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libnsl.so.1...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/libpthread.so.0...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /usr/local/apache/modules/mod_frontpage.so...done. Loaded symbols for /usr/local/apache/modules/mod_frontpage.so Reading symbols from /usr/local/apache/modules/mod_watch.so...done. Loaded symbols for /usr/local/apache/modules/mod_watch.so Reading symbols from /usr/local/apache/modules/mod_fastcgi.so...done. Loaded symbols for /usr/local/apache/modules/mod_fastcgi.so #0 0x0809edbb in ap_strcasecmp_match (str=0x0, expected=0x404dcb06 "text/html") at util.c:203 203 if (!str[x] && expected[y] != '*') (gdb) bt full #0 0x0809edbb in ap_strcasecmp_match (str=0x0, expected=0x404dcb06 "text/html") at util.c:203 x = 0 y = 0 #1 0x404d9a8b in watchCounters (r=0x82f3eb0) at mod_watch.c:554 data = (struct shEntry *) 0x82f3a90 wc = (watchConnectionIO *) 0x82f2210 #2 0x404da35b in watchLog (r=0x82f3eb0) at mod_watch.c:916 key = 0x1 <Address 0x1 out of bounds> data = (struct shEntry *) 0x20 dconf = (struct watchConfDir *) 0x811cf00 #3 0x080a74d2 in ap_run_log_transaction (r=0x82f3eb0) at protocol.c:1549 pHook = (ap_LINK_log_transaction_t *) 0x0 n = 1 rv = 0 #4 0x08082771 in ap_process_request (r=0x82f3eb0) at http_request.c:274 access_status = 0 #5 0x0807e6a1 in ap_process_http_connection (c=0x82e9cc0) at http_core.c:251 r = (request_rec *) 0x82f3eb0 csd_set = 1 csd = (apr_socket_t *) 0x82e9be8 #6 0x080a29b2 in ap_run_process_connection (c=0x82e9cc0) at connection.c:43 pHook = (ap_LINK_process_connection_t *) 0x5 ---Type <return> to continue, or q <return> to quit--- n = 0 rv = 5 #7 0x08097fbb in child_main (child_num_arg=5) at prefork.c:610 ptrans = (apr_pool_t *) 0x82e9bb0 allocator = (apr_allocator_t *) 0x82e7b20 current_conn = (conn_rec *) 0x82e9cc0 status = 137272512 i = 0 lr = (ap_listen_rec *) 0x82e9cc0 curr_pollfd = 0 last_pollfd = 1 pollset = (apr_pollfd_t *) 0x82e7c50 offset = 5 csd = (void *) 0x82e9be8 sbh = (ap_sb_handle_t *) 0x82e7c20 rv = 5 bucket_alloc = (apr_bucket_alloc_t *) 0x82ede60 #8 0x080980d8 in make_child (s=0x80e4218, slot=0) at prefork.c:704 pid = 0 #9 0x080981bf in startup_children (number_to_start=10) at prefork.c:722 i = 0 #10 0x080988cd in ap_mpm_run (_pconf=0x0, plog=0x8117130, s=0x80e4218) at prefork.c:941 ---Type <return> to continue, or q <return> to quit--- index = 137272512 remaining_children_to_start = 10 rv = 5 #11 0x0809d956 in main (argc=3, argv=0xbfffeaf4) at main.c:618 c = 102 'f' configtestonly = 0 confname = 0xbfffff50 "/etc/conf/httpd/httpd2.conf" def_server_root = 0x80c99c4 "/usr/local/apache-2.0.55" temp_error_log = 0x0 process = (process_rec *) 0x80dd0c8 server_conf = (server_rec *) 0x80e4218 pglobal = (apr_pool_t *) 0x80dd048 pconf = (apr_pool_t *) 0x80df050 plog = (apr_pool_t *) 0x8117130 ptemp = (apr_pool_t *) 0x811d148 pcommands = (apr_pool_t *) 0x80e1058 opt = (apr_getopt_t *) 0x80e10f0 rv = 5 mod = (module **) 0x80e4218 optarg = 0xbfffff50 "/etc/conf/httpd/httpd2.conf" signal_server = (apr_OFN_ap_signal_server_t *) 0 #12 0x403c462d in __libc_start_main () from /lib/libc.so.6 No symbol table info available. (gdb) quit ]0;root@berkana:/tmp [root@berkana tmp]#
(In reply to comment #4) > Alright, got it to core, and gdb "bt full" I hope this is more helpful!! Thanks, yes it is, but maybe not in the way you hoped :-). > Reading symbols from /usr/local/apache/modules/mod_watch.so...done. > Loaded symbols for /usr/local/apache/modules/mod_watch.so > Reading symbols from /usr/local/apache/modules/mod_fastcgi.so...done. > Loaded symbols for /usr/local/apache/modules/mod_fastcgi.so > #0 0x0809edbb in ap_strcasecmp_match (str=0x0, > expected=0x404dcb06 "text/html") at util.c:203 > 203 if (!str[x] && expected[y] != '*') > (gdb) bt full > #0 0x0809edbb in ap_strcasecmp_match (str=0x0, > expected=0x404dcb06 "text/html") at util.c:203 > x = 0 > y = 0 > #1 0x404d9a8b in watchCounters (r=0x82f3eb0) at mod_watch.c:554 > data = (struct shEntry *) 0x82f3a90 > wc = (watchConnectionIO *) 0x82f2210 The bug is in your third party module mod_watch. It calls ap_strcasecmp_match with a NULL pointer as first parameter. This causes a SIGSEGV. => invalid
(In reply to comment #5) > (In reply to comment #4) > > Alright, got it to core, and gdb "bt full" I hope this is more helpful!! > > Thanks, yes it is, but maybe not in the way you hoped :-). > As long as I know what it is, I'm happy. =) Thanks, and sorry for making you troubleshoot something that wasn't really Apache!