When apache is running for a long time a systematics segfault appear on log_io mod. This behaviour appear on several severs and seem systematics. An apachectl graceful can't block this issue, I must perfom an apache restart in order to avoid this bug. Here is the error message displayed in kernel logs : kernel: httpd[50551]: segfault at 7fc0b9b51b60 ip 00007fc0b9b51b60 sp 00007fc0af7fda38 error 7 in mod_logio.so[7fc0b9b50000+2000] The at and ip value are already the same on all occurences. Server version: Apache/2.4.6 (Red Hat Enterprise Linux) Server built: Mar 8 2017 05:09:47 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count)
Thanks for the report. I'd ask you a couple of questions: 1) Is https://httpd.apache.org/docs/current/mod/mpm_common.html#coredumpdirectory configured? If so, would it be possible to generate a coredump to analyze? 2) Is there a way for us to reproduce the issue? Specific httpd configuration, etc.. 3) Have you tried with more recent versions of httpd? Number 1) is surely the most important one :)
Thanks for your answers. 1) Core dump directory isn't activated in production context, we will activate it in an other environment and try to reproduce the behaviour. 2) Here is the apache configuration : - Vhost : <VirtualHost *:80> ServerName sante-espaceparticuliers-lppj.site.com DocumentRoot "/projet/drupal/extranet" <Directory "/projet/drupal/extranet"> AllowOverride All Require all granted </Directory> ErrorLog "/var/log/httpd/sante-espaceparticuliers-lppj_site_com_error.log" ServerSignature Off CustomLog "/var/log/httpd/sante-espaceparticuliers-lppj_site_com_access.log" "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%D\"" ServerAlias sante-espaceparticuliers-lppj.site.com <FilesMatch "\.php$"> SetHandler "proxy:fcgi://127.0.0.1:9000" </FilesMatch> <Directory /usr/share/fpm> AllowOverride None Require all granted </Directory> Alias /status /usr/share/fpm/status.html </VirtualHost> - apache configuration : ServerTokens OS ServerSignature On TraceEnable On ServerName "server.pub-pp.site-hds.may.jv-ig" ServerRoot "/etc/httpd" PidFile run/httpd.pid Timeout 120 KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 15 LimitRequestFieldSize 8190 User apache Group apache AccessFileName .htaccess <FilesMatch "^\.ht"> Require all denied </FilesMatch> <Directory /> Options FollowSymLinks AllowOverride None </Directory> HostnameLookups Off ErrorLog "/var/log/httpd/error_log" LogLevel warn EnableSendfile On Include "/etc/httpd/conf.modules.d/*.load" Include "/etc/httpd/conf.modules.d/*.conf" Include "/etc/httpd/conf/ports.conf" LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common LogFormat "%{Referer}i -> %U" referer LogFormat "%{User-agent}i" agent LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" forwarded IncludeOptional "/etc/httpd/conf.d/*.conf" Listen 80 - modules <IfModule alias_module> Alias /icons/ "/usr/share/httpd/icons/" <Directory "/usr/share/httpd/icons"> Options Indexes MultiViews AllowOverride None Require all granted </Directory> </IfModule> IndexOptions FancyIndexing VersionSort HTMLTable NameWidth=* DescriptionWidth=* Charset=UTF-8 AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip x-bzip2 AddIconByType (TXT,/icons/text.gif) text/* AddIconByType (IMG,/icons/image2.gif) image/* AddIconByType (SND,/icons/sound2.gif) audio/* AddIconByType (VID,/icons/movie.gif) video/* AddIcon /icons/binary.gif .bin .exe AddIcon /icons/binhex.gif .hqx AddIcon /icons/tar.gif .tar AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip AddIcon /icons/a.gif .ps .ai .eps AddIcon /icons/layout.gif .html .shtml .htm .pdf AddIcon /icons/text.gif .txt AddIcon /icons/c.gif .c AddIcon /icons/p.gif .pl .py AddIcon /icons/f.gif .for AddIcon /icons/dvi.gif .dvi AddIcon /icons/uuencoded.gif .uu AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl AddIcon /icons/tex.gif .tex AddIcon /icons/bomb.gif /core AddIcon (SND,/icons/sound2.gif) .ogg AddIcon (VID,/icons/movie.gif) .ogm AddIcon /icons/back.gif .. AddIcon /icons/hand.right.gif README AddIcon /icons/folder.gif ^^DIRECTORY^^ AddIcon /icons/blank.gif ^^BLANKICON^^ AddIcon /icons/odf6odt-20x22.png .odt AddIcon /icons/odf6ods-20x22.png .ods AddIcon /icons/odf6odp-20x22.png .odp AddIcon /icons/odf6odg-20x22.png .odg AddIcon /icons/odf6odc-20x22.png .odc AddIcon /icons/odf6odf-20x22.png .odf AddIcon /icons/odf6odb-20x22.png .odb AddIcon /icons/odf6odi-20x22.png .odi AddIcon /icons/odf6odm-20x22.png .odm AddIcon /icons/odf6ott-20x22.png .ott AddIcon /icons/odf6ots-20x22.png .ots AddIcon /icons/odf6otp-20x22.png .otp AddIcon /icons/odf6otg-20x22.png .otg AddIcon /icons/odf6otc-20x22.png .otc AddIcon /icons/odf6otf-20x22.png .otf AddIcon /icons/odf6oti-20x22.png .oti AddIcon /icons/odf6oth-20x22.png .oth DefaultIcon /icons/unknown.gif ReadmeName README.html HeaderName HEADER.html IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t DAVLockDB "/var/lib/dav/lockdb" AddOutputFilterByType DEFLATE application/json AddOutputFilterByType DEFLATE application/rss+xml AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE text/html text/plain text/xml DeflateFilterNote Input instream DeflateFilterNote Output outstream DeflateFilterNote Ratio ratio DirectoryIndex index.html index.html.var index.cgi index.pl index.php index.xhtml <IfModule mpm_event_module> ServerLimit 64 StartServers 2 MaxClients 350 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 ThreadLimit 64 ListenBacklog 1511 </IfModule> <Location /server-info> SetHandler server-info Require ip IP_VALUES_SUPPRESSED </Location> TypesConfig /etc/mime.types AddType application/x-compress .Z AddType application/x-gzip .gz .tgz AddType application/x-bzip2 .bz2 AddLanguage ca .ca AddLanguage cs .cz .cs AddLanguage da .dk AddLanguage de .de AddLanguage el .el AddLanguage en .en AddLanguage eo .eo AddLanguage es .es AddLanguage et .et AddLanguage fr .fr AddLanguage he .he AddLanguage hr .hr AddLanguage it .it AddLanguage ja .ja AddLanguage ko .ko AddLanguage ltz .ltz AddLanguage nl .nl AddLanguage nn .nn AddLanguage no .no AddLanguage pl .po AddLanguage pt .pt AddLanguage pt-BR .pt-br AddLanguage ru .ru AddLanguage sv .sv AddLanguage zh-CN .zh-cn AddLanguage zh-TW .zh-tw AddHandler type-map var AddOutputFilter INCLUDES .shtml AddType text/html .shtml MIMEMagicFile "/etc/httpd/conf/magic" LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW ForceLanguagePriority Prefer Fallback <IfModule mod_proxy.c> ProxyRequests Off <Proxy *> Require ip IP_VALUES_SUPPRESSED </Proxy> ProxyVia On </IfModule> BrowserMatch "Mozilla/2" nokeepalive BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0 BrowserMatch "RealPlayer 4\.0" force-response-1.0 BrowserMatch "Java/1\.0" force-response-1.0 BrowserMatch "JDK/1\.0" force-response-1.0 BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully BrowserMatch "MS FrontPage" redirect-carefully BrowserMatch "^WebDrive" redirect-carefully BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully BrowserMatch "^gnome-vfs/1.0" redirect-carefully BrowserMatch "^gvfs/1" redirect-carefully BrowserMatch "^XML Spy" redirect-carefully BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully BrowserMatch " Konqueror/4" redirect-carefully <IfModule mod_ssl.c> BrowserMatch "MSIE [2-6]" \ nokeepalive ssl-unclean-shutdown \ downgrade-1.0 force-response-1.0 # MSIE 7 and newer should be able to use keepalive BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown </IfModule> <Location /server-status> SetHandler server-status Require ip IP_VALUES_SUPPRESSED </Location> ExtendedStatus On <IfModule mod_proxy.c> # Show Proxy LoadBalancer status in mod_status ProxyStatus On </IfModule> <IfModule mod_fcgid.c> AddHandler fcgid-script fcg fcgi fpl FcgidIPCDir /run/mod_fcgid FcgidProcessTableFile /run/mod_fcgid/fcgid_shm </IfModule> 3) We could'nt tried to use a most rcent version of httpd, this version was imposed by the customer (owner of the drupal application). I stay available for any complementary informations. Thanks in advance.
Thanks a lot, the coredump will be vital to figure out what's happening. I am a bit ignorant about https://httpd.apache.org/docs/2.4/mod/mod_logio.html but I didn't find any trace of its configuration in your httpd config. Are you using it? Can you point me where?
We are doing our best in order to have a core dump. The logio module is load with log_config.load and logio.load Our log configuration is "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%D\"" but logio seems used only with %I %O %S %^FB. So I'm not sure using it. In fact this module was introduced in our configuration form puppet apache module. logio.load is include by default on Redhat and on Freebsd systems, and log_config is include for all systems, and seem usefull. For example it's the only one module default module loaded for suse' systems. After looking the detailled options of logio the FB option seems really useful, and be used in future on our infrastructure. We already log the time taken to serve the request, in microseconds (%D). I can't remove loading of this module without major changes in puppet apache module. According puppet module documentation, this module isn't configureable at this time. # These are modules required to run the default configuration. # They are not configurable at this time, so we just include # them to make sure it works. Is it a revelant test to remove loading this module in order to check if segfault already appear ?
The module register basic info about every request when it is loaded, and then it displays them when requested by CustomLog, so it makes sense that it could lead to a segfault even if not explicitly used with some directive. Let's see if we manage to get a coredump, I have some suspicions but I'd need to reproduce first. Your httpd version is very old (2.4.6 vs 2.4.26 now) so it might as well be that it is an old bug from the past (need to verify).
Have you noticed, by any chance, if the segfault is triggered by a particular set of requests? It might be easy to track using error/access logs comparing them with when the segfault happens.
Maybe I may add the FB option in order to use completely this module, this may solve this bug. What is your point of view about this ? About the core it will need somme time to reproduce the behaviour. In produciton context, this bug is reproduced only every 2 month without apache restart. Another team will build a test environnement in order to reproduce the behaviour as quickly as possible. About the behaviour, I haven't noticed a particular request making this segfault. It is reproduced on several differents website (3 drupal websites). We have a distributed infrastructure composed of 3 apache. When a segfault appear on a first apache, approximately 24h after, the two other apache have the same behaviour. When the segfault appear some static webpages are already served (GET /). But when the first segfault appear all child are exit on segfault signal.
(In reply to sgiraud from comment #7) > Maybe I may add the FB option in order to use completely this module, this > may solve this bug. What is your point of view about this ? I think that it will not solve the issue, mod_logio collects the data anyway, I think that you are hitting a corner case in which the module dereference a null pointer or something similar (but for the moment it is only a speculation). I checked https://github.com/apache/httpd/commits/trunk/modules/loggers/mod_logio.c and I don't see any recent fix for any segfault report, so it might still be an issue with 2.4.x recent versions. > > About the core it will need somme time to reproduce the behaviour. > In produciton context, this bug is reproduced only every 2 month without > apache restart. Another team will build a test environnement in order to > reproduce the behaviour as quickly as possible. > > About the behaviour, I haven't noticed a particular request making this > segfault. It is reproduced on several differents website (3 drupal > websites). > > We have a distributed infrastructure composed of 3 apache. When a segfault > appear on a first apache, approximately 24h after, the two other apache have > the same behaviour. When the segfault appear some static webpages are > already served (GET /). But when the first segfault appear all child are > exit on segfault signal. Thanks, let's wait the coredump then :)
Hi there, any chance that you got a core-dump?
Hi Luca, I let the apache coredump activated during a long time, but this bug don't seems appeared a new time. Have you heard another similar issue during this times ? Best regards,