Summary: | httpd high CPU usage after jboss-remoting client exits unexpectedly | ||
---|---|---|---|
Product: | Apache httpd-2 | Reporter: | Marius Tantareanu <mariustant> |
Component: | mod_proxy_http | Assignee: | Apache HTTPD Bugs Mailing List <bugs> |
Status: | RESOLVED FIXED | ||
Severity: | normal | ||
Priority: | P2 | ||
Version: | 2.4.48 | ||
Target Milestone: | --- | ||
Hardware: | PC | ||
OS: | Linux | ||
Attachments: |
httpd error log
httpd sample config file httpd error log when client is running on Windows Test EAR to be deployed in WildFly Test EJB client Handle POLLHUP when POLLOUT (only) is asked on shutdown |
Created attachment 37991 [details]
httpd sample config file
Created attachment 37992 [details]
httpd error log when client is running on Windows
Created attachment 37993 [details]
Test EAR to be deployed in WildFly
Created attachment 37994 [details]
Test EJB client
Steps to reproduce: 1. Deploy the attached EAR file (simple-ear.ear) in WildFly. The EAR contains a single EJB that will be accessed by the client. 2. Configure httpd as a reverse proxy to WildFly (httpd should handle HTTP protocol upgrade to jboss-remoting): ProxyPass "/" "ws://localhost:1026/" upgrade=jboss-remoting A sample configuration file is attached (httpd.conf) 3. Run the attached standalone Java client that attempts to invoke an EJB in WildFly via httpd. Before running the client, you need to copy the WildFly client library (jboss-client.jar) to the lib folder (this is not included in the archive due to its size). cd echo-client-wf20 java -cp ./bin:./lib/jboss-client.jar:./lib/simple-ejbClient.jar com.microfocus.echoclient.EchoClient "remote+https" "localhost" "4433" Port 4433 is SSL enabled. For clear text communication use protocol as "remote+http" and port as "80". The client requires Java 11 or above (testing was done with Java 11). After the client completes the EJB call, it simply exits without explicitly closing the connection to httpd / WildFly. In practice this can happen due to an unexpected crash but httpd should not get into this state due to a misbehaving client. Created attachment 38002 [details]
Handle POLLHUP when POLLOUT (only) is asked on shutdown
Could you please try this patch?
The issue seems to be that the system is returning POLLHUP when the connection is half closed (FIN received from the client) while we are waiting for writability to send the final FIN, and this wasn't expected.
Will test the fix and get back with the results. I was not able to reproduce the issue with the 2.4.49 release. Thanks! |
Created attachment 37990 [details] httpd error log When httpd is used as a reverse proxy to WildFly, if a jboss-remoting Java client (EJB client) exits unexpectedly (without closing the connection to httpd / WildFly), httpd enters a state where it causes high CPU usage. E.g. top - 19:01:21 up 2 days, 22:42, 3 users, load average: 0.29, 0.08, 0.02 Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie Cpu(s): 16.6%us, 27.6%sy, 0.0%ni, 52.3%id, 1.8%wa, 0.0%hi, 1.7%si, 0.0%st Mem: 16333992k total, 15372600k used, 961392k free, 150668k buffers Swap: 8241148k total, 483044k used, 7758104k free, 7163264k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2807 httpspro 20 0 82524 2984 1532 R 84.6 0.0 0:25.94 httpd If trace8 log level is activated, the logs are filled with lines like the following: [Thu Aug 26 15:37:02.854938 2021] [proxy:trace8] [pid 2085] proxy_util.c(4800): [client 127.0.0.1:57485] proxy: HTTP: polling (client=14, origin=0) [Thu Aug 26 15:37:02.854941 2021] [proxy:trace8] [pid 2085] proxy_util.c(4823): [client 127.0.0.1:57485] AH10215: proxy: HTTP: woken up, 1 result(s) [Thu Aug 26 15:37:02.854942 2021] [proxy:trace8] [pid 2085] proxy_util.c(4831): [client 127.0.0.1:57485] proxy: HTTP: #0: client: 30/14 [Thu Aug 26 15:37:02.854944 2021] [proxy:trace8] [pid 2085] proxy_util.c(4800): [client 127.0.0.1:57485] proxy: HTTP: polling (client=14, origin=0) [Thu Aug 26 15:37:02.854946 2021] [proxy:trace8] [pid 2085] proxy_util.c(4823): [client 127.0.0.1:57485] AH10215: proxy: HTTP: woken up, 1 result(s) [Thu Aug 26 15:37:02.854948 2021] [proxy:trace8] [pid 2085] proxy_util.c(4831): [client 127.0.0.1:57485] proxy: HTTP: #0: client: 30/14 The httpd logs (trace8 + dumpio) are attached. See error_log. In order to resolve the issue, httpd needs to be restarted. The issue does not reproduce when the following configuration directive is used: ProxyWebsocketFallbackToProxyHttp Off The issue was reproduced when both the Java client and httpd run on RHEL 6.7 (with WildFly 20). If the Java client runs on Windows the issue does not reproduce.