Bug 65521

Summary: httpd high CPU usage after jboss-remoting client exits unexpectedly
Product: Apache httpd-2 Reporter: Marius Tantareanu <mariustant>
Component: mod_proxy_httpAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: normal    
Priority: P2    
Version: 2.4.48   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Attachments: httpd error log
httpd sample config file
httpd error log when client is running on Windows
Test EAR to be deployed in WildFly
Test EJB client
Handle POLLHUP when POLLOUT (only) is asked on shutdown

Description Marius Tantareanu 2021-08-26 19:25:17 UTC
Created attachment 37990 [details]
httpd error log

When httpd is used as a reverse proxy to WildFly, if a jboss-remoting Java client (EJB client) exits unexpectedly (without closing the connection to httpd / WildFly), httpd enters a state where it causes high CPU usage. E.g.

top - 19:01:21 up 2 days, 22:42,  3 users,  load average: 0.29, 0.08, 0.02
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
Cpu(s): 16.6%us, 27.6%sy,  0.0%ni, 52.3%id,  1.8%wa,  0.0%hi,  1.7%si,  0.0%st
Mem:  16333992k total, 15372600k used,   961392k free,   150668k buffers
Swap:  8241148k total,   483044k used,  7758104k free,  7163264k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2807 httpspro  20   0 82524 2984 1532 R 84.6  0.0   0:25.94 httpd

If trace8 log level is activated, the logs are filled with lines like the following:

[Thu Aug 26 15:37:02.854938 2021] [proxy:trace8] [pid 2085] proxy_util.c(4800): [client 127.0.0.1:57485] proxy: HTTP: polling (client=14, origin=0)
[Thu Aug 26 15:37:02.854941 2021] [proxy:trace8] [pid 2085] proxy_util.c(4823): [client 127.0.0.1:57485] AH10215: proxy: HTTP: woken up, 1 result(s)
[Thu Aug 26 15:37:02.854942 2021] [proxy:trace8] [pid 2085] proxy_util.c(4831): [client 127.0.0.1:57485] proxy: HTTP: #0: client: 30/14
[Thu Aug 26 15:37:02.854944 2021] [proxy:trace8] [pid 2085] proxy_util.c(4800): [client 127.0.0.1:57485] proxy: HTTP: polling (client=14, origin=0)
[Thu Aug 26 15:37:02.854946 2021] [proxy:trace8] [pid 2085] proxy_util.c(4823): [client 127.0.0.1:57485] AH10215: proxy: HTTP: woken up, 1 result(s)
[Thu Aug 26 15:37:02.854948 2021] [proxy:trace8] [pid 2085] proxy_util.c(4831): [client 127.0.0.1:57485] proxy: HTTP: #0: client: 30/14

The httpd logs (trace8 + dumpio) are attached. See error_log.

In order to resolve the issue, httpd needs to be restarted. The issue does not reproduce when the following configuration directive is used: ProxyWebsocketFallbackToProxyHttp Off

The issue was reproduced when both the Java client and httpd run on RHEL 6.7 (with WildFly 20). If the Java client runs on Windows the issue does not reproduce.
Comment 1 Marius Tantareanu 2021-08-26 19:27:46 UTC
Created attachment 37991 [details]
httpd sample config file
Comment 2 Marius Tantareanu 2021-08-26 19:28:44 UTC
Created attachment 37992 [details]
httpd error log when client is running on Windows
Comment 3 Marius Tantareanu 2021-08-26 19:30:45 UTC
Created attachment 37993 [details]
Test EAR to be deployed in WildFly
Comment 4 Marius Tantareanu 2021-08-26 19:36:14 UTC
Created attachment 37994 [details]
Test EJB client
Comment 5 Marius Tantareanu 2021-08-26 19:40:46 UTC
Steps to reproduce:

1. Deploy the attached EAR file (simple-ear.ear) in WildFly. The EAR contains a single EJB that will be accessed by the client.
2. Configure httpd as a reverse proxy to WildFly (httpd should handle HTTP protocol upgrade to jboss-remoting):

ProxyPass "/"  "ws://localhost:1026/" upgrade=jboss-remoting

A sample configuration file is attached (httpd.conf)

3. Run the attached standalone Java client that attempts to invoke an EJB in WildFly via httpd. Before running the client, you need to copy the WildFly client library (jboss-client.jar) to the lib folder (this is not included in the archive due to its size). 

cd echo-client-wf20
java -cp ./bin:./lib/jboss-client.jar:./lib/simple-ejbClient.jar com.microfocus.echoclient.EchoClient "remote+https" "localhost" "4433"

Port 4433 is SSL enabled. For clear text communication use protocol as "remote+http" and port as "80".

The client requires Java 11 or above (testing was done with Java 11).

After the client completes the EJB call, it simply exits without explicitly closing the connection to httpd / WildFly. In practice this can happen due to an unexpected crash but httpd should not get into this state due to a misbehaving client.
Comment 6 Yann Ylavic 2021-08-30 09:47:29 UTC
Created attachment 38002 [details]
Handle POLLHUP when POLLOUT (only) is asked on shutdown

Could you please try this patch?

The issue seems to be that the system is returning POLLHUP when the connection is half closed (FIN received from the client) while we are waiting for writability to send the final FIN, and this wasn't expected.
Comment 7 Yann Ylavic 2021-09-20 22:48:52 UTC
Fixed in 2.4.49 (r1892971).
Comment 8 Marius Tantareanu 2021-09-23 14:52:31 UTC
Will test the fix and get back with the results.
Comment 9 Marius Tantareanu 2021-09-24 06:25:07 UTC
I was not able to reproduce the issue with the 2.4.49 release. Thanks!