A possibly workaround is "ProxyWebsocketFallbackToProxyHttp off" (https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html#proxywebsocketfallbacktoproxyhttp).

But since 2.4.52 websocket tunneling uses the smallest of Timeout and ProxyTimeout, you may want to try 2.4.52 (and then add "Timeout 300" to your configuration as you already tried in 2.4.51).

> But since 2.4.52 websocket tunneling uses the smallest of Timeout and
> ProxyTimeout, you may want to try 2.4.52 (and then add "Timeout 300"
> to your configuration as you already tried in 2.4.51).

Wha? Isn't it wrong to make a completely undocumented and backward incompatible change like that in a patch release?

FWIW, I can confirm that adding "Timeout 300" does work around the problem:

  AH10212: proxy: websocket: tunnel running (timeout 300.000000)

But this still seems like a bug to me.

Or at least a poor design decision.

Why should ProxyTimeout's DEFAULT value (Timeout) ever override its CONFIGURED value (ProxyTimeout) ? I definitely don't get it.

I agree that the need to increase Timeout is not a good solution. Maybe we should change the logic to choose the maximum of both timeouts (front end socket, backend socket) instead of the minimum as backend timeouts can be configured more selectively (per worker if needed) as front end timeouts and typically the backend timeouts reflect the application requirements.

Something like the following patch:

Index: modules/proxy/mod_proxy_http.c
===================================================================
--- modules/proxy/mod_proxy_http.c	(revision 1898054)
+++ modules/proxy/mod_proxy_http.c	(working copy)
@@ -1485,10 +1485,10 @@
                 return HTTP_INTERNAL_SERVER_ERROR;
             }
 
-            /* Set timeout to the lowest configured for client or backend */
+            /* Set timeout to the highest configured for client or backend */
             apr_socket_timeout_get(backend->sock, &backend_timeout);
             apr_socket_timeout_get(ap_get_conn_socket(c), &client_timeout);
-            if (backend_timeout >= 0 && backend_timeout < client_timeout) {
+            if (backend_timeout >= 0 && backend_timeout > client_timeout) {
                 tunnel->timeout = backend_timeout;
             }
             else {
Index: modules/proxy/proxy_util.c
===================================================================
--- modules/proxy/proxy_util.c	(revision 1898054)
+++ modules/proxy/proxy_util.c	(working copy)
@@ -4718,10 +4718,10 @@
     tunnel->origin->pfd->desc.s = ap_get_conn_socket(c_o);
     tunnel->origin->pfd->client_data = tunnel->origin;
 
-    /* Defaults to the smallest timeout of both connections */
+    /* Defaults to the biggest timeout of both connections */
     apr_socket_timeout_get(tunnel->client->pfd->desc.s, &timeout);
     apr_socket_timeout_get(tunnel->origin->pfd->desc.s, &tunnel->timeout);
-    if (timeout >= 0 && (tunnel->timeout < 0 || tunnel->timeout > timeout)) {
+    if (timeout >= 0 && (tunnel->timeout < 0 || tunnel->timeout < timeout)) {
         tunnel->timeout = timeout;
     }

Another option instead of increasing the Timeout is

ProxyWebsocketFallbackToProxyHttp off

but in this case no timeout is applied to the websocket connection at all and the proxy waits until one side closes the connection.

(In reply to Ruediger Pluem from comment #5)
> Something like the following patch:

+1

(In reply to Archie Cobbs from comment #3)
> 
> Why should ProxyTimeout's DEFAULT value (Timeout) ever override its
> CONFIGURED value (ProxyTimeout) ? I definitely don't get it.

The rationale is/was that since Timeout is the one applied on the client connection side, and ProxyTimeout (or ProxyPass timeout= parameter) the one(s) on the backend side, we'd use the smallest when tunneling between the client and backend because if one side times out the tunnel should be terminated. But this was probably a poor choice because in most cases it actually obliges to change existing configuration to set both timeouts at the same/highest value. Poor choices happen..

As Rüdiger said, note that in 2.4.43 no timeout was enforced in mod_proxy for websocket tunneling (neither ProxyTimeout nor Timeout, so the timeout was actually infinite), your configuration worked regardless of ProxyTimeout probably thanks to the heartbeat. The above change was made precisely to honor a timeout while tunneling without an ad hoc mechanism.

Let's change this again for 2.4.53.

(In reply to Yann Ylavic from comment #7)
> (In reply to Ruediger Pluem from comment #5)
> > Something like the following patch:
> 
> +1

r1898127

(In reply to Ruediger Pluem from comment #9)
> (In reply to Yann Ylavic from comment #7)
> > (In reply to Ruediger Pluem from comment #5)
> > > Something like the following patch:
> > 
> > +1
> 
> r1898127

Thanks! Since the mod_proxy_http hunk of your patch in comment 5 is 2.4 only, how about we simply remove the tunnel->timeout setting there in 2.4 (on backport) given that it's already what ap_proxy_tunnel_create() called just above will do?

Thanks guys, appreciate the fast response.

I have two updates and a new question related to this.


The updates are:

1 - Adding "ProxyWebsocketFallbackToProxyHttp off" does in fact work
around the problem.

2 - Previously I said that setting "Timeout 300" also works around the
problem, but that is not entirely accurate. What that does is simply
change how long it takes for the particular problem I'm seeing to occur,
from 60 seconds to 5 minutes. This is as you would probably expect.


Here's my question:

> As Rüdiger said, note that in 2.4.43 no timeout was enforced in
> mod_proxy for websocket tunneling (neither ProxyTimeout nor Timeout,
> so the timeout was actually infinite), your configuration worked
> regardless of ProxyTimeout probably thanks to the heartbeat. The
> above change was made precisely to honor a timeout while tunneling
> without an ad hoc mechanism.

OK this makes sense. Basically my web application is using websockets
and sending a heartbeat every ~3 minutes. This websocket connection
was previously being proxied by mod_proxy to Tomcat, and mod_proxy
was not applying any particular timeout to that connection, so it
would stay up indefinitely.

But here's my question: isn't a websocket connection SUPPOSED to stay
up indefinitely?

Or at least, shouldn't any "timeout" value that is applied to that
connection be an IDLE timeout rather than an ABSOLUTE timeout (measured
from when the connection was started)?

In other words, it seems to me like whatever mod_proxy was doing is
more correct, at least in the scenario of a proxied websocket connection,
than what mod_wstunnel is doing.

Can you explain to me what I'm missing? Thanks.

(In reply to Archie Cobbs from comment #11)

> 
> Or at least, shouldn't any "timeout" value that is applied to that
> connection be an IDLE timeout rather than an ABSOLUTE timeout (measured
> from when the connection was started)?

It is an idle timeout. Connections with activity stay up.

> > Or at least, shouldn't any "timeout" value that is applied to that
> > connection be an IDLE timeout rather than an ABSOLUTE timeout (measured
> > from when the connection was started)?
> 
> It is an idle timeout. Connections with activity stay up.

Yes, you're right.

What Vaadin appears to be doing is creating more than one websocket connection,
and sending its heartbeat messages on only one of them.

Actually that does make sense - the purpose of the Vaadin heartbeat is to let the
server know that the client is still alive, not to juice the websocket connection(s).

Philosophically, timeouts are appropriate for a request/response world, but the
websockets world is a completely different beast. When doing websockets, it really
should be left up to the two endpoints to decide if/when the connection has "timed
out" and should be torn down; any intervening proxies should not have any timeout
(in general) because (in general) they don't have any special knowledge of what
the two endpoint applications are doing with their TCP connection.

So I'd say that there is another subtle problem here, which is that there really
should be two separate configuration variables for configuring (a) the proxy
timeout for regular connections and (b) the proxy timeout for websocket
connections.

> So I'd say that there is another subtle problem here, which is that there really
> should be two separate configuration variables for configuring (a) the proxy
> timeout for regular connections and (b) the proxy timeout for websocket
> connections.

Any thoughts on this question?

Right now the only way to have different timeout values for normal vs. websockets
is to say "ProxyWebsocketFallbackToProxyHttp off" which seems like the wrong tool.

(In reply to Archie Cobbs from comment #14)
> > So I'd say that there is another subtle problem here, which is that there really
> > should be two separate configuration variables for configuring (a) the proxy
> > timeout for regular connections and (b) the proxy timeout for websocket
> > connections.
> 
> Any thoughts on this question?
> 
> Right now the only way to have different timeout values for normal vs.
> websockets
> is to say "ProxyWebsocketFallbackToProxyHttp off" which seems like the wrong
> tool.


You can configure an individual timeout per wss:// backend worker. Setting it to an arbitrary high number should give you what you need.

Proposed for backport as https://github.com/apache/httpd/pull/295

> You can configure an individual timeout per wss:// backend worker.
> Setting it to an arbitrary high number should give you what you need.

Thanks - I didn't realize that.

(In reply to Ruediger Pluem from comment #16)
> Proposed for backport as https://github.com/apache/httpd/pull/295

Backported as r1898559. Will be part of the next release.