Bug 59007 - mod_proxy disables balancer worker after a timeout from ajp even if the failontimeout set to off
Summary: mod_proxy disables balancer worker after a timeout from ajp even if the failo...
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.4.18
Hardware: Sun Solaris
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
Keywords: FixedInTrunk
Depends on:
Reported: 2016-02-15 10:14 UTC by Konstantin J. Chernov
Modified: 2017-09-09 12:30 UTC (History)
0 users

mod_proxy and mod_proxy_ajp patch (1.73 KB, patch)
2016-02-15 10:14 UTC, Konstantin J. Chernov
Details | Diff
patch that introduces new option ProxyAllowTimeouts (4.61 KB, patch)
2016-02-15 14:08 UTC, Konstantin J. Chernov
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Konstantin J. Chernov 2016-02-15 10:14:48 UTC
Created attachment 33556 [details]
mod_proxy and mod_proxy_ajp patch

I've encountered a very annoying bug in mod_proxy that causes it to disable balancer worker after a timeout from backend even if the failontimeout set to off.

Following messages in the error log appears:
[2016-02-15 13:00:08.042994] [proxy_ajp:error] [pid 11478:tid 25] [VsGhbMCoW2YAACzWK-0AAABW] AH00992: ajp_read_header: ajp_ilink_receive failed
[2016-02-15 13:00:08.043068] [proxy_ajp:error] [pid 11478:tid 25] [VsGhbMCoW2YAACzWK-0AAABW] (70007)The timeout specified has expired: AH00893: dialog to x.x.x.x:port (x.x.x.x) failed

I'm using sticky sessions, and after each such timeout (which is normal for some applications that does not handle too long queries well) half of the users are being disconnected and forced to re-login into the application.

I've made a small patch to make mod_proxy_ajp+mod_proxy respect failontimeout property. It is included.
Comment 1 Konstantin J. Chernov 2016-02-15 14:08:21 UTC
Created attachment 33557 [details]
patch that introduces new option ProxyAllowTimeouts

I did a quick research and it looks like that mod_proxy_http suffers from the same problem -- I got another timeout error that disabled my worker even though failontimeout set to off.

[Mon Feb 15 19:56:14.191054 2016] [proxy_http:error] [pid 14722:tid 22] (70007)The timeout specified has expired: [client x.x.x.x:port] AH01102: error reading stat
us line from remote server x.x.x.x:port, referer: http://x.x.x.x:port/url
[Mon Feb 15 19:56:14.191547 2016] [proxy:error] [pid 14722:tid 22] [client x.x.x.x:port] AH00898: Timeout on 100-Continue returned by /url/here, referer: http://x.x.x.x:port/referer/here

I attached another patch that adds new option ProxyAllowTimeouts that disables this annoying behaviour of the mod_proxy.
Comment 2 jfclere 2017-09-05 07:25:12 UTC
I have retry on trunk it is that the failontimeout=On is ignored with ajp and works with http

LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
<Proxy balancer://mycluster forcerecovery=Off failontimeout=On>
    #BalancerMember http://localhost:8080 ping=1 timeout=5
    BalancerMember ajp://localhost:8009 ping=1 timeout=5

I have fixed a related issue JBCS-390 in trunk
Comment 3 Konstantin J. Chernov 2017-09-05 08:40:45 UTC
Looks like it must fix an issue with mod_proxy_http.

I also found that the problem with mod_proxy_ajp was fixed a very long time ago in trunk: https://github.com/apache/httpd/commit/89b7aba0f33d296b52b58d73f2805352a3f83db9

However that change didn't make it into 2.4.x tree, so the problem is still there :(
Comment 4 jfclere 2017-09-06 15:58:14 UTC
According to my test in trunk it is fixed now,
Comment 5 Luca Toscano 2017-09-09 09:02:01 UTC
Should we keep this open until the fix lands to 2.4.x ?
Comment 6 Yann Ylavic 2017-09-09 12:30:12 UTC
This is how we usually do, it helps keeping track of fixes still to be backported (here r1807318 and r1807338).

Adding the keyword "FixedInTrunk", that's where we are now.