Bug 66019 - Files can randomly take 5 seconds to be served when KeepAlive is On
Summary: Files can randomly take 5 seconds to be served when KeepAlive is On
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.4.47
Hardware: Macintosh other
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: FixedInTrunk
Depends on:
Blocks:
 
Reported: 2022-04-19 21:12 UTC by tbaccelli
Modified: 2022-05-24 09:14 UTC (History)
0 users



Attachments
Random files taking 5 seconds to be served (220.79 KB, image/png)
2022-04-19 21:12 UTC, tbaccelli
Details
Result of tcpdump (645.87 KB, application/zip)
2022-04-20 14:06 UTC, tbaccelli
Details
Handle TCP_CORK vs TCP_NOPUSH peculiarities (2.04 KB, patch)
2022-04-20 17:01 UTC, Yann Ylavic
Details | Diff
Result of tcpdump after patch (643.76 KB, application/zip)
2022-04-20 18:10 UTC, tbaccelli
Details
Disable TCP_NOPUSH on OSX (2.69 KB, patch)
2022-04-20 20:02 UTC, Yann Ylavic
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description tbaccelli 2022-04-19 21:12:36 UTC
Created attachment 38253 [details]
Random files taking 5 seconds to be served

Since httpd 2.4.47 on my macbook pro running MacOS 12.3.1, files can take 5 seconds to be served as you can see on the screenshot. If I downgrade to 2.4.46, the problem disappears.

When searching on Google, I found this stackoverflow thread that can be helpful: https://stackoverflow.com/questions/70698918/apache-http-localhost-randomly-taking-5-seconds-on-macos-monterey-but-fast-on-ht

That's also in this thread that someone found that if you set KeepAlive to Off, the problem disappears too.
Comment 1 Ruediger Pluem 2022-04-20 06:21:01 UTC
Does the problem also appear with 2.4.53? Which MPM is used?
Comment 2 tbaccelli 2022-04-20 07:49:49 UTC
Yes, the problem also appear with 2.4.43, I'm using mpm_prefork
Comment 3 Ruediger Pluem 2022-04-20 11:51:27 UTC
2.4.43 or 2.4.53?
Comment 4 Ruediger Pluem 2022-04-20 11:54:15 UTC
What is your KeepaliveTimeout and your Timeout setting?
Comment 5 tbaccelli 2022-04-20 12:13:39 UTC
I meant 2.4.53, sorry 🤦‍♂️.

I use the default configuration for KeepaliveTimeout which seems to be 5 seconds. If I change it to 2 seconds for instance, I can see that it has an impact and files randomly take 2 seconds to be served.

Timeout is set to 60.
Comment 6 Yann Ylavic 2022-04-20 12:18:46 UTC
Please reproduce on 2.4.53 with "LogLevel trace8" and attach the error_log here.
Comment 7 Yann Ylavic 2022-04-20 12:21:28 UTC
Also with LogLevel trace8, loading mod_dumpio like the below:
  LoadModule dumpio_module modules/mod_dumpio.so
  DumpIOInput on
  DumpIOOutput on
would be useful.
Comment 8 tbaccelli 2022-04-20 12:39:05 UTC
The file is too big to be attached here, I uploaded on my dropbox: https://www.dropbox.com/s/xrt4l1vi6hyhizs/error_log?dl=0
Comment 9 tbaccelli 2022-04-20 12:51:05 UTC
To add a little more context, during the request logged, 3 files took 5 seconds to be served: jquery.min.js?v=5.1.3, bootstrap.bundle.min.js?v=5.1.3 and jquery-ui-timepicker-addon.js?v=5.1.3
Comment 10 Yann Ylavic 2022-04-20 13:36:46 UTC
On the httpd side it seems that those files are sent/flushed in a timely manner (within the same second), and then 5 seconds later the connection is closed because no new request arrives before the keepalive timeout.

Somehow the client/browser thinks the response is incomplete until the connection is closed, while it looks complete from the HTTP protocol point of vue (as shown in the traces).

Could you please capture a network trace on localhost (since both httpd and the client seem to be there) with something like (as root probably):
  # tcpdump -s0 -w /tmp/66019.cap tcp port 80
and attach "66019.cap" here?

(note that you can [g]zip the attachments before attaching them to reduce their size)
Comment 11 tbaccelli 2022-04-20 14:06:16 UTC
Created attachment 38255 [details]
Result of tcpdump

For this request, 3 files took 5 seconds to be served: theme.css?v=5.1.3&nocache=996186ltr&server=1, jquery-ui-timepicker-addon.js?v=5.1.3 and functions.js?v=5.1.3
Comment 12 Yann Ylavic 2022-04-20 17:01:34 UTC
Created attachment 38256 [details]
Handle TCP_CORK vs TCP_NOPUSH peculiarities

From the network traces, it seems that TCP packets are not PSH-ed appropriately on OSX, the last packet could be retained in the TCP stack until the connection is closed.

We may need to handle TCP_CORK on Linux and TCP_NOPUSH on BSDs/OSX differently in the core output filter, could you please try this patch on your system?
Comment 13 tbaccelli 2022-04-20 18:10:01 UTC
Created attachment 38257 [details]
Result of tcpdump after patch

Just tried with the patch, I still have the problem.

I attached the result of tcpdump with the patch, 3 files took 5 seconds: theme.css?v=5.1.3&nocache=1438139389ltr&server=1, bootstrap.bundle.min.js?v=5.1.3 and jquery-ui-timepicker-addon.js?v=5.1.3
Comment 14 Yann Ylavic 2022-04-20 20:02:35 UTC
Created attachment 38258 [details]
Disable TCP_NOPUSH on OSX

I read that OSX does not release corked data when unsetting TCP_NOPUSH (until the connection is shutdown?), which is not compatible with how httpd's core output filter works.
So this patches disables the TCP_NOPUSH optimization on OSX, would you please test it?
Comment 15 tbaccelli 2022-04-20 21:03:34 UTC
I confirm your latest patch works!
Comment 16 Yann Ylavic 2022-04-21 10:05:11 UTC
Thanks for testing and the great directions.

Checking in r1900100 (trunk).
Comment 17 tbaccelli 2022-04-21 10:18:18 UTC
Thanks for your time and reactivity!
Comment 18 Yann Ylavic 2022-05-24 09:14:00 UTC
Backported to 2.4 (r1901201), will be in the next release.