Bug 59333 - Version 2.4.20 server-status does not show page being retrieved
Summary: Version 2.4.20 server-status does not show page being retrieved
Status: RESOLVED FIXED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_status (show other bugs)
Version: 2.4.20
Hardware: PC Linux
: P2 minor (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: FixedInTrunk
Depends on:
Blocks:
 
Reported: 2016-04-15 17:19 UTC by Robert Dinse
Modified: 2016-08-29 07:54 UTC (History)
0 users



Attachments
Server Status (111.71 KB, text/html)
2016-04-15 17:19 UTC, Robert Dinse
Details
Clear old/unkown worker score entry at incoming request (630 bytes, patch)
2016-04-24 18:01 UTC, Yann Ylavic
Details | Diff
Save/restore last request info in/from conn_rec (2.96 KB, patch)
2016-04-25 21:37 UTC, Yann Ylavic
Details | Diff
Save/restore last request info in/from conn_rec (2.4.x) (6.27 KB, patch)
2016-04-26 17:53 UTC, Yann Ylavic
Details | Diff
Alternative to 33803/33808. (1.12 KB, patch)
2016-05-08 15:57 UTC, Rainer Jung
Details | Diff
Save/restore last request info in/from conn_rec (trunk) (3.39 KB, patch)
2016-05-08 20:04 UTC, Yann Ylavic
Details | Diff
Fix scoreboard for 2.4.20 (13.72 KB, patch)
2016-05-10 21:05 UTC, Yann Ylavic
Details | Diff
Fix scoreboard for 2.4.20 + avoid blanks (16.04 KB, patch)
2016-05-10 21:10 UTC, Yann Ylavic
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Dinse 2016-04-15 17:19:34 UTC
Created attachment 33769 [details]
Server Status

This was broken between version 2.4.18 and 2.4.20, version 2.4.20 wasn't an option in the version pull-down menu.  The server-status no longer shows the URL or domain name of the page being retrieved.
Comment 1 Eric Covener 2016-04-15 17:41:21 UTC
Was reported earlier and is proposed for backport

http://home.apache.org/~ylavic/patches/httpd-2.4.x-scoreboard_preserve-v2.patch
Comment 2 Nick Warne 2016-04-23 16:46:10 UTC
I don't think this is fixed - not properly at least.

I have a few 'hidden' pages that I use to monitor a few of my server scripts, and after applying this patch to get server-status back, I see wrong IP's accessing these pages when it is impossible to do so.
Comment 3 Yann Ylavic 2016-04-23 17:26:13 UTC
(In reply to Nick Warne from comment #2)
> I have a few 'hidden' pages that I use to monitor a few of my server
> scripts, and after applying this patch to get server-status back, I see
> wrong IP's accessing these pages when it is impossible to do so.

Another change from 2.4.20 is that the client (IP) reported by mod_status may now be the one of the user-agent (origin) as determined by mod_remoteip configuration.

Do you have mod_remoteip configured and the "hidden" pages accessed via a proxy?
Comment 4 Nick Warne 2016-04-24 08:16:12 UTC
Basically no and no.

The 'hidden' pages are only known by me on my local network, so it is impossible for anybody to know they exist, let alone guess what the names are - yet server-status shows remote IP's GETting them (which is why I think it is broken ).  Somehow the IP list is getting confused.

Also I just noticed it also reports my local IP address GETting my robots.txt file this morning - which I haven't.

Nick
Comment 5 Nick Warne 2016-04-24 09:42:55 UTC
OK, I doubted my ability to patch properly - so I started again and re-patched a virgin scoreboard.c - all clean.

Now, I just ran ssl labs test - and low and behold server-status shows one of the remote IP's accessing my server-status page (which isn't even called that) and also is tied down to only allow access from my local 192.168.x.x address.

Definitely something wonky going on.
Comment 6 Yann Ylavic 2016-04-24 10:16:33 UTC
OK, makes sense now, what you see is not remote IPs accessing *your* /status page (whatever its name/path is), but some bots (scanner or crawler) trying to access the default /server-status or /robots.txt.
Even if these requests are finally refused by your configuration, they still appear in the scoreboard.
You probably should find them in your access log too (as 403 or 404 depending on your configuration).
Comment 7 Nick Warne 2016-04-24 10:28:52 UTC
No you miss what I said - my server-status page is called something totally different - nobody knows except me so how a remote IP can know it is impossible!

Anyway, I done a test.  I created a txt file:  TesT.txt just now and visited it.  Now a few hours later, I checked server status and this appears:

64.41.200.103 http/1.1www.linicks.net:443 GET /TesT.txt HTTP/1.1

So you see the list gets mangled on what IP GET's what page.
Comment 8 Eric Covener 2016-04-24 12:57:59 UTC
(In reply to Nick Warne from comment #7)
> No you miss what I said - my server-status page is called something totally
> different - nobody knows except me so how a remote IP can know it is
> impossible!
> 
> Anyway, I done a test.  I created a txt file:  TesT.txt just now and visited
> it.  Now a few hours later, I checked server status and this appears:
> 
> 64.41.200.103 http/1.1www.linicks.net:443 GET /TesT.txt HTTP/1.1
> 
> So you see the list gets mangled on what IP GET's what page.


What did it say immediately after the access?
Do you have mod_http2 loaded?  
What are the "M" and "SS" fields immediately after and a few hours later?
Comment 9 Nick Warne 2016-04-24 13:09:40 UTC
Well, I can't really answer that sensibly, as each time I look at server-status the IP's seem to jump around to pages they never visited.

But here is one that GET's my renamed server-status page that only I know:

2-0 1218 0/1/1_ 1.9830340.00.010.01 80.12.36.250 http/1.1 www.linicks.net:443 GET (my status page)

I think (NEVER think!) the page requests are right - just the IP list gets mangled on who got what.
Comment 10 Nick Warne 2016-04-24 13:17:16 UTC
Sorry, forgot - no HTTP2 loaded.
Comment 11 Eric Covener 2016-04-24 13:24:06 UTC
(In reply to Nick Warne from comment #9)
> Well, I can't really answer that sensibly, as each time I look at
> server-status the IP's seem to jump around to pages they never visited.
> 
> But here is one that GET's my renamed server-status page that only I know:
> 
> 2-0 1218 0/1/1_ 1.9830340.00.010.01 80.12.36.250 http/1.1
> www.linicks.net:443 GET (my status page)
> 
> I think (NEVER think!) the page requests are right - just the IP list gets
> mangled on who got what.

You should see over time:

 - an access to your status page with a low value for SS (seconds-since). It'd be interesting to confirm if that the IP is ever correct (initially)

 - Over time, the same "slot" (n-th entry for a given PID) will continue to show your status page URL, with an increasing SS, until it is reused by a new request.

 - If it's reused, it be interesting to see what access_log activity from that IP address actually occurs and see if there's anything unique about it

For example, knowing whether the IP gets corrupted without SS being reset might be a clue to someone. Or if the URL itself jumped around.
Comment 12 Nick Warne 2016-04-24 14:10:15 UTC
I don't know how I can keep an eye on doing that - it changes all the time.

Anyway, another example - this time using one of my 'hidden' pages (I can easily change it, doesn't do a lot):

server-status reports:

3-0	1301	0/1/1	_ 	2.63	957	0	0.0	0.00	0.00 	164.132.161.67	http/1.1	www.linicks.net:443	GET /pte.txt HTTP/1.1

~# grep pte.txt access_log

192.168.1.1 - - [24/Apr/2016:08:55:18 +0100] "GET /pte.txt HTTP/1.1" 200 155 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:10:14:01 +0100] "GET /pte.txt HTTP/1.1" 200 155 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:10:14:03 +0100] "GET /pte.txt HTTP/1.1" 200 155 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:14:21:50 +0100] "GET /pte.txt HTTP/1.1" 200 155 "https://linicks.net/" "Dillo/3.1-dev"

As you see, 'tis only my local network IP that accessed that page - and server has been restarted several times today, so no way has 164.132.161.67 ever been there - nobody but me knows that page exists!
Comment 13 Nick Warne 2016-04-24 14:22:16 UTC
And another:

3-0	1301	0/3/3	_ 	3.25	796	0	0.0	0.02	0.02 	210.163.39.78	http/1.1	www.linicks.net:443	GET /TesT.txt HTTP/1.1

192.168.1.1 - - [24/Apr/2016:12:19:51 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:12:19:53 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:12:31:08 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:13:06:03 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:14:46:25 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:14:46:27 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:14:46:28 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"
192.168.1.1 - - [24/Apr/2016:14:46:28 +0100] "GET /TesT.txt HTTP/1.1" 200 5 "https://linicks.net/" "Dillo/3.1-dev"

Remember, I only created that file today as a test.
Comment 14 Yann Ylavic 2016-04-24 18:01:41 UTC
Created attachment 33801 [details]
Clear old/unkown worker score entry at incoming request

There is a window (more or less wide depending on the MPM) where the connection is accepted but the request not read/available yet, such that the connection informations are known (like remote IP) but not the request data (request line).
During that time the scoreboard will show the IP of the new client but still the Protocol/Vhost/Request of the previous request associated with the entry.

The attached patch resets the entry whenever a new request is being handled, which should show empty values for informations which are not yet known, instead of mixed requests data.

Could you please give it a try?
Comment 15 Nick Warne 2016-04-24 19:09:40 UTC
Tried it.

Now I get some full reports on what GET's what page, but some are also blank from vserver onwards as per the original report.

Obviously now, I don't know/haven't a clue how to debug.
Comment 16 Yann Ylavic 2016-04-24 20:06:41 UTC
As I said, the blank ones (for Protocol/Vhost/Request) are for new connections which obviously need a slot, the client IP is available but not the request yet to fill in the corresponding values.

It's either mixed values or blank ones, in both case this is a transient state anyway.
I don't know what's the better (I prefer blank over misleading values), let's see what others think... 

This is not the same as the original issue though (2.4.20 without httpd-2.4.x-scoreboard_preserve-v2.patch), previously the values remain blank possibly much longer (all the time waiting for a new connection), including the client IP.
With attachment 33801 [details], the client IP is always up to date and the new request data show up as soon as available.

PS: taking the slot only when the request is available is not an option, scoreboard slots are not just meant for mod_status.
Comment 17 Nick Warne 2016-04-25 15:29:00 UTC
(In reply to Yann Ylavic from comment #16)
> As I said, the blank ones (for Protocol/Vhost/Request) are for new
> connections which obviously need a slot, the client IP is available but not
> the request yet to fill in the corresponding values.
> 
> It's either mixed values or blank ones, in both case this is a transient
> state anyway.
> I don't know what's the better (I prefer blank over misleading values),
> let's see what others think... 

OK, I see where we are now - thanks for the hard work.

Yes, blanks is the way to go - at least that stops to bogus reporting which is what confused me.

I am setting this as 'resolved'.

Many thanks for your help Yann.
Comment 18 Yann Ylavic 2016-04-25 21:37:46 UTC
Created attachment 33803 [details]
Save/restore last request info in/from conn_rec

Maybe we can have the best of the two worlds finally.

This patch saves the informations (namely Request and VHost colomns) of the last finishing request into the connection structure, and restores them on the worker slot handling the new request (or more importantly the end of connection which gave blank values previously).

This requires some more bytes in conn_rec, but avoids mixing and blanks completely.

Nick, would you test it in your environment? Thanks!
Comment 19 Yann Ylavic 2016-04-25 21:38:46 UTC
Possibly the new patch would work better...
Comment 20 Nick Warne 2016-04-26 15:34:01 UTC
(In reply to Yann Ylavic from comment #19)
> Possibly the new patch would work better...

Stupid question - do I apply this over the other patch, or on the virgin scoreboard.c?

Thanks,

Nick
Comment 21 Nick Warne 2016-04-26 16:35:23 UTC
Ummm.

in include/httpd.h, I haven't got this so don't know where to patch:

 /**
  * @brief Structure to store things which are per connection
  */
@@ -1217,6 +1220,10 @@ struct conn_rec {
 
     /** The minimum level of filter type to allow setaside buckets */
     int async_filter;
+
+    /** Preserve scoreboard's worker last request infos */
+    char sb_lastreq[AP_SB_REQ_SIZE];
+    char sb_lastvhost[AP_SB_VHOST_SIZE];
 };

?

Nick
Comment 22 Rainer Jung 2016-04-26 17:36:23 UTC
(In reply to Nick Warne from comment #21)

Insert the lines which are indicated by the plus sign but without the plus sign itself at the plave below indicated with "INSERT HERE!".

1086 /**
1087  * @brief Structure to store things which are per connection
1088  */
1089 struct conn_rec {
1090     /** Pool associated with this connection */
1091     apr_pool_t *pool;
...
INSERT HERE!
1183
1184     /** The "real" master connection. NULL if I am the master. */
1185     conn_rec *master;
1186 };
1187

Thanks,

Rainer
Comment 23 Yann Ylavic 2016-04-26 17:53:39 UTC
Created attachment 33808 [details]
Save/restore last request info in/from conn_rec (2.4.x)

Sorry for not having specified how to apply the patch, moreover it applies cleanly to trunk only.

This new patch is against clean 2.4.x (2.4.20), and includes both httpd-2.4.x-scoreboard_preserve-v2.patch and attachment 33803 [details] (no need to apply any patch before).

Thanks for testing.
Comment 24 Nick Warne 2016-04-26 17:55:58 UTC
(In reply to Rainer Jung from comment #22)
> (In reply to Nick Warne from comment #21)
> 
> Insert the lines which are indicated by the plus sign but without the plus
> sign itself at the plave below indicated with "INSERT HERE!".
> 
> 1086 /**
> 1087  * @brief Structure to store things which are per connection
> 1088  */
> 1089 struct conn_rec {
> 1090     /** Pool associated with this connection */
> 1091     apr_pool_t *pool;
> ...
> INSERT HERE!
> 1183
> 1184     /** The "real" master connection. NULL if I am the master. */
> 1185     conn_rec *master;
> 1186 };
> 1187
> 
> Thanks,
> 
> Rainer

Sorry, I don't have that section that in include/httpd.h that starts with:

/**
* @brief Structure to store things which are per connection
*/
struct conn_rec {
/** Pool associated with this connection */
apr_pool_t *pool;

I am using latest stable 2.4.20 source - was this added in the dev trunk?  Also I looked for a link where to get the dev trunk, but can't find it?

Nick
Comment 25 Nick Warne 2016-04-26 18:59:58 UTC
(In reply to Yann Ylavic from comment #23)
> Created attachment 33808 [details]
> Save/restore last request info in/from conn_rec (2.4.x)
> 
> Sorry for not having specified how to apply the patch, moreover it applies
> cleanly to trunk only.
> 
> This new patch is against clean 2.4.x (2.4.20), and includes both
> httpd-2.4.x-scoreboard_preserve-v2.patch and attachment 33803 [details] (no
> need to apply any patch before).
> 
> Thanks for testing.

OK, built fine... but I now see the same results whereby the slots of IP show the wrong GET's.

I use https://www.ssllabs.com/ssltest/ as a base to monitor GET's on the fly, and the server-status page soon gets confused - i.e. it shows ssl labs IP accessing my server-status page.

Nick
Comment 26 Mike Rumph 2016-04-26 19:07:20 UTC
Hello Nick,

We check out the dev trunk directly from SVN.

See the following link:

- http://httpd.apache.org/dev/devnotes.html
Comment 27 Nick Warne 2016-04-26 20:34:05 UTC
(In reply to Mike Rumph from comment #26)
> Hello Nick,
> 
> We check out the dev trunk directly from SVN.
> 
> See the following link:
> 
> - http://httpd.apache.org/dev/devnotes.html

Thanks Mike, got the latest dev trunk.

OK, server-status still reports wrong GETs - I preferred the 'blanks'.

I can now patch on svn version :)

As an aside, seeing as a few devs are reading here (I know it's TABOO), but can somebody look at bug https://bz.apache.org/bugzilla/show_bug.cgi?id=57771 and fix it up - I have to apply this manually each new source code build - and it is a wrong assignment in code, not a bug.

Thanks,

Nick
Comment 28 William A. Rowe Jr. 2016-04-27 18:59:48 UTC
In addition to the patch mentioned in Comment #1, please also apply the following

https://raw.githubusercontent.com/wrowe/patches/master/backport-httpd-2.4-r1741310.patch

and report back if this entirely resolves the broken scoreboard behavior.

If it does not, please identify the MPM in use and whether you are trying
to find lingering http or http2 request workers.
Comment 29 William A. Rowe Jr. 2016-04-28 15:44:54 UTC
I've tweaked the solution in #28 to also empty the stale value of vhost or request on subsequent connections, which was broken in 2.4.20 and perhaps earlier.

See https://raw.githubusercontent.com/wrowe/patches/master/backport-httpd-2.4-r1741310-1741461.patch and remember to also apply the http://home.apache.org/~ylavic/patches/httpd-2.4.x-scoreboard_preserve-v2.patch first.
Comment 30 Nick Warne 2016-04-28 18:53:52 UTC
OK, tried this.

At least now no bogus reports, but I get a lot more 'blanks'.

The bogus reports in the original patch was the issue (i.e. false info is no info), so this OK to me as is.

As a comment, not a moan - why did this issue manifest on a stable release anyway?

Nick
Comment 31 Eric Covener 2016-04-28 19:02:38 UTC
> As a comment, not a moan - why did this issue manifest on a stable release
> anyway?

Nobody noticed the bug in the relatively short window the release was tested.
Comment 32 Rainer Jung 2016-05-08 10:38:56 UTC
I found another case where the request info gets clobbered: after a read timeout for the request line, e.g. when the client does not send another request before the connection times out during http keep-alive. After that the request field show "NULL".

An appropriate one-liner patch seems to be

http://svn.apache.org/r1742792

applied to trunk and proposed for backport.
Comment 33 Rainer Jung 2016-05-08 15:56:05 UTC
There's another case when the timeout replaces the request line with "NULL", noticable e.g. non-async MPMs. That should be fixed by:

http://svn.apache.org/r1742822

Applied to trunk and proposed for backport.

Then there are situations when updating the connection info also clobbers the request data and which should be fixed using Yann't attached patch. I'l attach another one that uses a slightly different approach. We'll see which one makes it. Feedback from people testing them is very welcome.
Comment 34 Rainer Jung 2016-05-08 15:57:25 UTC
Created attachment 33829 [details]
Alternative to 33803/33808.
Comment 35 Yann Ylavic 2016-05-08 20:04:05 UTC
Created attachment 33830 [details]
Save/restore last request info in/from conn_rec (trunk)

Possibly this new patch (on top of current trunk) could avoid both blanks (for kept alive or closing connections) and inconsistent (mixed) connection/request data.

Any chance you could try it with your test case Nick?
Thanks in advance.
Comment 36 Nick Warne 2016-05-10 17:47:13 UTC
(In reply to Yann Ylavic from comment #35)
> Created attachment 33830 [details]
> Save/restore last request info in/from conn_rec (trunk)
> 
> Possibly this new patch (on top of current trunk) could avoid both blanks
> (for kept alive or closing connections) and inconsistent (mixed)
> connection/request data.
> 
> Any chance you could try it with your test case Nick?
> Thanks in advance.

OK, I am confused now on what patch(es) should be applied in what order and what not.

Is there a backport svn pull on 2.4.20(xx) that I can try, as the svn -dev fubars date on server-status and log file times (my filed bug 59395) so I don't want to use that.

Maybe list each patch to apply in order so I get there - will be happy to test.

Nick
Comment 37 Yann Ylavic 2016-05-10 21:05:18 UTC
Created attachment 33835 [details]
Fix scoreboard for 2.4.20

This patch applying to vanilla 2.4.20 includes all the fixes accepted for next 2.4.21 (currently) regarding the scoreboard.

The scoreboard may contain blanks values (more or less time depending on the MPM used), but no inconsistent (mixed) connection/request data.
Comment 38 Yann Ylavic 2016-05-10 21:10:23 UTC
Created attachment 33836 [details]
Fix scoreboard for 2.4.20 + avoid blanks

This one (still applying to vanilla 2.4.20) includes attachment 33835 [details] plus an attempt to preserve request info accross states/workers, hopefully avoiding blank values.
Comment 39 Nick Warne 2016-05-11 16:50:37 UTC
Thank you Yann for all your hard work.

Patched cleanly, and built with no errors on virgin 2.4.20 source.

I am running now and report back on any issues I see.

Nick
Comment 40 Nick Warne 2016-05-11 17:11:45 UTC
One thing I just noticed - I don't know if it to do with this bug or what:

If I stop/start the server, and visit my server-status page immediately, then the:

Req Milliseconds required to process most recent request

shows:

15977568470

All subsequent queries show 2, 3, 4, 2 etc (i.e. expected ms response).

Nick
Comment 41 Nick Warne 2016-05-18 18:25:51 UTC
OK, been running this now for over 6 days, and apart from a few (minimal) blanks, all works as previous behaviour.

Thanks,

Nick
Comment 42 Stefan Eissing 2016-08-29 07:54:50 UTC
Old behaviour reported as restored in last release.