Bug 68561 - IDLE processes are not killed with mpm_event
Summary: IDLE processes are not killed with mpm_event
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mpm_event (show other bugs)
Version: 2.4.52
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-01-30 11:29 UTC by mskogorev
Modified: 2024-02-29 22:14 UTC (History)
0 users


Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description mskogorev 2024-01-30 11:29:11 UTC 
After a few reloads, IDLE processes stop being killed.

I am using Apache2=2.4.52 with patch 1901199 applied (from https://bz.apache.org/bugzilla/show_bug.cgi?id=66004), but it is still reproducible for me.

I am using Debian GNU/Linux 10 (buster)
mpm_event configuration:

<IfModule mpm_event_module>
        StartServers 50
        MinSpareThreads 350
        MaxSpareThreads 400
        ThreadLimit 7
        ThreadsPerChild 7
        MaxRequestWorkers 924
        MaxConnectionsPerChild 0
        ServerLimit 132
</IfModule>

I get these messages in the logs:

[Mon Jan 29 07:53:02.223009 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 824, max workers 924
[Mon Jan 29 07:53:03.223613 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 826, max workers 924
[Mon Jan 29 07:53:04.224772 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 823, max workers 924
[Mon Jan 29 07:53:05.225888 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 823, max workers 924
[Mon Jan 29 07:53:06.227033 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 823, max workers 924
[Mon Jan 29 07:53:07.228175 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 822, max workers 924
[Mon Jan 29 07:53:08.229332 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 823, max workers 924
[Mon Jan 29 07:53:09.230462 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 821, max workers 924
[Mon Jan 29 07:53:10.231594 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 824, max workers 924
[Mon Jan 29 07:53:11.234644 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 825, max workers 924
[Mon Jan 29 07:53:12.235627 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 825, max workers 924
[Mon Jan 29 07:53:13.236948 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 826, max workers 924
[Mon Jan 29 07:53:14.238283 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 826, max workers 924
[Mon Jan 29 07:53:15.239535 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 823, max workers 924
[Mon Jan 29 07:53:16.240305 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 825, max workers 924
[Mon Jan 29 07:53:17.241504 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 825, max workers 924
[Mon Jan 29 07:53:18.242678 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 826, max workers 924
[Mon Jan 29 07:53:19.243689 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 826, max workers 924
[Mon Jan 29 07:53:20.244875 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 130/132, total 131/132/132, idle threads 826, max workers 924

This problem does not reproduce immediately, but after some time.
Stopping/starting apache2 recovers from this situation. This is only reproduced in a production environment where there is a lot of traffic.

When I do a reboot, all child processes are terminated and re-created:

[Mon Jan 29 08:03:03.902993 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 1 started: pid 24591, gen 30, active 15/132, total 63/132/132
[Mon Jan 29 08:03:03.903172 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 4 stopped: pid 20114, gen 29, active 15/132, total 62/132/132, quiescing 2
[Mon Jan 29 08:03:03.903282 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 40 stopped: pid 20488, gen 29, active 15/132, total 61/132/132, quiescing 2
[Mon Jan 29 08:03:03.903369 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 44 stopped: pid 20491, gen 29, active 15/132, total 60/132/132, quiescing 2
[Mon Jan 29 08:03:03.903449 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 49 stopped: pid 20495, gen 29, active 15/132, total 59/132/132, quiescing 2
[Mon Jan 29 08:03:03.903524 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 51 stopped: pid 20497, gen 29, active 15/132, total 58/132/132, quiescing 2
[Mon Jan 29 08:03:03.903619 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 53 stopped: pid 20499, gen 29, active 15/132, total 57/132/132, quiescing 2
[Mon Jan 29 08:03:03.903696 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 55 stopped: pid 20501, gen 29, active 15/132, total 56/132/132, quiescing 2
[Mon Jan 29 08:03:03.903778 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 56 stopped: pid 20502, gen 29, active 15/132, total 55/132/132, quiescing 2
[Mon Jan 29 08:03:03.903863 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 59 stopped: pid 20505, gen 29, active 15/132, total 54/132/132, quiescing 2
[Mon Jan 29 08:03:03.903940 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 60 stopped: pid 20506, gen 29, active 15/132, total 53/132/132, quiescing 2
........
........
........
[Mon Jan 29 08:03:04.059700 2024] [mpm_event:trace1] [pid 24591:tid 139759720564480] event.c(1895): Idle workers: 0
[Mon Jan 29 08:03:04.908328 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 3 started: pid 24603, gen 30, active 16/132, total 18/132/132
[Mon Jan 29 08:03:04.908496 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(716): Child 79 stopped: pid 20850, gen 29, active 16/132, total 17/86/132, quiescing 2
[Mon Jan 29 08:03:05.014824 2024] [mpm_event:debug] [pid 24603:tid 139759856973568] event.c(2395): AH02471: start_threads: Using epoll (wakeable)
[Mon Jan 29 08:03:05.029307 2024] [mpm_event:debug] [pid 24603:tid 139759720564480] event.c(1892): Too many open connections (0), not accepting new conns in this process
[Mon Jan 29 08:03:05.033225 2024] [mpm_event:trace1] [pid 24603:tid 139759720564480] event.c(1895): Idle workers: 3
[Mon Jan 29 08:03:05.910375 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 4 started: pid 24690, gen 30, active 17/132, total 18/86/132
[Mon Jan 29 08:03:05.975729 2024] [mpm_event:debug] [pid 24690:tid 139759856973568] event.c(2395): AH02471: start_threads: Using epoll (wakeable)
[Mon Jan 29 08:03:06.006641 2024] [mpm_event:debug] [pid 24690:tid 139759720564480] event.c(1892): Too many open connections (6), not accepting new conns in this process
[Mon Jan 29 08:03:06.007728 2024] [mpm_event:trace1] [pid 24690:tid 139759720564480] event.c(1895): Idle workers: 0
[Mon Jan 29 08:03:06.912213 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 5 started: pid 24709, gen 30, active 18/132, total 19/86/132
[Mon Jan 29 08:03:07.035438 2024] [mpm_event:debug] [pid 24709:tid 139759856973568] event.c(2395): AH02471: start_threads: Using epoll (wakeable)
[Mon Jan 29 08:03:07.064820 2024] [mpm_event:debug] [pid 24709:tid 139759720564480] event.c(1892): Too many open connections (5), not accepting new conns in this process
[Mon Jan 29 08:03:07.065639 2024] [mpm_event:trace1] [pid 24709:tid 139759720564480] event.c(1895): Idle workers: 1
..........
..........
..........
[Mon Jan 29 08:03:14.926642 2024] [mpm_event:info] [pid 29303:tid 139759872828544] AH00486: server seems busy, (you may need to increase StartServers, ThreadsPerChild or Min/MaxSpareThreads), spawning 8 children, there are around 0 idle threads, 29 active children, and 30 children that are shutting down
[Mon Jan 29 08:03:14.927470 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 20 started: pid 24873, gen 30, active 30/132, total 31/86/132
[Mon Jan 29 08:03:14.927900 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 21 started: pid 24874, gen 30, active 31/132, total 32/86/132
[Mon Jan 29 08:03:14.928331 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 22 started: pid 24875, gen 30, active 32/132, total 33/86/132
[Mon Jan 29 08:03:14.928738 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 23 started: pid 24876, gen 30, active 33/132, total 34/86/132
[Mon Jan 29 08:03:14.929142 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 24 started: pid 24877, gen 30, active 34/132, total 35/86/132
[Mon Jan 29 08:03:14.929506 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 25 started: pid 24878, gen 30, active 35/132, total 36/86/132
[Mon Jan 29 08:03:14.929893 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 27 started: pid 24879, gen 30, active 36/132, total 37/86/132
[Mon Jan 29 08:03:14.930263 2024] [mpm_event:debug] [pid 29303:tid 139759872828544] event.c(741): Child 28 started: pid 24880, gen 30, active 37/132, total 38/86/132
[Mon Jan 29 08:03:15.095599 2024] [mpm_event:debug] [pid 24875:tid 139759856973568] event.c(2395): AH02471: start_threads: Using epoll (wakeable)
[Mon Jan 29 08:03:15.107594 2024] [mpm_event:debug] [pid 24873:tid 139759856973568] event.c(2395): AH02471: start_threads: Using epoll (wakeable)
[Mon Jan 29 08:03:15.108810 2024] [mpm_event:debug] [pid 24874:tid 139759856973568] event.c(2395): AH02471: start_threads: Using epoll (wakeable)
[Mon Jan 29 08:03:15.109032 2024] [mpm_event:debug] [pid 24877:tid 139759856973568] event.c(2395): AH02471: start_threads: Using epoll (wakeable)
[Mon Jan 29 08:03:15.117436 2024] [mpm_event:debug] [pid 24873:tid 139759720564480] event.c(1892): Too many open connections (0), not accepting new conns in this process
[Mon Jan 29 08:03:15.117466 2024] [mpm_event:trace1] [pid 24873:tid 139759720564480] event.c(1895): Idle workers: 0
[Mon Jan 29 08:03:15.123705 2024] [mpm_event:debug] [pid 24877:tid 139759720564480] event.c(1892): Too many open connections (0), not accepting new conns in this process
[Mon Jan 29 08:03:15.123761 2024] [mpm_event:trace1] [pid 24877:tid 139759720564480] event.c(1895): Idle workers: 0
........
........
........
[Mon Jan 29 08:18:38.202850 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 131/132, total 132/132/132, idle threads 825, max workers 924
[Mon Jan 29 08:18:38.247684 2024] [mpm_event:trace6] [pid 25885:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:38.628382 2024] [mpm_event:trace6] [pid 25564:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:38.628469 2024] [mpm_event:trace6] [pid 25539:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:38.910546 2024] [mpm_event:trace6] [pid 25180:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.003809 2024] [mpm_event:trace6] [pid 25197:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.003781 2024] [mpm_event:trace6] [pid 25886:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.080928 2024] [mpm_event:trace6] [pid 25567:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.083700 2024] [mpm_event:trace6] [pid 24982:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.204016 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 131/132, total 132/132/132, idle threads 824, max workers 924
[Mon Jan 29 08:18:39.254909 2024] [mpm_event:trace6] [pid 25885:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.409146 2024] [mpm_event:trace6] [pid 25577:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.409132 2024] [mpm_event:trace6] [pid 25543:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.720029 2024] [mpm_event:trace6] [pid 25883:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:39.797372 2024] [mpm_event:trace6] [pid 25564:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:40.016911 2024] [mpm_event:trace6] [pid 25887:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:40.060063 2024] [mpm_event:trace6] [pid 25886:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:40.194472 2024] [mpm_event:trace6] [pid 25539:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:40.205143 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 131/132, total 132/132/132, idle threads 824, max workers 924
[Mon Jan 29 08:18:40.304130 2024] [mpm_event:trace6] [pid 25885:tid 139759720564480] event.c(1707): connections: 1 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:40.565580 2024] [mpm_event:trace6] [pid 25567:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:40.751005 2024] [mpm_event:trace6] [pid 25192:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:40.865651 2024] [mpm_event:trace6] [pid 25564:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:41.075927 2024] [mpm_event:trace6] [pid 25553:tid 139759720564480] event.c(1707): connections: 0 (clogged: 0 write-completion: 0 keep-alive: 0 lingering: 0 suspended: 0)
[Mon Jan 29 08:18:41.206274 2024] [mpm_event:trace5] [pid 29303:tid 139759872828544] event.c(2936): Not shutting down one child: active 131/132, total 132/132/132, idle threads 825, max workers 924
Comment 1 Yann Ylavic 2024-01-30 14:05:46 UTC 
Did you also apply r1897149 ?
Comment 2 mskogorev 2024-02-01 10:52:42 UTC 
I checked and I don't have this patch. I'll try to apply it and test it.

I will add feedback here a little later.
Comment 3 mskogorev 2024-02-05 10:45:20 UTC 
(In reply to Yann Ylavic from comment #1)
> Did you also apply r1897149 ?

I can't apply this patch. I think I need to apply some more patches.

root@2e40a91a2837:/build/apache2-2.4.52# patch -p1 < /build/1897149.patch 
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- httpd/httpd/branches/2.4.x/server/mpm/event/event.c	2022/01/17 08:31:13	1897148
|+++ httpd/httpd/branches/2.4.x/server/mpm/event/event.c	2022/01/17 08:43:50	1897149
--------------------------
File to patch: /build/apache2-2.4.52/server/mpm/event/event.c
patching file /build/apache2-2.4.52/server/mpm/event/event.c
Hunk #2 succeeded at 807 (offset 28 lines).
Hunk #3 succeeded at 2035 (offset 28 lines).
Hunk #4 succeeded at 2097 (offset 28 lines).
Hunk #5 succeeded at 2112 (offset 28 lines).
Hunk #6 FAILED at 3054.
1 out of 6 hunks FAILED -- saving rejects to file /build/apache2-2.4.52/server/mpm/event/event.c.rej


This is my "event.c":
            /* non-fatal death... note that it's gone in the scoreboard. */
            if (child_slot >= 0) {
                event_note_child_stopped(child_slot, 0, 0);

                if (processed_status == APEXIT_CHILDSICK) {
                    /* resource shortage, minimize the fork rate */
                    retained->idle_spawn_rate[child_slot % num_buckets] = 1;
                }
                else if (remaining_children_to_start) {
                    /* we're still doing a 1-for-1 replacement of dead
                     * children with new children
                     */
                    make_child(ap_server_conf, child_slot,
                               child_slot % num_buckets);
                    --remaining_children_to_start;
                }
            }

and with patched:
	            /* non-fatal death... note that it's gone in the scoreboard. */
	            if (child_slot >= 0) {
	                process_score *ps;
	
	                for (i = 0; i < threads_per_child; i++)
	                    ap_update_child_status_from_indexes(child_slot, i,
	                                                        SERVER_DEAD, NULL);
	
	                event_note_child_killed(child_slot, 0, 0);
	                ps = &ap_scoreboard_image->parent[child_slot];
	                if (ps->quiescing != 2)
	                    retained->active_daemons--;
	                ps->quiescing = 0;
	                /* NOTE: We don't dec in the (child_slot < 0) case! */
	                retained->total_daemons--;
	                if (processed_status == APEXIT_CHILDSICK) {
	                    /* resource shortage, minimize the fork rate */
	                    retained->idle_spawn_rate[child_slot % num_buckets] = 1;
	                }
	                else if (remaining_children_to_start) {
	                    /* we're still doing a 1-for-1 replacement of dead
	                     * children with new children
	                     */
	                    make_child(ap_server_conf, child_slot,
	                               child_slot % num_buckets);
	                    --remaining_children_to_start;
	                }
	            }


Do you know which patches I should apply before r1897149?
Comment 4 Yann Ylavic 2024-02-05 11:53:25 UTC 
To resolve this conflict you should apply r1897149 before r1901199 (which you said you applied already).

But it also means that by applying r1901199 you probably had to resolve a conflict around this code already, namely in server_main_loop() function:

-                if (ps->quiescing != 2)
-                    retained->active_daemons--;

While the original code (before/without r1897149) was:

-                if (!ps->quiescing)
-                    retained->active_daemons--;

In r1901199 this moved to event_note_child_stopped() as:

+        if (ps->quiescing != 2) { /* vs perform_idle_server_maintenance() */
+            retained->active_daemons--;
+        }


So depending on how you resolved this conflict with r1901199 you may have the correct code already in event_note_child_stopped() (the test should be "(ps->quiescing != 2)"), but I'd suggest restarting from the original event.c from 2.4.52 and applying r1897149 and r1901199 in this order to avoid conflicts and be sure to run with the correct code.
Comment 5 mskogorev 2024-02-05 13:40:11 UTC 
(In reply to Yann Ylavic from comment #4)
> To resolve this conflict you should apply r1897149 before r1901199 (which
> you said you applied already).
> 
> But it also means that by applying r1901199 you probably had to resolve a
> conflict around this code already, namely in server_main_loop() function:
> 
> -                if (ps->quiescing != 2)
> -                    retained->active_daemons--;
> 
> While the original code (before/without r1897149) was:
> 
> -                if (!ps->quiescing)
> -                    retained->active_daemons--;
> 
> In r1901199 this moved to event_note_child_stopped() as:
> 
> +        if (ps->quiescing != 2) { /* vs perform_idle_server_maintenance() */
> +            retained->active_daemons--;
> +        }
> 
> 
> So depending on how you resolved this conflict with r1901199 you may have
> the correct code already in event_note_child_stopped() (the test should be
> "(ps->quiescing != 2)"), but I'd suggest restarting from the original
> event.c from 2.4.52 and applying r1897149 and r1901199 in this order to
> avoid conflicts and be sure to run with the correct code.

Thanks for telling me about the sequence. Earlier, I manually fixed event.c to fix the conflict. 

I will build apache2 with these patches today, test them, and give feedback. Thanks a lot.
Comment 6 mskogorev 2024-02-26 08:47:45 UTC 
today we upgraded apache2 version with both patches, I think I can give feedback in a few days.
Comment 7 mskogorev 2024-02-29 22:14:09 UTC 
This problem is being reproduced again. Do you have any ideas? Maybe I can provide some additional logs or do I need to apply some other patch?