We had production servers that failed to start because a certificate was revoked (LetsEncrypt problem of today obviously). Misleading error message AH02565 (could be interesting to fix it if possible). An invalid certificate should not block the whole server. Because of one invalid vhost, several hundreds sites are unavailable. "httpd -t" says the syntax is OK, so a (graceful) reload stops the service.
AH02565 is logged when a certificate and its key do not match. This has nothing to do with revocation, but points to a misconfiguration. For example, if you copy over a renewed certificate, but fail to also copy the corresponding key. One could argue if such a situation should stop the server from reloading, sacrificing just the one site to become inoperable. The check is there since 2014, so this is no new behaviour. You're sure that in todays scramble to correct Lets Encrypt configurations that is not what happened?
It was definitely provoked by the LetsEncrypt certificate revocation problem. We're using mod_md. Maybe mod_md did half of the job because of that problem, that's possible.
This will be hard to analyze. Let me explain: When a certificate for xxx.com is renewed. - $server_root/md/domains/xxx.com contains the working certs - $server_root/md/staging/xxx.com contains all about the renewal If the server reloads, it checks "staging/*" for complete file sets. When that indicates success, it - *creates* and *copies* a "tmp/xxx.com". The copy really parses key and certificates and PEM serializes them again - if *moves* the whole dir "domains/xxx.com" to "archive/xxx.com.N" to preserve the old file set - then it *moves* "tmp/xxx.com" to "domains/xxx.com". - then it *deletes" "staging/xxx.com" This is all done so that no interruption will produce a "half-updated" set of files where things do not match. In Apache httpd 2.4.49 the test for matching key and certificate was added during activation of a staging area to make sure mod_md never activates a set of files that do not match. You see, there is considerate thought gone into avoiding the thing you experienced. Especially with 2.4.49 or newer, the server should never load a cert+key that do not match, even if something was messed up in the "staging" subdir. Any thoughts? Otherwise I think we need to close this as not reproducable.
Problem was in 2.4.37. Are you confident that it cannot hapen in 2.4.49+ anymore? Here, we speak about a certificate that was fully loaded, then revoked.
Under the premises that no one is messing with the file system, e.g. a job that distributes certificates among nodes or other such production jobs, it should never have happened in the first place. In 2.4.49, an additional sanity check was added that "staging" file sets do not get activates if cert+key do not match. Assuming that is what affected your site. You can check in "md/archive/xxx.com* if there is an archived set from the time the problem occurred. That would indicate that a renewal was made and the faulty set of file came from there. If there is no archived set from that time, then this did not happen on a renewal. Then then files in "md/domains/xxx.com" that were working got changed. Since mod_md replaces only directory and has no read/write access there when handling traffic, it strongly hints to an outside agency that messed with the files.