Bug 58960 - Proxying Docker Registry HTTP PATCH command fails
Summary: Proxying Docker Registry HTTP PATCH command fails
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy_http (show other bugs)
Version: 2.4.18
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-02-02 16:39 UTC by dhertel
Modified: 2019-04-23 14:37 UTC (History)
1 user (show)



Attachments
VirtualHost configuration (1.86 KB, text/plain)
2016-02-02 16:39 UTC, dhertel
Details

Note You need to log in before you can comment on or make changes to this bug.
Description dhertel 2016-02-02 16:39:02 UTC
Created attachment 33514 [details]
VirtualHost configuration

I'm trying to set up Apache as a front end to a Docker registry as described here: https://docs.docker.com/registry/apache/

The host uses SNI if that matters.

When uploading an image into the docker registry using "docker push" the first 2 layers are submitted successfully using POST. The third layer, however, is bigger and the docker client tries to use PATCH instead, which fails.

Directly after server start-up or reload it will often get to about 1mb - subsequent requests will not (but maybe they fail before the output of the push command is refreshed - I don't know)

So, here's what happens:

$ docker push docker.redacted-tld.com/my-fancy-image:v1
The push refers to a repository [docker.redacted-tld.com/my-fancy-image] (len: 1)
ef3a12090a7a: Image already exists 
d72639db7d88: Image already exists 
406b6670cf87: Pushing [>                                                  ] 526.3 kB/56.62 MB
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br />\n</p>\n<hr>\n<address>Apache/2.4.18 (Debian) Server at docker.redacted-tld.com Port 443</address>\n</body></html>\n"


This is the log output from the registry:
time="2016-02-02T16:30:43Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=a523f926-7d6c-4cd1-8efc-4239ba33ff27 http.request.method=HEAD http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=3.864329ms http.response.status=200 http.response.written=0 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:43Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=ae6f39a5-0323-472f-80a4-46dc25a3f66e http.request.method=HEAD http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=6.432349ms http.response.status=200 http.response.written=0 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:43Z" level=error msg="response completed with error" err.code="BLOB_UNKNOWN" err.detail=sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 err.message="blob unknown to registry" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=2ae9b7c2-9040-4847-8b14-f32de106c476 http.request.method=HEAD http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.243912ms http.response.status=404 http.response.written=157 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 vars.digest="sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225" vars.name=my-fancy-image version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 157 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:44Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=18ff6234-bfa7-46c3-b143-00b0a3178434 http.request.method=POST http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/uploads/" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.duration=6.181671ms http.response.status=202 http.response.written=0 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:44 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:44Z" level=error msg="client disconnected during blob PATCH" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=c31e96a3-8fd5-4b63-9148-dd6783320e18 http.request.method=PATCH http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/uploads/9dfe32b8-0e17-459d-bc63-334de1bbd723?_state=2q8JSZCMkz83rbx9JV1Z_Bnn0ZbmBrt7KmWo0xzv4IZ7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjlkZmUzMmI4LTBlMTctNDU5ZC1iYzYzLTMzNGRlMWJiZDcyMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMDo0NC4wMTI5MTU3N1oifQ%3D%3D" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 vars.name=my-fancy-image vars.uuid=9dfe32b8-0e17-459d-bc63-334de1bbd723 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:44 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/9dfe32b8-0e17-459d-bc63-334de1bbd723?_state=2q8JSZCMkz83rbx9JV1Z_Bnn0ZbmBrt7KmWo0xzv4IZ7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjlkZmUzMmI4LTBlMTctNDU5ZC1iYzYzLTMzNGRlMWJiZDcyMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMDo0NC4wMTI5MTU3N1oifQ%3D%3D HTTP/1.1" 499 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"


Apache Logs from the requests:
==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 6071 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:30:44 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 6503 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:30:44 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/9dfe32b8-0e17-459d-bc63-334de1bbd723?_state=2q8JSZCMkz83rbx9JV1Z_Bnn0ZbmBrt7KmWo0xzv4IZ7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjlkZmUzMmI4LTBlMTctNDU5ZC1iYzYzLTMzNGRlMWJiZDcyMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMDo0NC4wMTI5MTU3N1oifQ%3D%3D HTTP/1.1" 400 6252 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - - [02/Feb/2016:16:32:39 +0000] "GET /v2/ HTTP/1.1" 401 6460 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 6071 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 6503 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D HTTP/1.1" 400 6252 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:32:39.741479 2016] [proxy_fcgi:debug] [pid 22885:tid 139957037344512] mod_proxy_fcgi.c(882): [client 172.19.28.1:38496] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D
[Tue Feb 02 16:32:39.741495 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:32:39.741509 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2213): [client 172.19.28.1:38496] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D to localhost:5000
[Tue Feb 02 16:32:39.741523 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2422): [client 172.19.28.1:38496] AH00947: connected /v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D to localhost:5000
[Tue Feb 02 16:32:39.752653 2016] [ssl:info] [pid 22885:tid 139957037344512] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old
[Tue Feb 02 16:32:39.752678 2016] [proxy_http:error] [pid 22885:tid 139957037344512] (20014)Internal error: [client 172.19.28.1:38496] AH02608: read request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:32:39.752689 2016] [proxy_http:error] [pid 22885:tid 139957037344512] [client 172.19.28.1:38496] AH01097: pass request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:32:39.752697 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2175): AH00943: HTTP: has released connection for (localhost)
[Tue Feb 02 16:32:39.752763 2016] [headers:debug] [pid 22885:tid 139957037344512] mod_headers.c(899): AH01503: headers: ap_headers_error_filter()
[Tue Feb 02 16:32:39.752996 2016] [ssl:debug] [pid 22885:tid 139957037344512] ssl_engine_io.c(1016): [client 172.19.28.1:38496] AH02001: Connection closed to child 88 with standard shutdown (server docker.redacted-tld.com:443)

==> docker.redacted-tld.com.access.log <==
109.233.151.161 - - [01/Feb/2016:16:46:35 +0000] "GET / HTTP/1.1" 301 591 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36"
109.233.151.161 - - [02/Feb/2016:08:08:45 +0000] "GET / HTTP/1.1" 301 591 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36"

==> docker.redacted-tld.com.error.log <==

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:23.917600 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:23.917659 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:23.917682 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:23.917699 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:23.917725 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:23.917743 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:23.917750 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:23.917862 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:23.917889 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:23.942824 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0xb2 -> subcache 18)
[Tue Feb 02 16:33:23.942848 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(845): AH00847: insert happened at idx=0, data=(0:32)
[Tue Feb 02 16:33:23.942855 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/204
[Tue Feb 02 16:33:23.942860 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:23.943090 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_engine_kernel.c(354): [client 172.19.28.1:38497] AH02034: Initial (No.1) HTTPS request received for child 76 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.943131 2016] [authz_core:debug] [pid 22885:tid 139957138056960] mod_authz_core.c(809): [client 172.19.28.1:38497] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:23.943140 2016] [authz_core:debug] [pid 22885:tid 139957138056960] mod_authz_core.c(809): [client 172.19.28.1:38497] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:23.943159 2016] [headers:debug] [pid 22885:tid 139957138056960] mod_headers.c(899): AH01503: headers: ap_headers_error_filter()
[Tue Feb 02 16:33:23.943234 2016] [ssl:info] [pid 22885:tid 139957138056960] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - - [02/Feb/2016:16:33:23 +0000] "GET /v2/ HTTP/1.1" 401 6460 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:23.946891 2016] [ssl:debug] [pid 22885:tid 139957129664256] ssl_engine_io.c(1016): [client 172.19.28.1:38497] AH02001: Connection closed to child 77 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.947308 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:23.947343 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:23.947369 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:23.947386 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:23.947395 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:23.947420 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:23.947431 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:23.947588 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:23.947626 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:23.977229 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x2c -> subcache 12)
[Tue Feb 02 16:33:23.977275 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(845): AH00847: insert happened at idx=0, data=(0:32)
[Tue Feb 02 16:33:23.977282 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/204
[Tue Feb 02 16:33:23.977287 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:23.977410 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_engine_kernel.c(354): [client 172.19.28.1:38498] AH02034: Initial (No.1) HTTPS request received for child 78 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.977451 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:23.977460 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:23.980293 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:23.980303 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:23.980341 2016] [proxy:debug] [pid 22885:tid 139957121271552] mod_proxy.c(1160): [client 172.19.28.1:38498] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:23.980350 2016] [proxy_fcgi:debug] [pid 22885:tid 139957121271552] mod_proxy_fcgi.c(879): [client 172.19.28.1:38498] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:23.980361 2016] [proxy_fcgi:debug] [pid 22885:tid 139957121271552] mod_proxy_fcgi.c(882): [client 172.19.28.1:38498] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
[Tue Feb 02 16:33:23.980369 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:23.980377 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2213): [client 172.19.28.1:38498] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to localhost:5000
[Tue Feb 02 16:33:23.980386 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2422): [client 172.19.28.1:38498] AH00947: connected /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to localhost:5000
[Tue Feb 02 16:33:23.980479 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2799): AH02824: HTTP: connection established with 127.0.0.1:5000 (localhost)
[Tue Feb 02 16:33:23.980494 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2965): AH00962: HTTP: connection complete to 127.0.0.1:5000 (localhost)
[Tue Feb 02 16:33:23.985888 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:23.985984 2016] [ssl:info] [pid 22885:tid 139957121271552] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:23 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:23.986766 2016] [ssl:debug] [pid 22885:tid 139957311522560] ssl_engine_io.c(1016): [client 172.19.28.1:38498] AH02001: Connection closed to child 64 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.988762 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:23.988802 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:23.988824 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:23.988837 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:23.988846 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:23.988871 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:23.988882 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:23.989040 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:23.989079 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.014270 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x8a -> subcache 10)
[Tue Feb 02 16:33:24.014298 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(845): AH00847: insert happened at idx=4, data=(816:848)
[Tue Feb 02 16:33:24.014304 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/5, data_pos/data_used=0/1020
[Tue Feb 02 16:33:24.014309 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.014455 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_engine_kernel.c(354): [client 172.19.28.1:38501] AH02034: Initial (No.1) HTTPS request received for child 79 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.014499 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.014509 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.017333 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.017342 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.017380 2016] [proxy:debug] [pid 22885:tid 139957112878848] mod_proxy.c(1160): [client 172.19.28.1:38501] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.017390 2016] [proxy_fcgi:debug] [pid 22885:tid 139957112878848] mod_proxy_fcgi.c(879): [client 172.19.28.1:38501] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.017396 2016] [proxy_fcgi:debug] [pid 22885:tid 139957112878848] mod_proxy_fcgi.c(882): [client 172.19.28.1:38501] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239
[Tue Feb 02 16:33:24.017405 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.017430 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2213): [client 172.19.28.1:38501] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 to localhost:5000
[Tue Feb 02 16:33:24.017438 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2422): [client 172.19.28.1:38501] AH00947: connected /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 to localhost:5000
[Tue Feb 02 16:33:24.022976 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:24.023185 2016] [ssl:info] [pid 22885:tid 139957112878848] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.023842 2016] [ssl:debug] [pid 22885:tid 139957303129856] ssl_engine_io.c(1016): [client 172.19.28.1:38501] AH02001: Connection closed to child 65 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.024815 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:24.024851 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:24.024868 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:24.024879 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:24.024887 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:24.024905 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:24.024914 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:24.025022 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:24.025050 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.052675 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x80 -> subcache 0)
[Tue Feb 02 16:33:24.052702 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(845): AH00847: insert happened at idx=1, data=(204:236)
[Tue Feb 02 16:33:24.052708 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/2, data_pos/data_used=0/408
[Tue Feb 02 16:33:24.052716 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.052855 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_engine_kernel.c(354): [client 172.19.28.1:38502] AH02034: Initial (No.1) HTTPS request received for child 7 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.052897 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.052907 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.055742 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.055769 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.055806 2016] [proxy:debug] [pid 22884:tid 139957180020480] mod_proxy.c(1160): [client 172.19.28.1:38502] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.055817 2016] [proxy_fcgi:debug] [pid 22884:tid 139957180020480] mod_proxy_fcgi.c(879): [client 172.19.28.1:38502] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.055824 2016] [proxy_fcgi:debug] [pid 22884:tid 139957180020480] mod_proxy_fcgi.c(882): [client 172.19.28.1:38502] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225
[Tue Feb 02 16:33:24.055833 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.055840 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2213): [client 172.19.28.1:38502] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 to localhost:5000
[Tue Feb 02 16:33:24.055848 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2422): [client 172.19.28.1:38502] AH00947: connected /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 to localhost:5000
[Tue Feb 02 16:33:24.060704 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:24.060839 2016] [ssl:info] [pid 22884:tid 139957180020480] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 6071 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.061504 2016] [ssl:debug] [pid 22884:tid 139957062522624] ssl_engine_io.c(1016): [client 172.19.28.1:38502] AH02001: Connection closed to child 21 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.064122 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:24.064160 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:24.064178 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:24.064190 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:24.064199 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:24.064221 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:24.064231 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:24.064385 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:24.064424 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.095162 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0xa1 -> subcache 1)
[Tue Feb 02 16:33:24.095186 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(728): AH00842: expiring 1 and reclaiming 0 removed socache entries
[Tue Feb 02 16:33:24.095192 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(747): AH00843: we now have 1 socache entries
[Tue Feb 02 16:33:24.095225 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(845): AH00847: insert happened at idx=2, data=(408:440)
[Tue Feb 02 16:33:24.095231 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=1/2, data_pos/data_used=204/408
[Tue Feb 02 16:33:24.095235 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.095418 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_engine_kernel.c(354): [client 172.19.28.1:38503] AH02034: Initial (No.1) HTTPS request received for child 80 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.095455 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.095464 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.098288 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.098300 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.098333 2016] [proxy:debug] [pid 22885:tid 139957104486144] mod_proxy.c(1160): [client 172.19.28.1:38503] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.098341 2016] [proxy_fcgi:debug] [pid 22885:tid 139957104486144] mod_proxy_fcgi.c(879): [client 172.19.28.1:38503] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/uploads/ proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.098347 2016] [proxy_fcgi:debug] [pid 22885:tid 139957104486144] mod_proxy_fcgi.c(882): [client 172.19.28.1:38503] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/uploads/
[Tue Feb 02 16:33:24.098359 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.098369 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2213): [client 172.19.28.1:38503] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/uploads/ to localhost:5000
[Tue Feb 02 16:33:24.098378 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2422): [client 172.19.28.1:38503] AH00947: connected /v2/my-fancy-image/blobs/uploads/ to localhost:5000
[Tue Feb 02 16:33:24.105509 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:24.105712 2016] [ssl:info] [pid 22885:tid 139957104486144] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 6503 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.106605 2016] [ssl:debug] [pid 22885:tid 139957096093440] ssl_engine_io.c(1016): [client 172.19.28.1:38503] AH02001: Connection closed to child 81 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.146414 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:24.146459 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:24.146479 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:24.146489 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:24.146497 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:24.146543 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:24.146552 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:24.146667 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:24.146697 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.179130 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x17 -> subcache 23)
[Tue Feb 02 16:33:24.179148 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(728): AH00842: expiring 1 and reclaiming 0 removed socache entries
[Tue Feb 02 16:33:24.179154 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(747): AH00843: we now have 2 socache entries
[Tue Feb 02 16:33:24.179159 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(845): AH00847: insert happened at idx=3, data=(612:644)
[Tue Feb 02 16:33:24.179163 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=1/3, data_pos/data_used=204/612
[Tue Feb 02 16:33:24.179168 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.179772 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_engine_kernel.c(354): [client 172.19.28.1:38504] AH02034: Initial (No.1) HTTPS request received for child 22 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.179815 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.179824 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.182960 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.182977 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.183027 2016] [proxy:debug] [pid 22884:tid 139957054129920] mod_proxy.c(1160): [client 172.19.28.1:38504] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.183040 2016] [proxy_fcgi:debug] [pid 22884:tid 139957054129920] mod_proxy_fcgi.c(879): [client 172.19.28.1:38504] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.183056 2016] [proxy_fcgi:debug] [pid 22884:tid 139957054129920] mod_proxy_fcgi.c(882): [client 172.19.28.1:38504] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D
[Tue Feb 02 16:33:24.183070 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.183097 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2213): [client 172.19.28.1:38504] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D to localhost:5000
[Tue Feb 02 16:33:24.183113 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2422): [client 172.19.28.1:38504] AH00947: connected /v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D to localhost:5000
[Tue Feb 02 16:33:24.198730 2016] [ssl:info] [pid 22884:tid 139957054129920] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old
[Tue Feb 02 16:33:24.198774 2016] [proxy_http:error] [pid 22884:tid 139957054129920] (20014)Internal error: [client 172.19.28.1:38504] AH02608: read request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:33:24.198788 2016] [proxy_http:error] [pid 22884:tid 139957054129920] [client 172.19.28.1:38504] AH01097: pass request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:33:24.198795 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2175): AH00943: HTTP: has released connection for (localhost)
[Tue Feb 02 16:33:24.198886 2016] [headers:debug] [pid 22884:tid 139957054129920] mod_headers.c(899): AH01503: headers: ap_headers_error_filter()

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D HTTP/1.1" 400 6252 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.199152 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_engine_io.c(1016): [client 172.19.28.1:38504] AH02001: Connection closed to child 22 with standard shutdown (server docker.redacted-tld.com:443)


I really don't know what to make of this
Comment 1 Andrey L 2019-04-23 14:37:13 UTC
I've got the same issue yesterday.
Did you find any fix for this issue? 

Thanks in advance.