58960 – Proxying Docker Registry HTTP PATCH command fails

Bug 58960 - Proxying Docker Registry HTTP PATCH command fails

Summary: Proxying Docker Registry HTTP PATCH command fails

Status:	NEW

Alias:	None

Product:	Apache httpd-2
Classification:	Unclassified
Component:	mod_proxy_http (show other bugs)
Version:	2.4.18
Hardware:	PC Linux

Importance:	P2 normal (vote)
Target Milestone:	---
Assignee:	Apache HTTPD Bugs Mailing List

URL:
Keywords:

Depends on:
Blocks:

Reported:	2016-02-02 16:39 UTC by Dominik Keil
Modified:	2019-04-23 14:37 UTC (History)
CC List:	1 user (show)

Attachments
VirtualHost configuration (1.86 KB, text/plain) 2016-02-02 16:39 UTC, Dominik Keil	Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Dominik Keil 2016-02-02 16:39:02 UTC

Created attachment 33514 [details]
VirtualHost configuration

I'm trying to set up Apache as a front end to a Docker registry as described here: https://docs.docker.com/registry/apache/

The host uses SNI if that matters.

When uploading an image into the docker registry using "docker push" the first 2 layers are submitted successfully using POST. The third layer, however, is bigger and the docker client tries to use PATCH instead, which fails.

Directly after server start-up or reload it will often get to about 1mb - subsequent requests will not (but maybe they fail before the output of the push command is refreshed - I don't know)

So, here's what happens:

$ docker push docker.redacted-tld.com/my-fancy-image:v1
The push refers to a repository [docker.redacted-tld.com/my-fancy-image] (len: 1)
ef3a12090a7a: Image already exists 
d72639db7d88: Image already exists 
406b6670cf87: Pushing [>                                                  ] 526.3 kB/56.62 MB
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1>\n<p>Your browser sent a request that this server could not understand.<br />\n</p>\n<hr>\n<address>Apache/2.4.18 (Debian) Server at docker.redacted-tld.com Port 443</address>\n</body></html>\n"


This is the log output from the registry:
time="2016-02-02T16:30:43Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=a523f926-7d6c-4cd1-8efc-4239ba33ff27 http.request.method=HEAD http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=3.864329ms http.response.status=200 http.response.written=0 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:43Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=ae6f39a5-0323-472f-80a4-46dc25a3f66e http.request.method=HEAD http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=6.432349ms http.response.status=200 http.response.written=0 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:43Z" level=error msg="response completed with error" err.code="BLOB_UNKNOWN" err.detail=sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 err.message="blob unknown to registry" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=2ae9b7c2-9040-4847-8b14-f32de106c476 http.request.method=HEAD http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.243912ms http.response.status=404 http.response.written=157 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 vars.digest="sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225" vars.name=my-fancy-image version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 157 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:44Z" level=info msg="response completed" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=18ff6234-bfa7-46c3-b143-00b0a3178434 http.request.method=POST http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/uploads/" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" http.response.duration=6.181671ms http.response.status=202 http.response.written=0 instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:44 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
time="2016-02-02T16:30:44Z" level=error msg="client disconnected during blob PATCH" go.version=go1.5.2 http.request.host=docker.redacted-tld.com http.request.id=c31e96a3-8fd5-4b63-9148-dd6783320e18 http.request.method=PATCH http.request.remoteaddr=172.19.28.1 http.request.uri="/v2/my-fancy-image/blobs/uploads/9dfe32b8-0e17-459d-bc63-334de1bbd723?_state=2q8JSZCMkz83rbx9JV1Z_Bnn0ZbmBrt7KmWo0xzv4IZ7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjlkZmUzMmI4LTBlMTctNDU5ZC1iYzYzLTMzNGRlMWJiZDcyMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMDo0NC4wMTI5MTU3N1oifQ%3D%3D" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64" instance.id=258bdb39-c474-4257-84b5-f99d0e642ab6 vars.name=my-fancy-image vars.uuid=9dfe32b8-0e17-459d-bc63-334de1bbd723 version=v2.2.1
172.19.32.1 - - [02/Feb/2016:16:30:44 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/9dfe32b8-0e17-459d-bc63-334de1bbd723?_state=2q8JSZCMkz83rbx9JV1Z_Bnn0ZbmBrt7KmWo0xzv4IZ7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjlkZmUzMmI4LTBlMTctNDU5ZC1iYzYzLTMzNGRlMWJiZDcyMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMDo0NC4wMTI5MTU3N1oifQ%3D%3D HTTP/1.1" 499 0 "" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"


Apache Logs from the requests:
==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:30:43 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 6071 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:30:44 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 6503 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:30:44 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/9dfe32b8-0e17-459d-bc63-334de1bbd723?_state=2q8JSZCMkz83rbx9JV1Z_Bnn0ZbmBrt7KmWo0xzv4IZ7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjlkZmUzMmI4LTBlMTctNDU5ZC1iYzYzLTMzNGRlMWJiZDcyMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMDo0NC4wMTI5MTU3N1oifQ%3D%3D HTTP/1.1" 400 6252 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - - [02/Feb/2016:16:32:39 +0000] "GET /v2/ HTTP/1.1" 401 6460 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 6071 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 6503 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"
myownhostname - authenticateduser [02/Feb/2016:16:32:39 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D HTTP/1.1" 400 6252 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:32:39.741479 2016] [proxy_fcgi:debug] [pid 22885:tid 139957037344512] mod_proxy_fcgi.c(882): [client 172.19.28.1:38496] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D
[Tue Feb 02 16:32:39.741495 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:32:39.741509 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2213): [client 172.19.28.1:38496] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D to localhost:5000
[Tue Feb 02 16:32:39.741523 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2422): [client 172.19.28.1:38496] AH00947: connected /v2/my-fancy-image/blobs/uploads/0f63ae74-8409-4905-b890-2f8ea8e1927c?_state=KwN5Bbx6ZIGEbqMR2QPZA3Jh32xesOUaARoJ1jxIHh97Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjBmNjNhZTc0LTg0MDktNDkwNS1iODkwLTJmOGVhOGUxOTI3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMjozOS42NzYyNjMwMDFaIn0%3D to localhost:5000
[Tue Feb 02 16:32:39.752653 2016] [ssl:info] [pid 22885:tid 139957037344512] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old
[Tue Feb 02 16:32:39.752678 2016] [proxy_http:error] [pid 22885:tid 139957037344512] (20014)Internal error: [client 172.19.28.1:38496] AH02608: read request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:32:39.752689 2016] [proxy_http:error] [pid 22885:tid 139957037344512] [client 172.19.28.1:38496] AH01097: pass request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:32:39.752697 2016] [proxy:debug] [pid 22885:tid 139957037344512] proxy_util.c(2175): AH00943: HTTP: has released connection for (localhost)
[Tue Feb 02 16:32:39.752763 2016] [headers:debug] [pid 22885:tid 139957037344512] mod_headers.c(899): AH01503: headers: ap_headers_error_filter()
[Tue Feb 02 16:32:39.752996 2016] [ssl:debug] [pid 22885:tid 139957037344512] ssl_engine_io.c(1016): [client 172.19.28.1:38496] AH02001: Connection closed to child 88 with standard shutdown (server docker.redacted-tld.com:443)

==> docker.redacted-tld.com.access.log <==
109.233.151.161 - - [01/Feb/2016:16:46:35 +0000] "GET / HTTP/1.1" 301 591 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36"
109.233.151.161 - - [02/Feb/2016:08:08:45 +0000] "GET / HTTP/1.1" 301 591 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36"

==> docker.redacted-tld.com.error.log <==

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:23.917600 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:23.917659 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:23.917682 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:23.917699 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:23.917725 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:23.917743 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:23.917750 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:23.917862 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:23.917889 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:23.942824 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0xb2 -> subcache 18)
[Tue Feb 02 16:33:23.942848 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(845): AH00847: insert happened at idx=0, data=(0:32)
[Tue Feb 02 16:33:23.942855 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/204
[Tue Feb 02 16:33:23.942860 2016] [socache_shmcb:debug] [pid 22885:tid 139957138056960] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:23.943090 2016] [ssl:debug] [pid 22885:tid 139957138056960] ssl_engine_kernel.c(354): [client 172.19.28.1:38497] AH02034: Initial (No.1) HTTPS request received for child 76 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.943131 2016] [authz_core:debug] [pid 22885:tid 139957138056960] mod_authz_core.c(809): [client 172.19.28.1:38497] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:23.943140 2016] [authz_core:debug] [pid 22885:tid 139957138056960] mod_authz_core.c(809): [client 172.19.28.1:38497] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:23.943159 2016] [headers:debug] [pid 22885:tid 139957138056960] mod_headers.c(899): AH01503: headers: ap_headers_error_filter()
[Tue Feb 02 16:33:23.943234 2016] [ssl:info] [pid 22885:tid 139957138056960] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - - [02/Feb/2016:16:33:23 +0000] "GET /v2/ HTTP/1.1" 401 6460 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:23.946891 2016] [ssl:debug] [pid 22885:tid 139957129664256] ssl_engine_io.c(1016): [client 172.19.28.1:38497] AH02001: Connection closed to child 77 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.947308 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:23.947343 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:23.947369 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:23.947386 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:23.947395 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:23.947420 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:23.947431 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:23.947588 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:23.947626 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:23.977229 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x2c -> subcache 12)
[Tue Feb 02 16:33:23.977275 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(845): AH00847: insert happened at idx=0, data=(0:32)
[Tue Feb 02 16:33:23.977282 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/204
[Tue Feb 02 16:33:23.977287 2016] [socache_shmcb:debug] [pid 22885:tid 139957121271552] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:23.977410 2016] [ssl:debug] [pid 22885:tid 139957121271552] ssl_engine_kernel.c(354): [client 172.19.28.1:38498] AH02034: Initial (No.1) HTTPS request received for child 78 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.977451 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:23.977460 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:23.980293 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:23.980303 2016] [authz_core:debug] [pid 22885:tid 139957121271552] mod_authz_core.c(809): [client 172.19.28.1:38498] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:23.980341 2016] [proxy:debug] [pid 22885:tid 139957121271552] mod_proxy.c(1160): [client 172.19.28.1:38498] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:23.980350 2016] [proxy_fcgi:debug] [pid 22885:tid 139957121271552] mod_proxy_fcgi.c(879): [client 172.19.28.1:38498] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:23.980361 2016] [proxy_fcgi:debug] [pid 22885:tid 139957121271552] mod_proxy_fcgi.c(882): [client 172.19.28.1:38498] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
[Tue Feb 02 16:33:23.980369 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:23.980377 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2213): [client 172.19.28.1:38498] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to localhost:5000
[Tue Feb 02 16:33:23.980386 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2422): [client 172.19.28.1:38498] AH00947: connected /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 to localhost:5000
[Tue Feb 02 16:33:23.980479 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2799): AH02824: HTTP: connection established with 127.0.0.1:5000 (localhost)
[Tue Feb 02 16:33:23.980494 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2965): AH00962: HTTP: connection complete to 127.0.0.1:5000 (localhost)
[Tue Feb 02 16:33:23.985888 2016] [proxy:debug] [pid 22885:tid 139957121271552] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:23.985984 2016] [ssl:info] [pid 22885:tid 139957121271552] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:23 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:23.986766 2016] [ssl:debug] [pid 22885:tid 139957311522560] ssl_engine_io.c(1016): [client 172.19.28.1:38498] AH02001: Connection closed to child 64 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:23.988762 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:23.988802 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:23.988824 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:23.988837 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:23.988846 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:23.988871 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:23.988882 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:23.989040 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:23.989079 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.014270 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x8a -> subcache 10)
[Tue Feb 02 16:33:24.014298 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(845): AH00847: insert happened at idx=4, data=(816:848)
[Tue Feb 02 16:33:24.014304 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/5, data_pos/data_used=0/1020
[Tue Feb 02 16:33:24.014309 2016] [socache_shmcb:debug] [pid 22885:tid 139957112878848] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.014455 2016] [ssl:debug] [pid 22885:tid 139957112878848] ssl_engine_kernel.c(354): [client 172.19.28.1:38501] AH02034: Initial (No.1) HTTPS request received for child 79 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.014499 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.014509 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.017333 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.017342 2016] [authz_core:debug] [pid 22885:tid 139957112878848] mod_authz_core.c(809): [client 172.19.28.1:38501] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.017380 2016] [proxy:debug] [pid 22885:tid 139957112878848] mod_proxy.c(1160): [client 172.19.28.1:38501] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.017390 2016] [proxy_fcgi:debug] [pid 22885:tid 139957112878848] mod_proxy_fcgi.c(879): [client 172.19.28.1:38501] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.017396 2016] [proxy_fcgi:debug] [pid 22885:tid 139957112878848] mod_proxy_fcgi.c(882): [client 172.19.28.1:38501] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239
[Tue Feb 02 16:33:24.017405 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.017430 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2213): [client 172.19.28.1:38501] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 to localhost:5000
[Tue Feb 02 16:33:24.017438 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2422): [client 172.19.28.1:38501] AH00947: connected /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 to localhost:5000
[Tue Feb 02 16:33:24.022976 2016] [proxy:debug] [pid 22885:tid 139957112878848] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:24.023185 2016] [ssl:info] [pid 22885:tid 139957112878848] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:30538e4e68350bdb08c3196ab30182fb7320770f9902dfdbbe40d644a9194239 HTTP/1.1" 200 6295 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.023842 2016] [ssl:debug] [pid 22885:tid 139957303129856] ssl_engine_io.c(1016): [client 172.19.28.1:38501] AH02001: Connection closed to child 65 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.024815 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:24.024851 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:24.024868 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:24.024879 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:24.024887 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:24.024905 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:24.024914 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:24.025022 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:24.025050 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.052675 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x80 -> subcache 0)
[Tue Feb 02 16:33:24.052702 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(845): AH00847: insert happened at idx=1, data=(204:236)
[Tue Feb 02 16:33:24.052708 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/2, data_pos/data_used=0/408
[Tue Feb 02 16:33:24.052716 2016] [socache_shmcb:debug] [pid 22884:tid 139957180020480] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.052855 2016] [ssl:debug] [pid 22884:tid 139957180020480] ssl_engine_kernel.c(354): [client 172.19.28.1:38502] AH02034: Initial (No.1) HTTPS request received for child 7 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.052897 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.052907 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.055742 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.055769 2016] [authz_core:debug] [pid 22884:tid 139957180020480] mod_authz_core.c(809): [client 172.19.28.1:38502] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.055806 2016] [proxy:debug] [pid 22884:tid 139957180020480] mod_proxy.c(1160): [client 172.19.28.1:38502] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.055817 2016] [proxy_fcgi:debug] [pid 22884:tid 139957180020480] mod_proxy_fcgi.c(879): [client 172.19.28.1:38502] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.055824 2016] [proxy_fcgi:debug] [pid 22884:tid 139957180020480] mod_proxy_fcgi.c(882): [client 172.19.28.1:38502] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225
[Tue Feb 02 16:33:24.055833 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.055840 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2213): [client 172.19.28.1:38502] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 to localhost:5000
[Tue Feb 02 16:33:24.055848 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2422): [client 172.19.28.1:38502] AH00947: connected /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 to localhost:5000
[Tue Feb 02 16:33:24.060704 2016] [proxy:debug] [pid 22884:tid 139957180020480] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:24.060839 2016] [ssl:info] [pid 22884:tid 139957180020480] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "HEAD /v2/my-fancy-image/blobs/sha256:4f4cc6adaff9207db34041e88418f2de69408d18ff2a4a5019e91c4e31c67225 HTTP/1.1" 404 6071 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.061504 2016] [ssl:debug] [pid 22884:tid 139957062522624] ssl_engine_io.c(1016): [client 172.19.28.1:38502] AH02001: Connection closed to child 21 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.064122 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:24.064160 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:24.064178 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:24.064190 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:24.064199 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:24.064221 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:24.064231 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:24.064385 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:24.064424 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.095162 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0xa1 -> subcache 1)
[Tue Feb 02 16:33:24.095186 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(728): AH00842: expiring 1 and reclaiming 0 removed socache entries
[Tue Feb 02 16:33:24.095192 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(747): AH00843: we now have 1 socache entries
[Tue Feb 02 16:33:24.095225 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(845): AH00847: insert happened at idx=2, data=(408:440)
[Tue Feb 02 16:33:24.095231 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=1/2, data_pos/data_used=204/408
[Tue Feb 02 16:33:24.095235 2016] [socache_shmcb:debug] [pid 22885:tid 139957104486144] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.095418 2016] [ssl:debug] [pid 22885:tid 139957104486144] ssl_engine_kernel.c(354): [client 172.19.28.1:38503] AH02034: Initial (No.1) HTTPS request received for child 80 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.095455 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.095464 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.098288 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.098300 2016] [authz_core:debug] [pid 22885:tid 139957104486144] mod_authz_core.c(809): [client 172.19.28.1:38503] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.098333 2016] [proxy:debug] [pid 22885:tid 139957104486144] mod_proxy.c(1160): [client 172.19.28.1:38503] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.098341 2016] [proxy_fcgi:debug] [pid 22885:tid 139957104486144] mod_proxy_fcgi.c(879): [client 172.19.28.1:38503] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/uploads/ proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.098347 2016] [proxy_fcgi:debug] [pid 22885:tid 139957104486144] mod_proxy_fcgi.c(882): [client 172.19.28.1:38503] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/uploads/
[Tue Feb 02 16:33:24.098359 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.098369 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2213): [client 172.19.28.1:38503] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/uploads/ to localhost:5000
[Tue Feb 02 16:33:24.098378 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2422): [client 172.19.28.1:38503] AH00947: connected /v2/my-fancy-image/blobs/uploads/ to localhost:5000
[Tue Feb 02 16:33:24.105509 2016] [proxy:debug] [pid 22885:tid 139957104486144] proxy_util.c(2175): AH00943: http: has released connection for (localhost)
[Tue Feb 02 16:33:24.105712 2016] [ssl:info] [pid 22885:tid 139957104486144] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "POST /v2/my-fancy-image/blobs/uploads/ HTTP/1.1" 202 6503 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.106605 2016] [ssl:debug] [pid 22885:tid 139957096093440] ssl_engine_io.c(1016): [client 172.19.28.1:38503] AH02001: Connection closed to child 81 with standard shutdown (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.146414 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Tue Feb 02 16:33:24.146459 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Tue Feb 02 16:33:24.146479 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xf6 -> subcache 2)
[Tue Feb 02 16:33:24.146489 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Tue Feb 02 16:33:24.146497 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Tue Feb 02 16:33:24.146543 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Tue Feb 02 16:33:24.146552 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Tue Feb 02 16:33:24.146667 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(389): AH01937: stapling_check_response: cached response expired
[Tue Feb 02 16:33:24.146697 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Tue Feb 02 16:33:24.179130 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x17 -> subcache 23)
[Tue Feb 02 16:33:24.179148 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(728): AH00842: expiring 1 and reclaiming 0 removed socache entries
[Tue Feb 02 16:33:24.179154 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(747): AH00843: we now have 2 socache entries
[Tue Feb 02 16:33:24.179159 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(845): AH00847: insert happened at idx=3, data=(612:644)
[Tue Feb 02 16:33:24.179163 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=1/3, data_pos/data_used=204/612
[Tue Feb 02 16:33:24.179168 2016] [socache_shmcb:debug] [pid 22884:tid 139957054129920] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Tue Feb 02 16:33:24.179772 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_engine_kernel.c(354): [client 172.19.28.1:38504] AH02034: Initial (No.1) HTTPS request received for child 22 (server docker.redacted-tld.com:443)
[Tue Feb 02 16:33:24.179815 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)
[Tue Feb 02 16:33:24.179824 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Tue Feb 02 16:33:24.182960 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of Require valid-user : granted
[Tue Feb 02 16:33:24.182977 2016] [authz_core:debug] [pid 22884:tid 139957054129920] mod_authz_core.c(809): [client 172.19.28.1:38504] AH01626: authorization result of <RequireAny>: granted
[Tue Feb 02 16:33:24.183027 2016] [proxy:debug] [pid 22884:tid 139957054129920] mod_proxy.c(1160): [client 172.19.28.1:38504] AH01143: Running scheme http handler (attempt 0)
[Tue Feb 02 16:33:24.183040 2016] [proxy_fcgi:debug] [pid 22884:tid 139957054129920] mod_proxy_fcgi.c(879): [client 172.19.28.1:38504] AH01076: url: http://localhost:5000/v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D proxyname: (null) proxyport: 0
[Tue Feb 02 16:33:24.183056 2016] [proxy_fcgi:debug] [pid 22884:tid 139957054129920] mod_proxy_fcgi.c(882): [client 172.19.28.1:38504] AH01077: declining URL http://localhost:5000/v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D
[Tue Feb 02 16:33:24.183070 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2160): AH00942: HTTP: has acquired connection for (localhost)
[Tue Feb 02 16:33:24.183097 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2213): [client 172.19.28.1:38504] AH00944: connecting http://localhost:5000/v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D to localhost:5000
[Tue Feb 02 16:33:24.183113 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2422): [client 172.19.28.1:38504] AH00947: connected /v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D to localhost:5000
[Tue Feb 02 16:33:24.198730 2016] [ssl:info] [pid 22884:tid 139957054129920] SSL Library Error: error:2707307F:OCSP routines:OCSP_check_validity:status too old
[Tue Feb 02 16:33:24.198774 2016] [proxy_http:error] [pid 22884:tid 139957054129920] (20014)Internal error: [client 172.19.28.1:38504] AH02608: read request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:33:24.198788 2016] [proxy_http:error] [pid 22884:tid 139957054129920] [client 172.19.28.1:38504] AH01097: pass request body failed to 127.0.0.1:5000 (localhost) from 172.19.28.1 ()
[Tue Feb 02 16:33:24.198795 2016] [proxy:debug] [pid 22884:tid 139957054129920] proxy_util.c(2175): AH00943: HTTP: has released connection for (localhost)
[Tue Feb 02 16:33:24.198886 2016] [headers:debug] [pid 22884:tid 139957054129920] mod_headers.c(899): AH01503: headers: ap_headers_error_filter()

==> docker.redacted-tld.com-ssl.access.log <==
myownhostname - authenticateduser [02/Feb/2016:16:33:24 +0000] "PATCH /v2/my-fancy-image/blobs/uploads/8c884b6f-4e2e-40a9-a64a-7fca36ff953b?_state=eNFlIu3fQpEzlR2EXQWWLNKdWeKuNlIpd3hKwPw971B7Ik5hbWUiOiJzZWxlbml1bS1oZWFkbGVzcy1waGFudG9tanMyIiwiVVVJRCI6IjhjODg0YjZmLTRlMmUtNDBhOS1hNjRhLTdmY2EzNmZmOTUzYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wMi0wMlQxNjozMzoyNC4xMDE0NjE4OTZaIn0%3D HTTP/1.1" 400 6252 "-" "docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.16.0-4-amd64 os/linux arch/amd64"

==> docker.redacted-tld.com-ssl.error.log <==
[Tue Feb 02 16:33:24.199152 2016] [ssl:debug] [pid 22884:tid 139957054129920] ssl_engine_io.c(1016): [client 172.19.28.1:38504] AH02001: Connection closed to child 22 with standard shutdown (server docker.redacted-tld.com:443)


I really don't know what to make of this

Comment 1 Andrey L 2019-04-23 14:37:13 UTC

I've got the same issue yesterday.
Did you find any fix for this issue? 

Thanks in advance.