Bug 1410434 - Docker 1.12.5 and OpenShift 3.4.0.38 : Frequent unexpected EOF during push causing build failures
Summary: Docker 1.12.5 and OpenShift 3.4.0.38 : Frequent unexpected EOF during push c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: rc
: 7.3
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-05 13:41 UTC by Mike Fiedler
Modified: 2019-03-06 02:16 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-17 20:45:56 UTC
Target Upstream Version:


Attachments (Terms of Use)
Sample build failure at loglevel=6 (189.91 KB, text/plain)
2017-01-05 13:41 UTC, Mike Fiedler
no flags Details
New build log with timestamps (211.89 KB, text/plain)
2017-01-05 13:55 UTC, Mike Fiedler
no flags Details
Registry log (2.85 MB, text/plain)
2017-01-05 13:56 UTC, Mike Fiedler
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0116 0 normal SHIPPED_LIVE Moderate: docker security, bug fix, and enhancement update 2017-01-18 01:39:43 UTC

Description Mike Fiedler 2017-01-05 13:41:33 UTC
Created attachment 1237646 [details]
Sample build failure at loglevel=6

Description of problem:

With 3.4.0.38 using an S3 registry, seeing frequent build failures during the push phase with :

Pushed
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
F0105 13:33:02.764460       1 helpers.go:115] error: build error: Failed to push image: unexpected EOF


Version-Release number of selected component (if applicable): 3.4.0.38


How reproducible: Frequent during concurrent builds


Steps to Reproduce:
1.  AWS install:  1 master, 1 etcd, 1 registry/router, 2 application nodes
2.  S3 configured as back-end registry storage
3.  Initial build of cakephp, django, dancer, rails, eap64, nodejs and tomcat8 quickstart apps successful.   All layers pushed to registry.
4.  Start running concurrent builds of 4 random applications at a time.

Actual results:

Many builds fail with "unexpected EOF" during the image push


Expected results:

Successful builds.  I last ran this test on 3.4.0.36 before the holidays and did not see this.

Additional info: build at loglevel 6 attached

Comment 1 Mike Fiedler 2017-01-05 13:54:18 UTC
Docker:

docker.x86_64                                            2:1.12.5-7.el7
Docker version 1.12.5, build 9932df9

Comment 2 Mike Fiedler 2017-01-05 13:55:46 UTC
Created attachment 1237686 [details]
New build log with timestamps

Comment 3 Mike Fiedler 2017-01-05 13:56:10 UTC
Created attachment 1237687 [details]
Registry log

Comment 4 Mike Fiedler 2017-01-05 13:59:10 UTC
There is also a docker panic in the node log

http: panic serving @: could not find ':' in digest: 
goroutine 346959 [running]:
net/http.(*conn).serve.func1(0xc4238f3480)
       /usr/lib/golang/src/net/http/server.go:1491 +0x12a
panic(0x164c4c0, 0xc42134e340)
       /usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/docker/distribution/digest.Digest.sepIndex(0x0, 0x0, 0xc4218984b0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/digest/digest.go:135 +0xec
github.com/docker/distribution/digest.Digest.Hex(0x0, 0x0, 0x5, 0xc4220ee320)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/digest/digest.go:124 +0x35
github.com/docker/distribution/manifest/schema1.(*configManifestBuilder).Build(0xc4231b21c0, 0x7f25dde518e0, 0xc421ff7680, 0x7f25dde518e0, 0xc421ff7680, 0x70, 0x1705360)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/manifest/schema1/config_builder.go:136 +0x2ac
github.com/docker/docker/distribution.manifestFromBuilder(0x2225b80, 0xc421ff7680, 0x221dd00, 0xc4231b21c0, 0xc421b7a600, 0x5, 0x8, 0x0, 0x0, 0x0, ...)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:261 +0x120
github.com/docker/docker/distribution.(*v2Pusher).pushV2Tag(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0x2229140, 0xc4214d7540, 0xc4220efa40, 0x47, 0x0, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:216 +0x199a
github.com/docker/docker/distribution.(*v2Pusher).pushV2Repository(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0x0, 0xc4222e2a80)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:102 +0x5f4
github.com/docker/docker/distribution.(*v2Pusher).Push(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0xc4222e2a80, 0x2)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:83 +0x33b
github.com/docker/docker/distribution.Push(0x2225b80, 0xc421ff7680, 0x7f25dded4540, 0xc4214d7540, 0xc42319d680, 0xc421eb6c00, 0xc421eb6c60)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push.go:166 +0x616
github.com/docker/docker/daemon.(*Daemon).PushImage(0xc420478ea0, 0x2225b80, 0xc421ff7680, 0xc422fee4ed, 0x3a, 0xc422fee531, 0x6, 0xc42303a8a0, 0xc42303aa20, 0x220efc0, ...)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/daemon/image_push.go:55 +0x363
github.com/docker/docker/api/server/router/image.(*imageRouter).postImagesPush(0xc420237380, 0x2225b80, 0xc421ff7540, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x0, 0x
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/image/image_routes.go:148 +0x55b
github.com/docker/docker/api/server/router/image.(*imageRouter).(github.com/docker/docker/api/server/router/image.postImagesPush)-fm(0x2225b80, 0xc421ff7540, 0x7f25dde72ba0, 0xc421ff7
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/image/image.go:44 +0x69
github.com/docker/docker/api/server/router.cancellableHandler.func1(0x2225c40, 0xc42303a870, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x0, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/local.go:84 +0x104
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x2225c40, 0xc42303a840, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x16e8220, 0xffff
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/version.go:56 +0x73c
github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x153e8e0, 0xc4
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/user_agent.go:45 +0x272
github.com/docker/docker/pkg/authorization.Middleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x2222300, 0xc423793380, 0xc421898960, 0xc423201440, 0xc423201440, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/pkg/authorization/middleware.go:49 +0x3e2
github.com/docker/docker/api/server/middleware.AuditMiddleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x2222300, 0xc423793380, 0xc421898960, 0xc423201440, 0xc421284bb0, 0x8)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/audit_linux.go:27 +0x9d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/server.go:146 +0xc5
net/http.HandlerFunc.ServeHTTP(0xc420a36a80, 0x2222300, 0xc423793380, 0xc421898960)
       /usr/lib/golang/src/net/http/server.go:1726 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc4201e8a50, 0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/gorilla/mux/mux.go:98 +0x255
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc4209aa840, 0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x70
net/http.serverHandler.ServeHTTP(0xc420297300, 0x2222300, 0xc423793380, 0xc421898960)
       /usr/lib/golang/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc4238f3480, 0x2224080, 0xc422f822c0)
       /usr/lib/golang/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
       /usr/lib/golang/src/net/http/server.go:2293 +0x44d

Comment 5 Michal Minar 2017-01-05 14:02:06 UTC
This is an issue https://github.com/docker/docker/issues/28596 fixed upstream https://github.com/docker/docker/pull/28966. The patch needs to be back-ported.

Comment 6 Mike Fiedler 2017-01-05 14:03:19 UTC
Same as :   https://github.com/docker/docker/issues/28596

Comment 7 Mike Fiedler 2017-01-05 14:32:16 UTC
Seeing the same issue on 1.12.5-8 which is the latest on our extras mirror.

Comment 8 Antonio Murdaca 2017-01-05 14:35:37 UTC
I'll backport that fix to docker-1.12.5. I'll re-assign to Lokesh to rebuild for RHEL.

Comment 13 Mike Fiedler 2017-01-10 12:35:25 UTC
This issue is resolved in 1.12.5-9 - the problem no longer occurs and all OpenShift builds are successful.

Comment 14 David Darrah/Red Hat QE 2017-01-10 18:56:25 UTC
Moving to verified based on comment 13.

Comment 16 errata-xmlrpc 2017-01-17 20:45:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0116.html


Note You need to log in before you can comment on or make changes to this bug.