RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1410434 - Docker 1.12.5 and OpenShift 3.4.0.38 : Frequent unexpected EOF during push causing build failures
Summary: Docker 1.12.5 and OpenShift 3.4.0.38 : Frequent unexpected EOF during push c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: rc
: 7.3
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-05 13:41 UTC by Mike Fiedler
Modified: 2019-03-06 02:16 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-17 20:45:56 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Sample build failure at loglevel=6 (189.91 KB, text/plain)
2017-01-05 13:41 UTC, Mike Fiedler 		no flags Details
New build log with timestamps (211.89 KB, text/plain)
2017-01-05 13:55 UTC, Mike Fiedler 		no flags Details
Registry log (2.85 MB, text/plain)
2017-01-05 13:56 UTC, Mike Fiedler 		no flags Details
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0116 0 normal SHIPPED_LIVE Moderate: docker security, bug fix, and enhancement update 2017-01-18 01:39:43 UTC

Description Mike Fiedler 2017-01-05 13:41:33 UTC 
Created attachment 1237646 [details]
Sample build failure at loglevel=6

Description of problem:

With 3.4.0.38 using an S3 registry, seeing frequent build failures during the push phase with :

Pushed
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount
Registry server Password: <<non-empty>>
F0105 13:33:02.764460       1 helpers.go:115] error: build error: Failed to push image: unexpected EOF


Version-Release number of selected component (if applicable): 3.4.0.38


How reproducible: Frequent during concurrent builds


Steps to Reproduce:
1.  AWS install:  1 master, 1 etcd, 1 registry/router, 2 application nodes
2.  S3 configured as back-end registry storage
3.  Initial build of cakephp, django, dancer, rails, eap64, nodejs and tomcat8 quickstart apps successful.   All layers pushed to registry.
4.  Start running concurrent builds of 4 random applications at a time.

Actual results:

Many builds fail with "unexpected EOF" during the image push


Expected results:

Successful builds.  I last ran this test on 3.4.0.36 before the holidays and did not see this.

Additional info: build at loglevel 6 attached
Comment 1 Mike Fiedler 2017-01-05 13:54:18 UTC 
Docker:

docker.x86_64                                            2:1.12.5-7.el7
Docker version 1.12.5, build 9932df9
Comment 2 Mike Fiedler 2017-01-05 13:55:46 UTC 
Created attachment 1237686 [details]
New build log with timestamps
Comment 3 Mike Fiedler 2017-01-05 13:56:10 UTC 
Created attachment 1237687 [details]
Registry log
Comment 4 Mike Fiedler 2017-01-05 13:59:10 UTC 
There is also a docker panic in the node log

http: panic serving @: could not find ':' in digest: 
goroutine 346959 [running]:
net/http.(*conn).serve.func1(0xc4238f3480)
       /usr/lib/golang/src/net/http/server.go:1491 +0x12a
panic(0x164c4c0, 0xc42134e340)
       /usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/docker/distribution/digest.Digest.sepIndex(0x0, 0x0, 0xc4218984b0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/digest/digest.go:135 +0xec
github.com/docker/distribution/digest.Digest.Hex(0x0, 0x0, 0x5, 0xc4220ee320)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/digest/digest.go:124 +0x35
github.com/docker/distribution/manifest/schema1.(*configManifestBuilder).Build(0xc4231b21c0, 0x7f25dde518e0, 0xc421ff7680, 0x7f25dde518e0, 0xc421ff7680, 0x70, 0x1705360)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/manifest/schema1/config_builder.go:136 +0x2ac
github.com/docker/docker/distribution.manifestFromBuilder(0x2225b80, 0xc421ff7680, 0x221dd00, 0xc4231b21c0, 0xc421b7a600, 0x5, 0x8, 0x0, 0x0, 0x0, ...)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:261 +0x120
github.com/docker/docker/distribution.(*v2Pusher).pushV2Tag(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0x2229140, 0xc4214d7540, 0xc4220efa40, 0x47, 0x0, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:216 +0x199a
github.com/docker/docker/distribution.(*v2Pusher).pushV2Repository(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0x0, 0xc4222e2a80)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:102 +0x5f4
github.com/docker/docker/distribution.(*v2Pusher).Push(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0xc4222e2a80, 0x2)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:83 +0x33b
github.com/docker/docker/distribution.Push(0x2225b80, 0xc421ff7680, 0x7f25dded4540, 0xc4214d7540, 0xc42319d680, 0xc421eb6c00, 0xc421eb6c60)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push.go:166 +0x616
github.com/docker/docker/daemon.(*Daemon).PushImage(0xc420478ea0, 0x2225b80, 0xc421ff7680, 0xc422fee4ed, 0x3a, 0xc422fee531, 0x6, 0xc42303a8a0, 0xc42303aa20, 0x220efc0, ...)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/daemon/image_push.go:55 +0x363
github.com/docker/docker/api/server/router/image.(*imageRouter).postImagesPush(0xc420237380, 0x2225b80, 0xc421ff7540, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x0, 0x
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/image/image_routes.go:148 +0x55b
github.com/docker/docker/api/server/router/image.(*imageRouter).(github.com/docker/docker/api/server/router/image.postImagesPush)-fm(0x2225b80, 0xc421ff7540, 0x7f25dde72ba0, 0xc421ff7
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/image/image.go:44 +0x69
github.com/docker/docker/api/server/router.cancellableHandler.func1(0x2225c40, 0xc42303a870, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x0, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/local.go:84 +0x104
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x2225c40, 0xc42303a840, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x16e8220, 0xffff
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/version.go:56 +0x73c
github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x153e8e0, 0xc4
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/user_agent.go:45 +0x272
github.com/docker/docker/pkg/authorization.Middleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x2222300, 0xc423793380, 0xc421898960, 0xc423201440, 0xc423201440, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/pkg/authorization/middleware.go:49 +0x3e2
github.com/docker/docker/api/server/middleware.AuditMiddleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x2222300, 0xc423793380, 0xc421898960, 0xc423201440, 0xc421284bb0, 0x8)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/audit_linux.go:27 +0x9d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/server.go:146 +0xc5
net/http.HandlerFunc.ServeHTTP(0xc420a36a80, 0x2222300, 0xc423793380, 0xc421898960)
       /usr/lib/golang/src/net/http/server.go:1726 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc4201e8a50, 0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/gorilla/mux/mux.go:98 +0x255
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc4209aa840, 0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x70
net/http.serverHandler.ServeHTTP(0xc420297300, 0x2222300, 0xc423793380, 0xc421898960)
       /usr/lib/golang/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc4238f3480, 0x2224080, 0xc422f822c0)
       /usr/lib/golang/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
       /usr/lib/golang/src/net/http/server.go:2293 +0x44d
Comment 5 Michal Minar 2017-01-05 14:02:06 UTC 
This is an issue https://github.com/docker/docker/issues/28596 fixed upstream https://github.com/docker/docker/pull/28966. The patch needs to be back-ported.
Comment 6 Mike Fiedler 2017-01-05 14:03:19 UTC 
Same as :   https://github.com/docker/docker/issues/28596
Comment 7 Mike Fiedler 2017-01-05 14:32:16 UTC 
Seeing the same issue on 1.12.5-8 which is the latest on our extras mirror.
Comment 8 Antonio Murdaca 2017-01-05 14:35:37 UTC 
I'll backport that fix to docker-1.12.5. I'll re-assign to Lokesh to rebuild for RHEL.
Comment 9 Antonio Murdaca 2017-01-05 15:38:12 UTC 
Backported https://github.com/projectatomic/docker/commit/047e51b797564227b0bf26f3aa448f563bea5c71
Comment 13 Mike Fiedler 2017-01-10 12:35:25 UTC 
This issue is resolved in 1.12.5-9 - the problem no longer occurs and all OpenShift builds are successful.
Comment 14 David Darrah/Red Hat QE 2017-01-10 18:56:25 UTC 
Moving to verified based on comment 13.
Comment 16 errata-xmlrpc 2017-01-17 20:45:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0116.html
Note You need to log in before you can comment on or make changes to this bug.