Bug 1410434

Summary:

Docker 1.12.5 and OpenShift 3.4.0.38 : Frequent unexpected EOF during push causing build failures

Product:

Red Hat Enterprise Linux 7

Reporter:

Mike Fiedler <mifiedle>

Component:

docker

Assignee:

Lokesh Mandvekar <lsm5>

Status:

CLOSED ERRATA

QA Contact:

atomic-bugs <atomic-bugs>

Severity:

urgent

Docs Contact:

Priority:

unspecified

Version:

7.3

CC:

amurdaca, aos-bugs, ddarrah, jeder, lsm5, lsu, mifiedle, miminar, vlaad

Target Milestone:

Keywords:

Extras

Target Release:

7.3

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-01-17 20:45:56 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Sample build failure at loglevel=6	none
New build log with timestamps	none
Registry log	none

Description Mike Fiedler 2017-01-05 13:41:33 UTC

Created attachment 1237646 [details]
Sample build failure at loglevel=6

Description of problem:

With 3.4.0.38 using an S3 registry, seeing frequent build failures during the push phase with :

Pushed
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount
Registry server Password: <<non-empty>>
F0105 13:33:02.764460       1 helpers.go:115] error: build error: Failed to push image: unexpected EOF


Version-Release number of selected component (if applicable): 3.4.0.38


How reproducible: Frequent during concurrent builds


Steps to Reproduce:
1.  AWS install:  1 master, 1 etcd, 1 registry/router, 2 application nodes
2.  S3 configured as back-end registry storage
3.  Initial build of cakephp, django, dancer, rails, eap64, nodejs and tomcat8 quickstart apps successful.   All layers pushed to registry.
4.  Start running concurrent builds of 4 random applications at a time.

Actual results:

Many builds fail with "unexpected EOF" during the image push


Expected results:

Successful builds.  I last ran this test on 3.4.0.36 before the holidays and did not see this.

Additional info: build at loglevel 6 attached

Comment 1 Mike Fiedler 2017-01-05 13:54:18 UTC

Docker:

docker.x86_64                                            2:1.12.5-7.el7
Docker version 1.12.5, build 9932df9

Comment 2 Mike Fiedler 2017-01-05 13:55:46 UTC

Created attachment 1237686 [details]
New build log with timestamps

Comment 3 Mike Fiedler 2017-01-05 13:56:10 UTC

Created attachment 1237687 [details]
Registry log

Comment 4 Mike Fiedler 2017-01-05 13:59:10 UTC

There is also a docker panic in the node log

http: panic serving @: could not find ':' in digest: 
goroutine 346959 [running]:
net/http.(*conn).serve.func1(0xc4238f3480)
       /usr/lib/golang/src/net/http/server.go:1491 +0x12a
panic(0x164c4c0, 0xc42134e340)
       /usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/docker/distribution/digest.Digest.sepIndex(0x0, 0x0, 0xc4218984b0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/digest/digest.go:135 +0xec
github.com/docker/distribution/digest.Digest.Hex(0x0, 0x0, 0x5, 0xc4220ee320)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/digest/digest.go:124 +0x35
github.com/docker/distribution/manifest/schema1.(*configManifestBuilder).Build(0xc4231b21c0, 0x7f25dde518e0, 0xc421ff7680, 0x7f25dde518e0, 0xc421ff7680, 0x70, 0x1705360)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/docker/distribution/manifest/schema1/config_builder.go:136 +0x2ac
github.com/docker/docker/distribution.manifestFromBuilder(0x2225b80, 0xc421ff7680, 0x221dd00, 0xc4231b21c0, 0xc421b7a600, 0x5, 0x8, 0x0, 0x0, 0x0, ...)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:261 +0x120
github.com/docker/docker/distribution.(*v2Pusher).pushV2Tag(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0x2229140, 0xc4214d7540, 0xc4220efa40, 0x47, 0x0, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:216 +0x199a
github.com/docker/docker/distribution.(*v2Pusher).pushV2Repository(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0x0, 0xc4222e2a80)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:102 +0x5f4
github.com/docker/docker/distribution.(*v2Pusher).Push(0xc4222e2c80, 0x2225b80, 0xc421ff7680, 0xc4222e2a80, 0x2)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push_v2.go:83 +0x33b
github.com/docker/docker/distribution.Push(0x2225b80, 0xc421ff7680, 0x7f25dded4540, 0xc4214d7540, 0xc42319d680, 0xc421eb6c00, 0xc421eb6c60)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/distribution/push.go:166 +0x616
github.com/docker/docker/daemon.(*Daemon).PushImage(0xc420478ea0, 0x2225b80, 0xc421ff7680, 0xc422fee4ed, 0x3a, 0xc422fee531, 0x6, 0xc42303a8a0, 0xc42303aa20, 0x220efc0, ...)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/daemon/image_push.go:55 +0x363
github.com/docker/docker/api/server/router/image.(*imageRouter).postImagesPush(0xc420237380, 0x2225b80, 0xc421ff7540, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x0, 0x
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/image/image_routes.go:148 +0x55b
github.com/docker/docker/api/server/router/image.(*imageRouter).(github.com/docker/docker/api/server/router/image.postImagesPush)-fm(0x2225b80, 0xc421ff7540, 0x7f25dde72ba0, 0xc421ff7
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/image/image.go:44 +0x69
github.com/docker/docker/api/server/router.cancellableHandler.func1(0x2225c40, 0xc42303a870, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x0, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router/local.go:84 +0x104
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x2225c40, 0xc42303a840, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x16e8220, 0xffff
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/version.go:56 +0x73c
github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x7f25dde72ba0, 0xc421ff7480, 0xc421898960, 0xc423201440, 0x153e8e0, 0xc4
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/user_agent.go:45 +0x272
github.com/docker/docker/pkg/authorization.Middleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x2222300, 0xc423793380, 0xc421898960, 0xc423201440, 0xc423201440, 0x0)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/pkg/authorization/middleware.go:49 +0x3e2
github.com/docker/docker/api/server/middleware.AuditMiddleware.WrapHandler.func1(0x2225bc0, 0xc42000e648, 0x2222300, 0xc423793380, 0xc421898960, 0xc423201440, 0xc421284bb0, 0x8)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/middleware/audit_linux.go:27 +0x9d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/server.go:146 +0xc5
net/http.HandlerFunc.ServeHTTP(0xc420a36a80, 0x2222300, 0xc423793380, 0xc421898960)
       /usr/lib/golang/src/net/http/server.go:1726 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc4201e8a50, 0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/vendor/src/github.com/gorilla/mux/mux.go:98 +0x255
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc4209aa840, 0x2222300, 0xc423793380, 0xc421898960)
       /builddir/build/BUILD/docker-60099055d5aea23c540efd8e29a6aef31f55e75d/_build/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x70
net/http.serverHandler.ServeHTTP(0xc420297300, 0x2222300, 0xc423793380, 0xc421898960)
       /usr/lib/golang/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc4238f3480, 0x2224080, 0xc422f822c0)
       /usr/lib/golang/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
       /usr/lib/golang/src/net/http/server.go:2293 +0x44d

Comment 5 Michal Minar 2017-01-05 14:02:06 UTC

This is an issue https://github.com/docker/docker/issues/28596 fixed upstream https://github.com/docker/docker/pull/28966. The patch needs to be back-ported.

Comment 6 Mike Fiedler 2017-01-05 14:03:19 UTC

Same as :   https://github.com/docker/docker/issues/28596

Comment 7 Mike Fiedler 2017-01-05 14:32:16 UTC

Seeing the same issue on 1.12.5-8 which is the latest on our extras mirror.

Comment 8 Antonio Murdaca 2017-01-05 14:35:37 UTC

I'll backport that fix to docker-1.12.5. I'll re-assign to Lokesh to rebuild for RHEL.

Comment 9 Antonio Murdaca 2017-01-05 15:38:12 UTC

Backported https://github.com/projectatomic/docker/commit/047e51b797564227b0bf26f3aa448f563bea5c71

Comment 13 Mike Fiedler 2017-01-10 12:35:25 UTC

This issue is resolved in 1.12.5-9 - the problem no longer occurs and all OpenShift builds are successful.

Comment 14 David Darrah/Red Hat QE 2017-01-10 18:56:25 UTC

Moving to verified based on comment 13.

Comment 16 errata-xmlrpc 2017-01-17 20:45:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0116.html