Bug 1359200

Summary:	Race condition writing to mcs map
Product:	Red Hat Enterprise Linux 7	Reporter:	Andy Goldstein <agoldste>
Component:	docker	Assignee:	Antonio Murdaca <amurdaca>
Status:	CLOSED ERRATA	QA Contact:	atomic-bugs <atomic-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.2	CC:	dwalsh, lsm5, lsu, mifiedle
Target Milestone:	rc	Keywords:	Extras, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-03 20:57:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andy Goldstein 2016-07-22 13:47:14 UTC

Description of problem: The docker daemon can crash due to a race condition writing to the mcs map


Version-Release number of selected component (if applicable): docker-1.10.3-44.el7.x86_64


How reproducible: sometimes


Steps to Reproduce:
1. Create and start multiple containers at the same time
2.
3.

Actual results: sometimes the docker daemon crashes


Expected results: should not crash


Additional info:

I've seen this happen pretty frequently just by building openshift origin and running 'oc cluster up'.

Backtrace:

goroutine 18115 [running]:
runtime.throw(0x17f6820, 0x15)
/usr/lib/golang/src/runtime/panic.go:547 +0x90 fp=0xc821588ef8 sp=0xc821588ee0
runtime.mapassign1(0x120b100, 0xc8212bc240, 0xc821588fe8, 0xc821588fc0)
/usr/lib/golang/src/runtime/hashmap.go:540 +0x54b fp=0xc821588fa0 sp=0xc821588ef8
github.com/docker/docker/layer.(*layerStore).Get(0xc8205e4ac0, 0xc82339c1e0, 0x47, 0x0, 0x0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/layer/layer_store.go:342 +0x141 fp=0xc821589000 sp=0xc821588fa0
github.com/docker/docker/daemon.(*Daemon).LookupImage(0xc820001b00, 0xc82152c1cc, 0x1c, 0x0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/daemon/daemon.go:1104 +0x525 fp=0xc821589298 sp=0xc821589000
github.com/docker/docker/api/server/router/local.(*router).getImagesByName(0xc820a2ce00, 0x7fd2d22d6c48, 0xc82298bbf0, 0x7fd2d22d6c10, 0xc82104df40, 0xc821595420, 0xc82298b7d0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/router/local/image.go:286 +0xa6 fp=0xc821589318 sp=0xc821589298
github.com/docker/docker/api/server/router/local.(*router).(github.com/docker/docker/api/server/router/local.getImagesByName)-fm(0x7fd2d22d6c48, 0xc82298bbf0, 0x7fd2d22d6c10, 0xc82104df40,
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/router/local/local.go:97 +0x74 fp=0xc821589368 sp=0xc821589318
github.com/docker/docker/api/server.(*Server).auditMiddleware.func1(0x7fd2d22d6c48, 0xc82298bbf0, 0x7fd2d22d6c10, 0xc82104df40, 0xc821595420, 0xc82298b7d0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/middleware.go:163 +0xa2 fp=0xc8215893b8 sp=0xc821589368
github.com/docker/docker/api/server.versionMiddleware.func1(0x7fd2d22d6c48, 0xc82298bbf0, 0x7fd2d22d6c10, 0xc82104df40, 0xc821595420, 0xc82298b7d0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/middleware.go:155 +0x83a fp=0xc821589538 sp=0xc8215893b8
github.com/docker/docker/api/server.(*Server).corsMiddleware.func1(0x7fd2d22be538, 0xc820013340, 0x7fd2d22d6c10, 0xc82104df40, 0xc821595420, 0xc82298b7d0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/middleware.go:134 +0xfa fp=0xc821589598 sp=0xc821589538
github.com/docker/docker/api/server.(*Server).userAgentMiddleware.func1(0x7fd2d22be538, 0xc820013340, 0x7fd2d22d6c10, 0xc82104df40, 0xc821595420, 0xc82298b7d0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/middleware.go:117 +0x4af fp=0xc8215896a8 sp=0xc821589598
github.com/docker/docker/api/server.(*Server).authorizationMiddleware.func1(0x7fd2d22be538, 0xc820013340, 0x7fd2d2231400, 0xc822575e10, 0xc821595420, 0xc82298b7d0, 0x0, 0x0)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/middleware.go:86 +0x5ba fp=0xc8215898d0 sp=0xc8215896a8
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7fd2d2231400, 0xc822575e10, 0xc821595420)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/server.go:175 +0x2ad fp=0xc8215899f8 sp=0xc8215898d0
net/http.HandlerFunc.ServeHTTP(0xc820b1c940, 0x7fd2d2231400, 0xc822575e10, 0xc821595420)
/usr/lib/golang/src/net/http/server.go:1618 +0x3a fp=0xc821589a18 sp=0xc8215899f8
github.com/gorilla/mux.(*Router).ServeHTTP(0xc820928050, 0x7fd2d2231400, 0xc822575e10, 0xc821595420)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e fp=0xc821589b30 sp=0xc821589a18
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc820bab5c0, 0x7fd2d2231400, 0xc822575e10, 0xc821595420)
/builddir/build/BUILD/docker-a46c31af70ca8d15521e312ad9ef7085cfe2fd3f/_build/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72 fp=0xc821589b60 sp=0xc821589b30
net/http.serverHandler.ServeHTTP(0xc82007ce80, 0x7fd2d2231400, 0xc822575e10, 0xc821595420)
/usr/lib/golang/src/net/http/server.go:2081 +0x19e fp=0xc821589bc0 sp=0xc821589b60
net/http.(*conn).serve(0xc821658280)
/usr/lib/golang/src/net/http/server.go:1472 +0xf2e fp=0xc821589f88 sp=0xc821589bc0
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc821589f90 sp=0xc821589f88
created by net/http.(*Server).Serve
/usr/lib/golang/src/net/http/server.go:2137 +0x44e

Fixed by https://github.com/opencontainers/runc/pull/735

Comment 3 Luwen Su 2016-07-25 08:59:53 UTC

I just could verify that this patch has been in docker-1.10.3-46.el7.8.x86_64, move to verified.

Comment 5 errata-xmlrpc 2016-08-03 20:57:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1446.html