Bug 1455319
| Summary: | docker crashes due to SIGSEGV signal in dm_task_run() | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Chris Evich <cevich> |
| Component: | docker | Assignee: | Antonio Murdaca <amurdaca> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.4 | CC: | amurdaca, dornelas, hannsj_uhl, jamills, lsm5, nhorman, santiago, vgoyal |
| Target Milestone: | rc | Keywords: | Extras |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-21 22:14:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1186913 | ||
|
Description
Chris Evich
2017-05-24 18:55:17 UTC
This problem was reproduced during the docker_cli/negativeusage/if1 sub-subtest when it executed: /usr/bin/docker run --detach registry.access.redhat.com/rhel7/rhel:latest true Exit status: 125 Duration: 0.813946962357 stderr: /usr/bin/docker-current: An error occurred trying to connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create: EOF. See '/usr/bin/docker-current run --help'. Following that, all subsequent docker commands throw one of those 'Is the docker daemon running?' type errors. Antonio commented: From looking at docker version and messages in logs this is a devmapper issue (see below for thoughts): [...truncated...] 290 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: fatal error: unexpected signal during runtime execution 291 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: [signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x7fd7d1349090] 292 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: runtime stack: 293 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: runtime.throw(0x186956b, 0x2a) 294 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: /usr/lib/golang/src/runtime/panic.go:596 +0x95 fp=0x7fd7c57f9740 sp=0x7fd7c57f9720 295 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: runtime.sigpanic() 296 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: /usr/lib/golang/src/runtime/signal_unix.go:274 +0x2db fp=0x7fd7c57f9790 sp=0x7fd7c57f9740 297 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: goroutine 8425 [syscall, locked to thread]: 298 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: runtime.cgocall(0x144aeb0, 0xc43caf8a98, 0x18668bf) 299 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: /usr/lib/golang/src/runtime/cgocall.go:131 +0xe2 fp=0xc43caf8a58 sp=0xc43caf8a18 300 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: github.com/docker/docker/pkg/devicemapper._C2func_dm_task_run(0x7fd7a0031950, 0x7fd700000000, 0x0, 0x0) 301 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: github.com/docker/docker/pkg/devicemapper/_obj/_cgo_gotypes.go:95 +0x68 fp=0xc43caf8a98 sp=0xc43caf8a58 302 May 24 06:43:23 rhel-7-4-adept-latest-brew-142-adf65e8362c2 dockerd-current: github.com/docker/docker/pkg/devicemapper.dmTaskRunFct.func1(0x7fd7a0031950, 0xc4209c80f8, 0xc43caf8b10, 0xe08d72) [...truncated...] The docker being tested is commit "1398f24" which means, since it's a devmapper issue, one of the following commits in docker-1.12.6 (and 1.13.1) broke this as those are the only changes we made to devmapper code paths: https://github.com/projectatomic/docker/commit/1398f249013601ab999d286910664d70fd1329a2 https://github.com/projectatomic/docker/commit/bf35c1e9528787b88d9001f682be6bff0c2a130c https://github.com/projectatomic/docker/commit/fe4151df3c2a025722b22cab6838f59afe3f37de You can notice the first commit is the one the docker build came from also: https://github.com/projectatomic/docker/commits/docker-1.12.6 So, those commits were a backport Vivek asked me to do, Vivek thoughts on this? should we revert those commits? Should we investigate it? Do you know what's happening? Following PR from Neil, might solve this issue. https://github.com/moby/moby/pull/33376 I'll backport Neil's fix once merged upstream Backported. Installed... ---> Package container-selinux.noarch 2:2.12-2.gite7096ce.el7 will be installed ---> Package docker.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-client.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-common.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-debuginfo.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-logrotate.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-lvm-plugin.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-novolume-plugin.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-rhel-push-plugin.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package docker-v1.10-migrator.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ---> Package oci-umount.x86_64 2:1.12.6-29.git97ba2c0.el7 will be installed ...running Docker Autotest in a loop again. Same system. So good so far, no failures after 3-4 runs. I'll let it run overnight to be sure, but my guts tell me this is fixed. after 20+ runs and no crashes, I'm calling this good'nuf (previously we saw at least one crash about every 3-5 runs). |