|Summary:||docker-1.10.3-3.gitd93ee51.fc24 fails to run anything|
|Product:||[Fedora] Fedora||Reporter:||Dennis Gilmore <dennis>|
|Component:||docker||Assignee:||Lokesh Mandvekar <lsm5>|
|Status:||CLOSED EOL||QA Contact:||Fedora Extras Quality Assurance <extras-qa>|
|Version:||24||CC:||adimania, admiller, amurdaca, andrew, awilliam, dennis, dwalsh, fedora.243908, ichavero, jbrooks, jcajka, jchaloup, jpazdziora, jsefler, lbrabec, lionelve, lsm5, marianne, massi.ergosum, miminar, nalin, nehaljw.kkd1, pschindl, pwhalen, rhatlapa, robatino, sgallagh, vbatts, vgoyal|
|Target Milestone:||---||Keywords:||CommonBugs, Reopened|
|Fixed In Version:||docker-1.10.3-4.gitf8a9a2a.fc24||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2017-08-08 14:08:15 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
Description Dennis Gilmore 2016-03-31 15:40:21 UTC
Comment 1 Fedora Update System 2016-03-31 15:47:34 UTC
docker-1.10.3-4.gitf8a9a2a.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-976f49409c
Comment 2 Fedora Update System 2016-04-03 17:57:10 UTC
docker-1.10.3-4.gitf8a9a2a.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Comment 3 Paul Whalen 2016-04-20 14:51:42 UTC
Reopening, this is still happening with docker-1.10.3-4.gitf8a9a2a.fc24 on x86_64 and armhfp. Working on aarch64.
Comment 4 Antonio Murdaca 2016-04-20 14:59:35 UTC
could you provide the output of docker info? I'm about to rebuild docker for f24 (I'll link the bodhi update to test it out and see if it's still happening)
Comment 5 Antonio Murdaca 2016-04-20 15:14:08 UTC
Just built this new docker version, might want to try it out and leave comments/karmas https://bodhi.fedoraproject.org/updates/FEDORA-2016-f656582969
Comment 6 Paul Whalen 2016-04-20 15:48:20 UTC
no change with docker-1.10.3-5.gitef2fa35.fc24.x86_64 [root@localhost ~]# docker info Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 2 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: docker-253:3-6161116-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 501.5 MB Data Space Total: 107.4 GB Data Space Available: 106.9 GB Metadata Space Used: 843.8 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.122 (2016-04-09) Execution Driver: native-0.2 Logging Driver: journald Plugins: Volume: local Network: host bridge null Kernel Version: 4.5.0-302.fc24.x86_64 Operating System: Fedora 24 (Twenty Four) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 4 Total Memory: 3.745 GiB Name: localhost.localdomain ID: NKVP:ISGA:XV3P:2KSQ:TQDF:HAW7:3W34:P7QK:VJSU:UL6C:WFRR:3KR6 Registries: docker.io (secure)
Comment 7 Paul Whalen 2016-04-20 15:49:32 UTC
Created attachment 1149121 [details] journalctl output
Comment 8 Stephen Gallagher 2016-04-20 17:50:54 UTC
Dennis, what blocker criterion do you believe this is violating?
Comment 9 Dennis Gilmore 2016-04-21 17:55:22 UTC
while technically not virt https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Virtualization_requirements I think it falls into the same boat.
Comment 10 Stephen Gallagher 2016-04-21 19:14:13 UTC
Hmm, this is very odd. I'm able to run things just fine under docker-1.10.3-5.gitef2fa35.fc24.x86_64 on my personal workstation, but if I spin up a new Fedora Server VM, I can see the error you're reporting here. I'm not sure what the difference is between the two systems.
Comment 11 Jason Brooks 2016-04-22 00:20:13 UTC
I just tested this w/ success on my laptop w/ f24, and again in an f24 server VM w/ 1.10.3-5.gitef2fa35.fc24. I was running: docker run -t -i docker.io/fedora /bin/bash I see that one of the commenters was using loopback storage, which I don't use on my laptop. My VM didn't have loopback storage configured, either, and I didn't do anything special to avoid loopback -- the auto-partitioning set me up correctly. I installed my VM w/ a netinstall iso from http://mirror.math.princeton.edu/pub/fedora/linux//development/24/Server/x86_64/iso/, and gave it a 20GB disk, if that's relevant. [root@localhost ~]# docker info Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 1 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: fedora-docker--pool Pool Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 284.2 MB Data Space Total: 2.147 GB Data Space Available: 1.863 GB Metadata Space Used: 90.11 kB Metadata Space Total: 20.97 MB Metadata Space Available: 20.88 MB Udev Sync Supported: true Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.122 (2016-04-09) Execution Driver: native-0.2 Logging Driver: journald Plugins: Volume: local Network: null host bridge Kernel Version: 4.5.1-300.fc24.x86_64 Operating System: Fedora 24 (Server Edition) OSType: linux Architecture: x86_64 Number of Docker Hooks: 0 CPUs: 2 Total Memory: 1.954 GiB Name: localhost.localdomain ID: 3EPJ:NXD7:BLT6:DTF7:G3C7:7PG4:HN7B:INK4:LB5X:Y6CI:UM4S:ITGS Registries: docker.io (secure)
Comment 12 Paul Whalen 2016-04-22 17:57:41 UTC
Changing the root filesystem of the host to xfs, docker runs as expected on armhfp. Failure seems to occur when using ext4.
Comment 13 Paul Whalen 2016-04-22 18:30:52 UTC
filesystem appears to be a red herring, new install using the development repo and docker-1.10.3-5.gitef2fa35.fc24.armv7hl is working fine on armhfp with an ext4 root.
Comment 14 Paul Whalen 2016-04-22 18:57:37 UTC
This is reproducible by 1) install docker-1.10.3-4.gitf8a9a2a.fc24, attempt to start a docker image 2) Update to docker-1.10.3-5.gitef2fa35.fc24.x86_64, attempt to start a docker image 3) fixed by removing docker and installing docker-1.10.3-5.gitef2fa35.fc24.x86_64
Comment 15 Antonio Murdaca 2016-04-22 19:19:52 UTC
Weird, not sure what might have caused that.
Comment 16 Daniel Walsh 2016-04-23 10:59:27 UTC
Well if the latest works, we will not be fixing this update issue. We should just be shipping docker-1.10.3-5.gitef2fa35.fc24.x86_64
Comment 17 Adam Williamson 2016-04-23 13:59:10 UTC
dwalsh: but this was initially filed with -3, then reported with -4, now pwhalen says there's a reproducer when going from -4 to -5. Could it perhaps be the case that there's something in the docker scriptlets or something which is sabotaging updates, so it's just the case that whenever there's a docker update, it blows up? If so that seems like something we'd want to fix. I'm only speculating, though.
Comment 18 Biji 2016-04-25 10:31:32 UTC
it works by following step from Paul Whalen, and rm -rf /var/lib/docker/*
Comment 19 Stephen Gallagher 2016-04-25 13:59:31 UTC
It's not just upgrades; I reproduced this issue on a clean install of Fedora Server from the nightly build on April 21st with no updates.
Comment 20 Daniel Walsh 2016-04-25 14:05:39 UTC
Stephen, did you do it with the -5 version?
Comment 21 Stephen Gallagher 2016-04-25 15:27:09 UTC
OK, new information. I installed from http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/x86_64/iso/Fedora-Server-dvd-x86_64-Rawhide-20160425.n.0.iso this morning which has docker-1.10.3-4.gitf8a9a2a.fc24.x86_64 This configuration created a 15GiB main partition and left a few gigs available for docker-storage-setup to turn into a thin pool, which it did successfully. The `docker pull` and `docker run` commands noted in the description above both worked successfully. I updated to docker-1.10.3-5.gitef2fa35.fc24 and confirmed that it worked as well. So far, so good. I then downgraded to docker-1.10.3-3.gitd93ee51.fc24 and verified that the original bug reappears, which it did. (I think this may have been the version I was using in my initial testing before). I then upgraded back to docker-1.10.3-5.gitef2fa35.fc24 to see if the problem went away. *It did not*. I deleted the image, pulled it again and tried to start it again: it still failed with the original bug report. I stopped docker, ran `rm -rf /var/lib/docker` and tried to restart. It failed with an error about the thin pool having the wrong configuration, so I deleted the LVM thin pool and then restarted docker-storage-setup to recreate it. After all of this, the system was restored to a working state. (*) So the problem we have right now is that for any user who gets into the bad state, they have no simple recourse for getting back out of it. We should also test whether this is a problem when upgrading from Fedora 23. (Though I expect that this will skip over the problematic versions now that -4 is in the stable repo). (*) For people using the loopback device, I *think* that should just have been cleared by removing /var/lib/docker, so this part may be optional, but I haven't tested it.
Comment 22 Petr Schindler 2016-04-25 17:53:36 UTC
Discussed at 2016-04-25 blocker review meeting: . This bug was rejecte as Beta blocker: this is rejected for now as sgallagh's testing indicates the current stable package is not affected and there are no docker criteria in any case, but it may be re-evaluated if the bug affects -4 and we add docker criteria as proposed  http://meetbot.fedoraproject.org/fedora-blocker-review/2016-02-29/f24-blocker-review.2016-04-25-17.02.html
Comment 23 Daniel Walsh 2016-04-25 18:12:09 UTC
Stephen, I think you need to destroy your devmapper image as well as removing /var/lib/docker
Comment 24 Andrew Cathrow 2016-05-11 20:03:31 UTC
Seeing this with docker-1.10.3-9.git667d6d1.fc24 taking oci-register-machine-0.1.1 off koji resolves the issue. Dupe of 1320601?
Comment 25 Daniel Walsh 2016-06-03 12:30:14 UTC
Antonio doesn't this happen if the cgroup config in docker.service is not setup correctly?
Comment 26 Antonio Murdaca 2016-06-03 12:39:14 UTC
Dan, right, I believe we fixed that in latest 1.10.3 for Fedora.
Comment 27 Antonio Murdaca 2016-06-03 12:42:46 UTC
Can anyone see if the latest docker fixes this BZ?
Comment 28 Nehal J Wani 2016-07-04 06:25:22 UTC
docker-1.10.3-21.git19b5791.fc24.x86_64 fixes this for me after installing docker-selinux-1.10.3-21.git19b5791.fc24.x86_64 and restarting the docker daemon.
Comment 31 Daniel Walsh 2016-09-01 11:01:56 UTC
The problem here is docker-current is not labeled correctly. chcon -t docker_exec_t /usr/bin/docker-current systemctl restart docker Should fix it. Looks like we have the wrong version of docker-selinux, which does not label docker-currect correctly.
Comment 32 Lionel Orellana 2016-09-02 05:08:07 UTC
Hi I'm seeing this issue on RHEL 7.2 (Maipo). -bash-4.2$ sudo docker info Containers: 14 Running: 0 Paused: 0 Stopped: 14 Images: 1 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: docker-0:38-148560608-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 90.24 MB Data Space Total: 107.4 GB Data Space Available: 44.64 GB Metadata Space Used: 700.4 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2016-06-09) Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local Network: bridge null host Kernel Version: 3.10.0-327.28.3.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 1 Total Memory: 1.797 GiB Name: poc-docker02.aipo.gov.au ID: S7TX:CAUS:656Z:TLJN:IS2F:CICD:RISE:7N76:MRSA:HUBP:MINL:WU7M Debug mode (server): true File Descriptors: 14 Goroutines: 19 System Time: 2016-09-02T15:03:28.299818764+10:00 EventsListeners: 0 Init SHA1: 1f87ef7fa7fd7401b9aa61ca3a3e096b4fd2228e Init Path: Docker Root Dir: /var/lib/docker Http Proxy: http://proxy.aipo.gov.au:3128/ Https Proxy: http://proxy.aipo.gov.au:3128/ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Registries: docker.io (secure) -------------- -bash-4.2$ sudo docker version Client: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-46.el7.10.x86_64 Go version: go1.6.2 Git commit: 2a93377-unsupported Built: Fri Jul 29 13:45:25 2016 OS/Arch: linux/amd64 Server: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-46.el7.10.x86_64 Go version: go1.6.2 Git commit: 2a93377-unsupported Built: Fri Jul 29 13:45:25 2016 OS/Arch: linux/amd64 -------------------- -bash-4.2$ sudo docker run hello-world docker: Error response from daemon: Cannot start container 64e71ba9b9f627267d35c2c17f0b023ed9cf9f147d246d284907e32c3218a029:  System error: exit status 1. -bash-4.2$ ---------------------- Daniel's solution with chcon didn't work for us. Neither did reinstalling docker.
Comment 33 Daniel Walsh 2016-09-02 13:00:20 UTC
ps -eZ | grep docker
Comment 34 Lionel Orellana 2016-09-02 21:01:38 UTC
(In reply to Daniel Walsh from comment #33) > ps -eZ | grep docker -bash-4.2$ ps -eZ | grep docker system_u:system_r:docker_t:s0 2977 ? 00:00:21 docker-current
Comment 35 Daniel Walsh 2016-09-03 09:33:51 UTC
That looks correct, so I don't believe you are having the same problem as reported here. Please open a separate bug and attach the output of journalctl -b -u docker.service, after causing the issue.
Comment 36 Lionel Orellana 2016-09-04 23:31:51 UTC
(In reply to Daniel Walsh from comment #35) > That looks correct, so I don't believe you are having the same problem as > reported here. > > Please open a separate bug and attach the output of journalctl -b -u > docker.service, after causing the issue. Thanks Daniel.
Comment 39 Fedora End Of Life 2017-07-25 20:27:01 UTC
This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Comment 40 Jan Pazdziora 2017-07-28 12:26:30 UTC
Lokesh, if a fix went out as errata to Fedora 24, could you please set the state of this bugzilla accordingly?
Comment 41 Fedora End Of Life 2017-08-08 14:08:15 UTC
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.