Description of problem: docker run -t -i docker.io/fedora /bin/bash docker: Error response from daemon: Cannot start container 25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61: [9] System error: exit status 1. Version-Release number of selected component (if applicable): docker-1.10.3-3.gitd93ee51.fc24 How reproducible: always Steps to Reproduce: 1. docker pull docker.io/fedora 2. docker run -t -i docker.io/fedora /bin/bash 3. Actual results: journalctl shows mar 31 10:28:15 anubis.ausil.us docker[6644]: time="2016-03-31T10:28:15.700555196-05:00" level=info msg="{Action=start, ID=25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61, LoginUID=217600001, PID=7578, Config={Hostname=25e7cb9242e1, AttachStdin=true, AttachStdout=true, AttachStderr=true, Tty=true, OpenStdin=true, StdinOnce=true, Cmd={parts:[/bin/bash]}, ArgsEscaped=false, Image=docker.io/fedora, NetworkDisabled=false, Labels=map[], StopSignal=SIGTERM}, HostConfig={LogConfig={Type:journald Config:map[]}, NetworkMode=default, PortBindings=map[], RestartPolicy={Name:no MaximumRetryCount:0}, DNS=[], DNSOptions=[], DNSSearch=[], Privileged=false, PublishAllPorts=false, ReadonlyRootfs=false, ShmSize=67108864, Resources={CPUShares:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CpusetCpus: CpusetMems: Devices:[] KernelMemory:0 Memory:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:0xc820f7e220 OomKillDisable:0xc820f7e22a PidsLimit:0 Ulimits:[]}}}" mar 31 10:28:15 anubis.ausil.us docker[6644]: time="2016-03-31T10:28:15.822749576-05:00" level=info msg="Config: &{CommonCommand:{ContainerPid:0 ID:25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61 InitPath:/.dockerinit MountLabel:system_u:object_r:svirt_sandbox_file_t:s0:c976,c1012 Mounts:[{Source:/var/lib/docker/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/secrets Destination:/run/secrets Writable:true Data: Propagation:} {Source:/var/lib/docker/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/resolv.conf Destination:/etc/resolv.conf Writable:true Data: Propagation:rprivate} {Source:/var/lib/docker/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/hostname Destination:/etc/hostname Writable:true Data: Propagation:rprivate} {Source:/var/lib/docker/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/hosts Destination:/etc/hosts Writable:true Data: Propagation:rprivate} {Source:/var/lib/docker/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/shm Destination:/dev/shm Writable:true Data: Propagation:rprivate}] Network:0xc821050690 ProcessConfig:{CommonProcessConfig:{Cmd:{Path: Args:[] Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=25e7cb9242e1 TERM=xterm] Dir: Stdin:<nil> Stdout:<nil> Stderr:<nil> ExtraFiles:[] SysProcAttr:0xc821a97f80 Process:<nil> ProcessState:<nil> lookPathErr:<nil> finished:false childFiles:[] closeAfterStart:[] closeAfterWait:[] goroutine:[] errch:<nil>} Tty:true Entrypoint:/bin/bash Arguments:[] Terminal:<nil>} Privileged:false User: Console:} ProcessLabel:system_u:system_r:svirt_lxc_net_t:s0:c976,c1012 Resources:0xc820fddc00 Rootfs:/var/lib/docker/devicemapper/mnt/89bb08cb85b93bbce5af4ef09b90f166c2119a296a5c5c4ceef1f467de534657/rootfs WorkingDir: TmpDir:/var/lib/docker/tmp/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61437894604 ContainerJSONPath:/var/lib/docker/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/conf mar 31 10:28:15 anubis.ausil.us docker[6644]: ig.v2.json} AllowedDevices:[0xc82007ad70 0xc82007adc0 0xc82007ae10 0xc82007ae60 0xc82007aeb0 0xc82007af00 0xc82007af50 0xc82007afa0 0x214ad40 0x214ada0 0x214ae00 0x214ae60 0x214aec0 0x214af20] AppArmorProfile: AutoCreatedDevices:[0xc82007aff0 0x214ad40 0x214ada0 0x214ae00 0x214ae60 0x214aec0 0x214af20] CapAdd:[] CapDrop:[] CgroupParent:/docker GIDMapping:[] GroupAdd:[] Ipc:0xc820fa8040 OomScoreAdj:0 Pid:0xc820e1ee99 ReadonlyRootfs:false RemappedRoot:0xc820e1eea0 SeccompProfile: UIDMapping:[] UTS:0xc820e1ee9a}" mar 31 10:28:15 anubis.ausil.us oci-register-machine[7652]: 2016/03/31 10:28:15 Register machine: prestart %!d(string=25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61) %!s(int=7641) /var/lib/docker/devicemapper/mnt/89bb08cb85b93bbce5af4ef09b90f166c2119a296a5c5c4ceef1f467de534657/rootfs mar 31 10:28:16 anubis.ausil.us docker[6644]: time="2016-03-31T10:28:16.068558138-05:00" level=warning msg="signal: killed" mar 31 10:28:16 anubis.ausil.us oci-register-machine[7660]: 2016/03/31 10:28:16 Register machine: poststop %!d(string=25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61) %!s(int=0) /var/lib/docker/devicemapper/mnt/89bb08cb85b93bbce5af4ef09b90f166c2119a296a5c5c4ceef1f467de534657/rootfs mar 31 10:28:16 anubis.ausil.us docker[6644]: time="2016-03-31T10:28:16.302559264-05:00" level=error msg="error locating sandbox id dc40e70de8a9b3f27b4124b9569557ae1b4fbcde6981e27d30140dfdf78c80be: sandbox dc40e70de8a9b3f27b4124b9569557ae1b4fbcde6981e27d30140dfdf78c80be not found" mar 31 10:28:16 anubis.ausil.us docker[6644]: time="2016-03-31T10:28:16.302641257-05:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/shm: invalid argument" mar 31 10:28:16 anubis.ausil.us docker[6644]: time="2016-03-31T10:28:16.302677784-05:00" level=error msg="Error unmounting container 25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61: not mounted" mar 31 10:28:16 anubis.ausil.us docker[6644]: time="2016-03-31T10:28:16.302778076-05:00" level=error msg="Handler for POST /v1.22/containers/25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61/start returned error: Cannot start container 25e7cb9242e1d5228037b83546ce07646e365e1f4a5f1d0a2dd3e8e39bcbed61: [9] System error: exit status 1" Expected results: container to run and get a shell Additional info: I was able to also reproduce on arm but using docker load to import a Fedora base image
docker-1.10.3-4.gitf8a9a2a.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-976f49409c
docker-1.10.3-4.gitf8a9a2a.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Reopening, this is still happening with docker-1.10.3-4.gitf8a9a2a.fc24 on x86_64 and armhfp. Working on aarch64.
could you provide the output of docker info? I'm about to rebuild docker for f24 (I'll link the bodhi update to test it out and see if it's still happening)
Just built this new docker version, might want to try it out and leave comments/karmas https://bodhi.fedoraproject.org/updates/FEDORA-2016-f656582969
no change with docker-1.10.3-5.gitef2fa35.fc24.x86_64 [root@localhost ~]# docker info Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 2 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: docker-253:3-6161116-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 501.5 MB Data Space Total: 107.4 GB Data Space Available: 106.9 GB Metadata Space Used: 843.8 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.122 (2016-04-09) Execution Driver: native-0.2 Logging Driver: journald Plugins: Volume: local Network: host bridge null Kernel Version: 4.5.0-302.fc24.x86_64 Operating System: Fedora 24 (Twenty Four) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 4 Total Memory: 3.745 GiB Name: localhost.localdomain ID: NKVP:ISGA:XV3P:2KSQ:TQDF:HAW7:3W34:P7QK:VJSU:UL6C:WFRR:3KR6 Registries: docker.io (secure)
Created attachment 1149121 [details] journalctl output
Dennis, what blocker criterion do you believe this is violating?
while technically not virt https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Virtualization_requirements I think it falls into the same boat.
Hmm, this is very odd. I'm able to run things just fine under docker-1.10.3-5.gitef2fa35.fc24.x86_64 on my personal workstation, but if I spin up a new Fedora Server VM, I can see the error you're reporting here. I'm not sure what the difference is between the two systems.
I just tested this w/ success on my laptop w/ f24, and again in an f24 server VM w/ 1.10.3-5.gitef2fa35.fc24. I was running: docker run -t -i docker.io/fedora /bin/bash I see that one of the commenters was using loopback storage, which I don't use on my laptop. My VM didn't have loopback storage configured, either, and I didn't do anything special to avoid loopback -- the auto-partitioning set me up correctly. I installed my VM w/ a netinstall iso from http://mirror.math.princeton.edu/pub/fedora/linux//development/24/Server/x86_64/iso/, and gave it a 20GB disk, if that's relevant. [root@localhost ~]# docker info Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 1 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: fedora-docker--pool Pool Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 284.2 MB Data Space Total: 2.147 GB Data Space Available: 1.863 GB Metadata Space Used: 90.11 kB Metadata Space Total: 20.97 MB Metadata Space Available: 20.88 MB Udev Sync Supported: true Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.122 (2016-04-09) Execution Driver: native-0.2 Logging Driver: journald Plugins: Volume: local Network: null host bridge Kernel Version: 4.5.1-300.fc24.x86_64 Operating System: Fedora 24 (Server Edition) OSType: linux Architecture: x86_64 Number of Docker Hooks: 0 CPUs: 2 Total Memory: 1.954 GiB Name: localhost.localdomain ID: 3EPJ:NXD7:BLT6:DTF7:G3C7:7PG4:HN7B:INK4:LB5X:Y6CI:UM4S:ITGS Registries: docker.io (secure)
Changing the root filesystem of the host to xfs, docker runs as expected on armhfp. Failure seems to occur when using ext4.
filesystem appears to be a red herring, new install using the development repo and docker-1.10.3-5.gitef2fa35.fc24.armv7hl is working fine on armhfp with an ext4 root.
This is reproducible by 1) install docker-1.10.3-4.gitf8a9a2a.fc24, attempt to start a docker image 2) Update to docker-1.10.3-5.gitef2fa35.fc24.x86_64, attempt to start a docker image 3) fixed by removing docker and installing docker-1.10.3-5.gitef2fa35.fc24.x86_64
Weird, not sure what might have caused that.
Well if the latest works, we will not be fixing this update issue. We should just be shipping docker-1.10.3-5.gitef2fa35.fc24.x86_64
dwalsh: but this was initially filed with -3, then reported with -4, now pwhalen says there's a reproducer when going from -4 to -5. Could it perhaps be the case that there's something in the docker scriptlets or something which is sabotaging updates, so it's just the case that whenever there's a docker update, it blows up? If so that seems like something we'd want to fix. I'm only speculating, though.
it works by following step from Paul Whalen, and rm -rf /var/lib/docker/*
It's not just upgrades; I reproduced this issue on a clean install of Fedora Server from the nightly build on April 21st with no updates.
Stephen, did you do it with the -5 version?
OK, new information. I installed from http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/x86_64/iso/Fedora-Server-dvd-x86_64-Rawhide-20160425.n.0.iso this morning which has docker-1.10.3-4.gitf8a9a2a.fc24.x86_64 This configuration created a 15GiB main partition and left a few gigs available for docker-storage-setup to turn into a thin pool, which it did successfully. The `docker pull` and `docker run` commands noted in the description above both worked successfully. I updated to docker-1.10.3-5.gitef2fa35.fc24 and confirmed that it worked as well. So far, so good. I then downgraded to docker-1.10.3-3.gitd93ee51.fc24 and verified that the original bug reappears, which it did. (I think this may have been the version I was using in my initial testing before). I then upgraded back to docker-1.10.3-5.gitef2fa35.fc24 to see if the problem went away. *It did not*. I deleted the image, pulled it again and tried to start it again: it still failed with the original bug report. I stopped docker, ran `rm -rf /var/lib/docker` and tried to restart. It failed with an error about the thin pool having the wrong configuration, so I deleted the LVM thin pool and then restarted docker-storage-setup to recreate it. After all of this, the system was restored to a working state. (*) So the problem we have right now is that for any user who gets into the bad state, they have no simple recourse for getting back out of it. We should also test whether this is a problem when upgrading from Fedora 23. (Though I expect that this will skip over the problematic versions now that -4 is in the stable repo). (*) For people using the loopback device, I *think* that should just have been cleared by removing /var/lib/docker, so this part may be optional, but I haven't tested it.
Discussed at 2016-04-25 blocker review meeting: [1]. This bug was rejecte as Beta blocker: this is rejected for now as sgallagh's testing indicates the current stable package is not affected and there are no docker criteria in any case, but it may be re-evaluated if the bug affects -4 and we add docker criteria as proposed [1] http://meetbot.fedoraproject.org/fedora-blocker-review/2016-02-29/f24-blocker-review.2016-04-25-17.02.html
Stephen, I think you need to destroy your devmapper image as well as removing /var/lib/docker
Seeing this with docker-1.10.3-9.git667d6d1.fc24 taking oci-register-machine-0.1.1 off koji resolves the issue. Dupe of 1320601?
Antonio doesn't this happen if the cgroup config in docker.service is not setup correctly?
Dan, right, I believe we fixed that in latest 1.10.3 for Fedora.
Can anyone see if the latest docker fixes this BZ?
docker-1.10.3-21.git19b5791.fc24.x86_64 fixes this for me after installing docker-selinux-1.10.3-21.git19b5791.fc24.x86_64 and restarting the docker daemon.
The problem here is docker-current is not labeled correctly. chcon -t docker_exec_t /usr/bin/docker-current systemctl restart docker Should fix it. Looks like we have the wrong version of docker-selinux, which does not label docker-currect correctly.
Hi I'm seeing this issue on RHEL 7.2 (Maipo). -bash-4.2$ sudo docker info Containers: 14 Running: 0 Paused: 0 Stopped: 14 Images: 1 Server Version: 1.10.3 Storage Driver: devicemapper Pool Name: docker-0:38-148560608-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 90.24 MB Data Space Total: 107.4 GB Data Space Available: 44.64 GB Metadata Space Used: 700.4 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2016-06-09) Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local Network: bridge null host Kernel Version: 3.10.0-327.28.3.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 1 Total Memory: 1.797 GiB Name: poc-docker02.aipo.gov.au ID: S7TX:CAUS:656Z:TLJN:IS2F:CICD:RISE:7N76:MRSA:HUBP:MINL:WU7M Debug mode (server): true File Descriptors: 14 Goroutines: 19 System Time: 2016-09-02T15:03:28.299818764+10:00 EventsListeners: 0 Init SHA1: 1f87ef7fa7fd7401b9aa61ca3a3e096b4fd2228e Init Path: Docker Root Dir: /var/lib/docker Http Proxy: http://proxy.aipo.gov.au:3128/ Https Proxy: http://proxy.aipo.gov.au:3128/ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Registries: docker.io (secure) -------------- -bash-4.2$ sudo docker version Client: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-46.el7.10.x86_64 Go version: go1.6.2 Git commit: 2a93377-unsupported Built: Fri Jul 29 13:45:25 2016 OS/Arch: linux/amd64 Server: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-46.el7.10.x86_64 Go version: go1.6.2 Git commit: 2a93377-unsupported Built: Fri Jul 29 13:45:25 2016 OS/Arch: linux/amd64 -------------------- -bash-4.2$ sudo docker run hello-world docker: Error response from daemon: Cannot start container 64e71ba9b9f627267d35c2c17f0b023ed9cf9f147d246d284907e32c3218a029: [9] System error: exit status 1. -bash-4.2$ ---------------------- Daniel's solution with chcon didn't work for us. Neither did reinstalling docker.
ps -eZ | grep docker
(In reply to Daniel Walsh from comment #33) > ps -eZ | grep docker -bash-4.2$ ps -eZ | grep docker system_u:system_r:docker_t:s0 2977 ? 00:00:21 docker-current
That looks correct, so I don't believe you are having the same problem as reported here. Please open a separate bug and attach the output of journalctl -b -u docker.service, after causing the issue.
(In reply to Daniel Walsh from comment #35) > That looks correct, so I don't believe you are having the same problem as > reported here. > > Please open a separate bug and attach the output of journalctl -b -u > docker.service, after causing the issue. Thanks Daniel.
This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Lokesh, if a fix went out as errata to Fedora 24, could you please set the state of this bugzilla accordingly?
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.