Bug 1322909

docker-1.10.3-4.gitf8a9a2a.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-976f49409c

docker-1.10.3-4.gitf8a9a2a.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Reopening, this is still happening with docker-1.10.3-4.gitf8a9a2a.fc24 on x86_64 and armhfp. Working on aarch64.

could you provide the output of docker info? I'm about to rebuild docker for f24 (I'll link the bodhi update to test it out and see if it's still happening)

Just built this new docker version, might want to try it out and leave comments/karmas

https://bodhi.fedoraproject.org/updates/FEDORA-2016-f656582969

no change with docker-1.10.3-5.gitef2fa35.fc24.x86_64 

[root@localhost ~]# docker info 
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 2
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-253:3-6161116-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 501.5 MB
 Data Space Total: 107.4 GB
 Data Space Available: 106.9 GB
 Metadata Space Used: 843.8 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.122 (2016-04-09)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.5.0-302.fc24.x86_64
Operating System: Fedora 24 (Twenty Four)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 4
Total Memory: 3.745 GiB
Name: localhost.localdomain
ID: NKVP:ISGA:XV3P:2KSQ:TQDF:HAW7:3W34:P7QK:VJSU:UL6C:WFRR:3KR6
Registries: docker.io (secure)

Created attachment 1149121 [details]
journalctl output

Dennis, what blocker criterion do you believe this is violating?

while technically not virt https://fedoraproject.org/wiki/Fedora_24_Beta_Release_Criteria#Virtualization_requirements I think it falls into the same boat.

Hmm, this is very odd. I'm able to run things just fine under docker-1.10.3-5.gitef2fa35.fc24.x86_64 on my personal workstation, but if I spin up a new Fedora Server VM, I can see the error you're reporting here.

I'm not sure what the difference is between the two systems.

I just tested this w/ success on my laptop w/ f24, and again in an f24 server VM w/ 1.10.3-5.gitef2fa35.fc24. 

I was running: docker run -t -i docker.io/fedora /bin/bash

I see that one of the commenters was using loopback storage, which I don't use on my laptop. My VM didn't have loopback storage configured, either, and I didn't do anything special to avoid loopback -- the auto-partitioning set me up correctly. I installed my VM w/ a netinstall iso from http://mirror.math.princeton.edu/pub/fedora/linux//development/24/Server/x86_64/iso/, and gave it a 20GB disk, if that's relevant.

[root@localhost ~]# docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: fedora-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 284.2 MB
 Data Space Total: 2.147 GB
 Data Space Available: 1.863 GB
 Metadata Space Used: 90.11 kB
 Metadata Space Total: 20.97 MB
 Metadata Space Available: 20.88 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.122 (2016-04-09)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.5.1-300.fc24.x86_64
Operating System: Fedora 24 (Server Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 0
CPUs: 2
Total Memory: 1.954 GiB
Name: localhost.localdomain
ID: 3EPJ:NXD7:BLT6:DTF7:G3C7:7PG4:HN7B:INK4:LB5X:Y6CI:UM4S:ITGS
Registries: docker.io (secure)

Changing the root filesystem of the host to xfs, docker runs as expected on armhfp. 

Failure seems to occur when using ext4.

filesystem appears to be a red herring, new install using the development repo and docker-1.10.3-5.gitef2fa35.fc24.armv7hl is working fine on armhfp with an ext4 root.

This is reproducible by 

1) install docker-1.10.3-4.gitf8a9a2a.fc24, attempt to start a docker image

2) Update to docker-1.10.3-5.gitef2fa35.fc24.x86_64, attempt to start a docker image

3) fixed by removing docker and installing docker-1.10.3-5.gitef2fa35.fc24.x86_64

Weird, not sure what might have caused that.

Well if the latest works, we will not be fixing this update issue.  We should just be shipping docker-1.10.3-5.gitef2fa35.fc24.x86_64

dwalsh: but this was initially filed with -3, then reported with -4, now pwhalen says there's a reproducer when going from -4 to -5. Could it perhaps be the case that there's something in the docker scriptlets or something which is sabotaging updates, so it's just the case that whenever there's a docker update, it blows up? If so that seems like something we'd want to fix. I'm only speculating, though.

it works by following step from Paul Whalen, and rm -rf /var/lib/docker/*

It's not just upgrades; I reproduced this issue on a clean install of Fedora Server from the nightly build on April 21st with no updates.

Stephen, did you do it with the -5 version?

OK, new information. I installed from http://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/x86_64/iso/Fedora-Server-dvd-x86_64-Rawhide-20160425.n.0.iso this morning which has docker-1.10.3-4.gitf8a9a2a.fc24.x86_64

This configuration created a 15GiB main partition and left a few gigs available for docker-storage-setup to turn into a thin pool, which it did successfully.

The `docker pull` and `docker run` commands noted in the description above both worked successfully. I updated to docker-1.10.3-5.gitef2fa35.fc24 and confirmed that it worked as well. So far, so good.

I then downgraded to docker-1.10.3-3.gitd93ee51.fc24 and verified that the original bug reappears, which it did. (I think this may have been the version I was using in my initial testing before).

I then upgraded back to docker-1.10.3-5.gitef2fa35.fc24 to see if the problem went away. *It did not*. I deleted the image, pulled it again and tried to start it again: it still failed with the original bug report.

I stopped docker, ran `rm -rf /var/lib/docker` and tried to restart. It failed with an error about the thin pool having the wrong configuration, so I deleted the LVM thin pool and then restarted docker-storage-setup to recreate it. After all of this, the system was restored to a working state. (*)

So the problem we have right now is that for any user who gets into the bad state, they have no simple recourse for getting back out of it. We should also test whether this is a problem when upgrading from Fedora 23. (Though I expect that this will skip over the problematic versions now that -4 is in the stable repo).


(*) For people using the loopback device, I *think* that should just have been cleared by removing /var/lib/docker, so this part may be optional, but I haven't tested it.

Discussed at 2016-04-25 blocker review meeting: [1]. 

This bug was rejecte as Beta blocker: this is rejected for now as sgallagh's testing indicates the current stable package is not affected and there are no docker criteria in any case, but it may be re-evaluated if the bug affects -4 and we add docker criteria as proposed

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2016-02-29/f24-blocker-review.2016-04-25-17.02.html

Stephen, I think you need to destroy your devmapper image as well as removing /var/lib/docker

Seeing this with docker-1.10.3-9.git667d6d1.fc24
taking oci-register-machine-0.1.1 off koji resolves the issue.
Dupe of 1320601?

Antonio doesn't this happen if the cgroup config in docker.service is not setup correctly?

Dan, right, I believe we fixed that in latest 1.10.3 for Fedora.

Can anyone see if the latest docker fixes this BZ?

docker-1.10.3-21.git19b5791.fc24.x86_64 fixes this for me after installing docker-selinux-1.10.3-21.git19b5791.fc24.x86_64 and restarting the docker daemon.

The problem here is docker-current is not labeled correctly.

chcon -t docker_exec_t /usr/bin/docker-current
systemctl restart docker

Should fix it.

Looks like we have the wrong version of docker-selinux, which does not label docker-currect correctly.

Hi 

I'm seeing this issue on RHEL 7.2 (Maipo).

-bash-4.2$ sudo docker info
Containers: 14
 Running: 0
 Paused: 0
 Stopped: 14
Images: 1
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-0:38-148560608-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 90.24 MB
 Data Space Total: 107.4 GB
 Data Space Available: 44.64 GB
 Metadata Space Used: 700.4 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.10.0-327.28.3.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 1.797 GiB
Name: poc-docker02.aipo.gov.au
ID: S7TX:CAUS:656Z:TLJN:IS2F:CICD:RISE:7N76:MRSA:HUBP:MINL:WU7M
Debug mode (server): true
 File Descriptors: 14
 Goroutines: 19
 System Time: 2016-09-02T15:03:28.299818764+10:00
 EventsListeners: 0
 Init SHA1: 1f87ef7fa7fd7401b9aa61ca3a3e096b4fd2228e
 Init Path: 
 Docker Root Dir: /var/lib/docker
Http Proxy: http://proxy.aipo.gov.au:3128/
Https Proxy: http://proxy.aipo.gov.au:3128/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Registries: docker.io (secure)

--------------

-bash-4.2$ sudo docker version
Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.10.x86_64
 Go version:      go1.6.2
 Git commit:      2a93377-unsupported
 Built:           Fri Jul 29 13:45:25 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.10.x86_64
 Go version:      go1.6.2
 Git commit:      2a93377-unsupported
 Built:           Fri Jul 29 13:45:25 2016
 OS/Arch:         linux/amd64

--------------------

-bash-4.2$ sudo docker run  hello-world                          
docker: Error response from daemon: Cannot start container 64e71ba9b9f627267d35c2c17f0b023ed9cf9f147d246d284907e32c3218a029: [9] System error: exit status 1.
-bash-4.2$

----------------------

Daniel's solution with chcon didn't work for us. Neither did reinstalling docker.

ps -eZ | grep docker

(In reply to Daniel Walsh from comment #33)
> ps -eZ | grep docker

-bash-4.2$ ps -eZ | grep docker
system_u:system_r:docker_t:s0    2977 ?        00:00:21 docker-current

That looks correct, so I don't believe you are having the same problem as reported here.

Please open a separate bug and attach the output of journalctl -b -u docker.service, after causing the issue.

(In reply to Daniel Walsh from comment #35)
> That looks correct, so I don't believe you are having the same problem as
> reported here.
> 
> Please open a separate bug and attach the output of journalctl -b -u
> docker.service, after causing the issue.

Thanks Daniel.

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Lokesh, if a fix went out as errata to Fedora 24, could you please set the state of this bugzilla accordingly?

Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.