Bug 1433223
| Summary: | Docker refuses to start containers on RHEL 7.2 | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Gan Huang <ghuang> |
| Component: | container-selinux | Assignee: | Lokesh Mandvekar <lsm5> |
| Status: | CLOSED ERRATA | QA Contact: | Martin Jenner <mjenner> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.2 | CC: | ajia, amurdaca, anli, cevich, ddarrah, dwalsh, ghuang, lsm5, lsu, santiago, wmeng |
| Target Milestone: | rc | Keywords: | Extras |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | container-selinux-2.10-2.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-04-12 14:50:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(In reply to Gan Huang from comment #0) > Description of problem: > Docker refuses to start containers on RHEL 7.2 > > Version-Release number of selected component (if applicable): > 3.10.0-327.49.2.el7.x86_64 > Red Hat Enterprise Linux Server release 7.2 (Maipo) > > docker-1.12.6-13.el7.x86_64 > docker-client-1.12.6-13.el7.x86_64 > docker-common-1.12.6-13.el7.x86_64 > docker-rhel-push-plugin-1.12.6-13.el7.x86_64 > > libselinux-2.2.2-6.el7.x86_64 > libselinux-python-2.2.2-6.el7.x86_64 > container-selinux-2.10-1.el7.noarch > selinux-policy-targeted-3.13.1-60.el7_2.13.noarch > libselinux-utils-2.2.2-6.el7.x86_64 > selinux-policy-3.13.1-60.el7_2.13.noarch > Please update your libselinux and selinux-policy version, the docker-1.12.6-13.el7.x86_64 and docker-1.12.6-14.el7.x86_64 both works well for me. [root@dell-per630-02 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo) [root@dell-per630-02 ~]# rpm -q docker container-selinux selinux-policy libselinux docker-1.12.6-13.el7.x86_64 container-selinux-2.10-1.el7.noarch selinux-policy-3.13.1-130.el7.noarch libselinux-2.5-10.el7.x86_64 [root@dell-per630-02 ~]# docker run -it --rm busybox /bin/sh / # exit Yes this looks like the 7.2 update problem we have been seeing. Usually reinstalling container-selinux fixes the issue. yum reinstall container-selinux systemctl restart docker The problem is caused by the extras branch which we are shipping docker and friends in. Extras does not follow the standard Z Stream efforts, it is more of a rolling release. So when you update a 7.2 version to the latest docker, it pulls in the 7.3 selinux-policy and container-selinux, this seems to cause container-selinux installation to fail, reinstalling it fixes the issue, and then docker will run with the proper labels. (In reply to Daniel Walsh from comment #3) > Yes this looks like the 7.2 update problem we have been seeing. > Usually reinstalling container-selinux fixes the issue. > > yum reinstall container-selinux > systemctl restart docker > Unfortunately hit the issue after reinstalling container-selinux, "/usr/sbin/semodule: invalid option -- 'X'" was seen when reinstalling container-selinux. # yum reinstall container-selinux Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. fast-datapath | 4.0 kB 00:00:00 pulp-rhel-7.2-server-rpms | 3.5 kB 00:00:00 rhel7 | 4.1 kB 00:00:00 rhel7-extra | 3.0 kB 00:00:00 rhel7-extra/primary_db | 40 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package container-selinux.noarch 2:2.10-1.el7 will be reinstalled --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================= Reinstalling: container-selinux noarch 2:2.10-1.el7 rhel7-extra 26 k Transaction Summary ============================================================================================================================================================================= Reinstall 1 Package Total download size: 26 k Installed size: 32 k Is this ok [y/d/N]: y Downloading packages: container-selinux-2.10-1.el7.noarch.rpm | 26 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 2:container-selinux-2.10-1.el7.noarch 1/1 /usr/sbin/semodule: invalid option -- 'X' Verifying : 2:container-selinux-2.10-1.el7.noarch 1/1 Installed: container-selinux.noarch 2:2.10-1.el7 Complete! [root@jenkins-service-openstack-images-rhel-7 ~]# systemctl restart docker [root@jenkins-service-openstack-images-rhel-7 ~]# docker run -it --rm busybox /bin/sh panic: standard_init_linux.go:178: exec user process caused "permission denied" [recovered] panic: standard_init_linux.go:178: exec user process caused "permission denied" goroutine 1 [running, locked to thread]: panic(0x6f3000, 0xc42011e9a0) /usr/lib/golang/src/runtime/panic.go:500 +0x1a1 github.com/urfave/cli.HandleAction.func1(0xc42007f748) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247 panic(0x6f3000, 0xc42011e9a0) /usr/lib/golang/src/runtime/panic.go:458 +0x243 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc42007f198, 0xc42001e098, 0xc42007f238) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc42004f400, 0xaac9c0, 0xc42011e9a0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353 main.glob..func8(0xc420082a00, 0x0, 0x0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main_unix.go:26 +0x66 reflect.Value.call(0x6ddd80, 0x769ce8, 0x13, 0x73c1c9, 0x4, 0xc42007f708, 0x1, 0x1, 0x4d17a8, 0x732020, ...) /usr/lib/golang/src/reflect/value.go:434 +0x5c8 reflect.Value.Call(0x6ddd80, 0x769ce8, 0x13, 0xc42007f708, 0x1, 0x1, 0xac2700, 0xc42007f6e8, 0x4da786) /usr/lib/golang/src/reflect/value.go:302 +0xa4 github.com/urfave/cli.HandleAction(0x6ddd80, 0x769ce8, 0xc420082a00, 0x0, 0x0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0 github.com/urfave/cli.Command.Run(0x73c395, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d9d9, 0x51, 0x0, ...) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b github.com/urfave/cli.(*App).Run(0xc4200ac000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611 main.main() /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main.go:137 +0xbd6 (In reply to Alex Jia from comment #2) > Please update your libselinux and selinux-policy version, the > docker-1.12.6-13.el7.x86_64 and docker-1.12.6-14.el7.x86_64 both works well > for me. > Hit another issue after updating these packages to latest. Note: we only hit this issue on RHEL 7.2. I suggest you re-test it on RHEL 7.2 if you don't mind. # rpm -q docker container-selinux selinux-policy libselinux docker-1.12.6-13.el7.x86_64 container-selinux-2.10-1.el7.noarch selinux-policy-3.13.1-129.el7.noarch libselinux-2.5-10.el7.x86_64 # systemctl restart docker # docker run -it --rm busybox /bin/sh /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n". So this looks like we need to add a policycoreutils update requirement to container-selinux. The problem is we need the semanage from RHEL7.3 which has the new -X Flag. New card: https://trello.com/c/id2OeNeL Ok I just kicked off builds of container-selinux container-selinux-2.10-2.1.el7 for 7.4 container-selinux-2.10-2.el7 for 7.3 Which require an updated version of policycoreutils. This will hopefully solve the issue we are seeing on container-selinux installation on 7.2 boxes. Fixed in container-selinux-2.10-2.el7 When testing make sure we test a 7.2 box updating to latest extras packages. I can reproduce the bug, and it works well in container-selinux-2.10-2.el7 w/ the following RPMs: [root@localhost ~]# rpm -q docker container-selinux oci-register-machine oci-systemd-hook skopeo-containers policycoreutils libselinux libsemanage libsepol systemd docker-1.12.6-13.el7.x86_64 container-selinux-2.10-2.el7.noarch oci-register-machine-0-3.11.1.gitdd0daef.el7.x86_64 oci-systemd-hook-0.1.7-1.git1788cf2.el7.x86_64 skopeo-containers-0.1.18-1.2.el7.x86_64 policycoreutils-2.5-11.el7_3.x86_64 libselinux-2.5-6.el7.x86_64 libsemanage-2.5-5.el7_3.x86_64 libsepol-2.5-6.el7.x86_64 systemd-219-20.el7.x86_64 [root@localhost ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.2 (Maipo) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0942 |
Description of problem: Docker refuses to start containers on RHEL 7.2 Version-Release number of selected component (if applicable): 3.10.0-327.49.2.el7.x86_64 Red Hat Enterprise Linux Server release 7.2 (Maipo) docker-1.12.6-13.el7.x86_64 docker-client-1.12.6-13.el7.x86_64 docker-common-1.12.6-13.el7.x86_64 docker-rhel-push-plugin-1.12.6-13.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libselinux-python-2.2.2-6.el7.x86_64 container-selinux-2.10-1.el7.noarch selinux-policy-targeted-3.13.1-60.el7_2.13.noarch libselinux-utils-2.2.2-6.el7.x86_64 selinux-policy-3.13.1-60.el7_2.13.noarch How reproducible: always Steps to Reproduce: # docker run --rm busybox echo 'hello' Unable to find image 'busybox:latest' locally Trying to pull repository registry.access.redhat.com/busybox ... Trying to pull repository docker.io/library/busybox ... latest: Pulling from docker.io/library/busybox 7520415ce762: Pull complete Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f panic: standard_init_linux.go:178: exec user process caused "permission denied" [recovered] panic: standard_init_linux.go:178: exec user process caused "permission denied" goroutine 1 [running, locked to thread]: panic(0x6f3000, 0xc42012e9a0) /usr/lib/golang/src/runtime/panic.go:500 +0x1a1 github.com/urfave/cli.HandleAction.func1(0xc42007f748) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247 panic(0x6f3000, 0xc42012e9a0) /usr/lib/golang/src/runtime/panic.go:458 +0x243 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc42007f198, 0xc42001e098, 0xc42007f238) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc42004e780, 0xaac9c0, 0xc42012e9a0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353 main.glob..func8(0xc420082780, 0x0, 0x0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main_unix.go:26 +0x66 reflect.Value.call(0x6ddd80, 0x769ce8, 0x13, 0x73c1c9, 0x4, 0xc42007f708, 0x1, 0x1, 0x4d17a8, 0x732020, ...) /usr/lib/golang/src/reflect/value.go:434 +0x5c8 reflect.Value.Call(0x6ddd80, 0x769ce8, 0x13, 0xc42007f708, 0x1, 0x1, 0xac2700, 0xc42007f6e8, 0x4da786) /usr/lib/golang/src/reflect/value.go:302 +0xa4 github.com/urfave/cli.HandleAction(0x6ddd80, 0x769ce8, 0xc420082780, 0x0, 0x0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0 github.com/urfave/cli.Command.Run(0x73c395, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d9d9, 0x51, 0x0, ...) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b github.com/urfave/cli.(*App).Run(0xc4200c6000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0) /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611 main.main() /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main.go:137 +0xbd6 Actual results: Expected results: Additional info: # ps -eZ | grep docker system_u:system_r:unconfined_service_t:s0 10311 ? 00:00:01 dockerd-current system_u:system_r:unconfined_service_t:s0 10318 ? 00:00:00 docker-containe # ausearch -m avc -ts recent <no matches> # docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 1.12.6 Storage Driver: devicemapper Pool Name: docker-253:1-42027658-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 11.8 MB Data Space Total: 107.4 GB Data Space Available: 41.28 GB Metadata Space Used: 581.6 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Thin Pool Minimum Free Space: 10.74 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.135-RHEL7 (2016-11-16) Logging Driver: journald Cgroup Driver: systemd Plugins: Volume: local Network: bridge null host overlay Authorization: rhel-push-plugin Swarm: inactive Runtimes: docker-runc runc Default Runtime: docker-runc Security Options: seccomp selinux Kernel Version: 3.10.0-327.49.2.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo) OSType: linux Architecture: x86_64 Number of Docker Hooks: 2 CPUs: 2 Total Memory: 3.702 GiB Name: jenkins-service-openstack-images-rhel-7.2-updated-165-073d9a5ca ID: SSDJ:PTV2:STVF:LT3C:HKZI:BIJG:RH3A:NEKD:EGIB:OO6J:G46A:GO7V Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://registry.access.redhat.com/v1/ WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Insecure Registries: 127.0.0.0/8 Registries: registry.access.redhat.com (secure), docker.io (secure)