RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1433223 - Docker refuses to start containers on RHEL 7.2
Summary: Docker refuses to start containers on RHEL 7.2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: container-selinux
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: Martin Jenner
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-17 05:46 UTC by Gan Huang
Modified: 2020-08-13 08:58 UTC (History)
11 users (show)

Fixed In Version: container-selinux-2.10-2.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-12 14:50:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0942 0 normal SHIPPED_LIVE container-selinux bug fix and enhancement update 2017-04-12 18:31:56 UTC

Description Gan Huang 2017-03-17 05:46:58 UTC 
Description of problem:
Docker refuses to start containers on RHEL 7.2

Version-Release number of selected component (if applicable):
3.10.0-327.49.2.el7.x86_64
Red Hat Enterprise Linux Server release 7.2 (Maipo)

docker-1.12.6-13.el7.x86_64
docker-client-1.12.6-13.el7.x86_64
docker-common-1.12.6-13.el7.x86_64
docker-rhel-push-plugin-1.12.6-13.el7.x86_64

libselinux-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
container-selinux-2.10-1.el7.noarch
selinux-policy-targeted-3.13.1-60.el7_2.13.noarch
libselinux-utils-2.2.2-6.el7.x86_64
selinux-policy-3.13.1-60.el7_2.13.noarch

How reproducible:
always

Steps to Reproduce:
# docker run --rm busybox echo 'hello'
Unable to find image 'busybox:latest' locally
Trying to pull repository registry.access.redhat.com/busybox ... 
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
7520415ce762: Pull complete 
Digest: sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f
panic: standard_init_linux.go:178: exec user process caused "permission denied" [recovered]
    panic: standard_init_linux.go:178: exec user process caused "permission denied"

goroutine 1 [running, locked to thread]:
panic(0x6f3000, 0xc42012e9a0)
    /usr/lib/golang/src/runtime/panic.go:500 +0x1a1
github.com/urfave/cli.HandleAction.func1(0xc42007f748)
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247
panic(0x6f3000, 0xc42012e9a0)
    /usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc42007f198, 0xc42001e098, 0xc42007f238)
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc42004e780, 0xaac9c0, 0xc42012e9a0)
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc420082780, 0x0, 0x0)
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main_unix.go:26 +0x66
reflect.Value.call(0x6ddd80, 0x769ce8, 0x13, 0x73c1c9, 0x4, 0xc42007f708, 0x1, 0x1, 0x4d17a8, 0x732020, ...)
    /usr/lib/golang/src/reflect/value.go:434 +0x5c8
reflect.Value.Call(0x6ddd80, 0x769ce8, 0x13, 0xc42007f708, 0x1, 0x1, 0xac2700, 0xc42007f6e8, 0x4da786)
    /usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6ddd80, 0x769ce8, 0xc420082780, 0x0, 0x0)
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0
github.com/urfave/cli.Command.Run(0x73c395, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d9d9, 0x51, 0x0, ...)
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b
github.com/urfave/cli.(*App).Run(0xc4200c6000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0)
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611
main.main()
    /builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main.go:137 +0xbd6



Actual results:


Expected results:


Additional info:
# ps -eZ | grep docker
system_u:system_r:unconfined_service_t:s0 10311 ? 00:00:01 dockerd-current
system_u:system_r:unconfined_service_t:s0 10318 ? 00:00:00 docker-containe

# ausearch -m avc -ts recent
<no matches>

# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-253:1-42027658-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 41.28 GB
 Metadata Space Used: 581.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge null host overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-327.49.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 3.702 GiB
Name: jenkins-service-openstack-images-rhel-7.2-updated-165-073d9a5ca
ID: SSDJ:PTV2:STVF:LT3C:HKZI:BIJG:RH3A:NEKD:EGIB:OO6J:G46A:GO7V
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.access.redhat.com/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8
Registries: registry.access.redhat.com (secure), docker.io (secure)
Comment 2 Alex Jia 2017-03-17 10:51:41 UTC 
(In reply to Gan Huang from comment #0)
> Description of problem:
> Docker refuses to start containers on RHEL 7.2
> 
> Version-Release number of selected component (if applicable):
> 3.10.0-327.49.2.el7.x86_64
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> 
> docker-1.12.6-13.el7.x86_64
> docker-client-1.12.6-13.el7.x86_64
> docker-common-1.12.6-13.el7.x86_64
> docker-rhel-push-plugin-1.12.6-13.el7.x86_64
> 
> libselinux-2.2.2-6.el7.x86_64
> libselinux-python-2.2.2-6.el7.x86_64
> container-selinux-2.10-1.el7.noarch
> selinux-policy-targeted-3.13.1-60.el7_2.13.noarch
> libselinux-utils-2.2.2-6.el7.x86_64
> selinux-policy-3.13.1-60.el7_2.13.noarch
> 

Please update your libselinux and selinux-policy version, the docker-1.12.6-13.el7.x86_64 and docker-1.12.6-14.el7.x86_64 both works well for me.

[root@dell-per630-02 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 (Maipo)

[root@dell-per630-02 ~]# rpm -q docker container-selinux selinux-policy libselinux
docker-1.12.6-13.el7.x86_64
container-selinux-2.10-1.el7.noarch
selinux-policy-3.13.1-130.el7.noarch
libselinux-2.5-10.el7.x86_64
[root@dell-per630-02 ~]# docker run -it --rm busybox /bin/sh
/ # exit
Comment 3 Daniel Walsh 2017-03-17 12:36:06 UTC 
Yes this looks like the 7.2 update problem we have been seeing.
Usually reinstalling container-selinux fixes the issue.

yum reinstall container-selinux
systemctl restart docker

The problem is caused by the extras branch which we are shipping docker and friends in.  Extras does not follow the standard Z Stream efforts, it is more of a rolling release.  So when you update a 7.2 version to the latest docker, it pulls in the 7.3 selinux-policy and container-selinux, this seems to cause container-selinux installation to fail, reinstalling it fixes the issue, and then docker will run with the proper labels.
Comment 5 Gan Huang 2017-03-17 14:02:02 UTC 
(In reply to Daniel Walsh from comment #3)
> Yes this looks like the 7.2 update problem we have been seeing.
> Usually reinstalling container-selinux fixes the issue.
> 
> yum reinstall container-selinux
> systemctl restart docker
> 

Unfortunately hit the issue after reinstalling container-selinux, "/usr/sbin/semodule: invalid option -- 'X'" was seen when reinstalling container-selinux.

# yum reinstall container-selinux
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
fast-datapath                                                                                                                                         | 4.0 kB  00:00:00     
pulp-rhel-7.2-server-rpms                                                                                                                             | 3.5 kB  00:00:00     
rhel7                                                                                                                                                 | 4.1 kB  00:00:00     
rhel7-extra                                                                                                                                           | 3.0 kB  00:00:00     
rhel7-extra/primary_db                                                                                                                                |  40 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package container-selinux.noarch 2:2.10-1.el7 will be reinstalled
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================
 Package                                        Arch                                Version                                   Repository                                Size
=============================================================================================================================================================================
Reinstalling:
 container-selinux                              noarch                              2:2.10-1.el7                              rhel7-extra                               26 k

Transaction Summary
=============================================================================================================================================================================
Reinstall  1 Package

Total download size: 26 k
Installed size: 32 k
Is this ok [y/d/N]: y
Downloading packages:
container-selinux-2.10-1.el7.noarch.rpm                                                                                                               |  26 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 2:container-selinux-2.10-1.el7.noarch                                                                                                                     1/1 
/usr/sbin/semodule: invalid option -- 'X'
  Verifying  : 2:container-selinux-2.10-1.el7.noarch                                                                                                                     1/1 

Installed:
  container-selinux.noarch 2:2.10-1.el7                                                                                                                                      

Complete!
[root@jenkins-service-openstack-images-rhel-7 ~]# systemctl restart docker
[root@jenkins-service-openstack-images-rhel-7 ~]# docker run -it --rm busybox /bin/sh
panic: standard_init_linux.go:178: exec user process caused "permission denied" [recovered]
	panic: standard_init_linux.go:178: exec user process caused "permission denied"

goroutine 1 [running, locked to thread]:
panic(0x6f3000, 0xc42011e9a0)
	/usr/lib/golang/src/runtime/panic.go:500 +0x1a1
github.com/urfave/cli.HandleAction.func1(0xc42007f748)
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247
panic(0x6f3000, 0xc42011e9a0)
	/usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc42007f198, 0xc42001e098, 0xc42007f238)
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc42004f400, 0xaac9c0, 0xc42011e9a0)
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc420082a00, 0x0, 0x0)
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main_unix.go:26 +0x66
reflect.Value.call(0x6ddd80, 0x769ce8, 0x13, 0x73c1c9, 0x4, 0xc42007f708, 0x1, 0x1, 0x4d17a8, 0x732020, ...)
	/usr/lib/golang/src/reflect/value.go:434 +0x5c8
reflect.Value.Call(0x6ddd80, 0x769ce8, 0x13, 0xc42007f708, 0x1, 0x1, 0xac2700, 0xc42007f6e8, 0x4da786)
	/usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6ddd80, 0x769ce8, 0xc420082a00, 0x0, 0x0)
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0
github.com/urfave/cli.Command.Run(0x73c395, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d9d9, 0x51, 0x0, ...)
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b
github.com/urfave/cli.(*App).Run(0xc4200ac000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0)
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611
main.main()
	/builddir/build/BUILD/docker-3a094bd354f91c08f76655da01921369b04714b5/runc-81b254244390bc636b20c87c34a3d9e1a8645069/main.go:137 +0xbd6
Comment 6 Gan Huang 2017-03-17 14:12:52 UTC 
(In reply to Alex Jia from comment #2)

> Please update your libselinux and selinux-policy version, the
> docker-1.12.6-13.el7.x86_64 and docker-1.12.6-14.el7.x86_64 both works well
> for me.
> 
Hit another issue after updating these packages to latest. Note: we only hit this issue on RHEL 7.2. I suggest you re-test it on RHEL 7.2 if you don't mind.

# rpm -q docker container-selinux selinux-policy libselinux
docker-1.12.6-13.el7.x86_64
container-selinux-2.10-1.el7.noarch
selinux-policy-3.13.1-129.el7.noarch
libselinux-2.5-10.el7.x86_64

# systemctl restart docker

# docker run -it --rm busybox /bin/sh
/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:334: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: \\\"\"\n".
Comment 7 Daniel Walsh 2017-03-17 14:26:33 UTC 
So this looks like we need to add a policycoreutils update requirement to container-selinux.

The problem is we need the semanage from RHEL7.3 which has the new -X Flag.
Comment 10 Chris Evich 2017-03-20 13:49:46 UTC 
New card: https://trello.com/c/id2OeNeL
Comment 11 Daniel Walsh 2017-03-20 14:33:22 UTC 
Ok I just kicked off builds of container-selinux 
container-selinux-2.10-2.1.el7 for 7.4
container-selinux-2.10-2.el7 for 7.3

Which require an updated version of policycoreutils.  This will hopefully solve the issue we are seeing on container-selinux installation on 7.2 boxes.
Comment 12 Daniel Walsh 2017-03-20 14:34:56 UTC 
Fixed in container-selinux-2.10-2.el7

When testing make sure we test a 7.2 box updating to latest extras packages.
Comment 25 Alex Jia 2017-04-01 09:19:16 UTC 
I can reproduce the bug, and it works well in container-selinux-2.10-2.el7 w/ the following RPMs:

[root@localhost ~]# rpm -q docker container-selinux oci-register-machine oci-systemd-hook skopeo-containers policycoreutils libselinux libsemanage libsepol systemd
docker-1.12.6-13.el7.x86_64
container-selinux-2.10-2.el7.noarch
oci-register-machine-0-3.11.1.gitdd0daef.el7.x86_64
oci-systemd-hook-0.1.7-1.git1788cf2.el7.x86_64
skopeo-containers-0.1.18-1.2.el7.x86_64
policycoreutils-2.5-11.el7_3.x86_64
libselinux-2.5-6.el7.x86_64
libsemanage-2.5-5.el7_3.x86_64
libsepol-2.5-6.el7.x86_64
systemd-219-20.el7.x86_64

[root@localhost ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Comment 27 errata-xmlrpc 2017-04-12 14:50:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0942
Note You need to log in before you can comment on or make changes to this bug.