Bug 1550332

Summary: system container failed to start due to container-selinux identified unexpected selinux label against binary /bin/bash
Product: Red Hat Enterprise Linux 7 Reporter: Gan Huang <ghuang>
Component: container-selinuxAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED ERRATA QA Contact: Johnny Liu <jialiu>
Severity: high Docs Contact:
Priority: high    
Version: 7.5CC: aos-bugs, ddarrah, dwalsh, ghuang, gscrivan, jokerman, mmccomas, sdodson
Target Milestone: betaKeywords: Extras
Target Release: 7.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: container-selinux-2.51-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-11 00:03:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gan Huang 2018-03-01 03:05:05 UTC
Description of problem:
system container failed to start due to container-selinux identified unexpected selinux label against /bin/bash

Version-Release number of the following components:
container-selinux-2.48-1.el7.noarch
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)

How reproducible:
always

Steps to Reproduce:
1. Trigger OCP installation with docker system container enabled
2.
3.

Actual results:
Installation failed as container-engine failed to start

# ll -Z /bin/bash
-rwxr-xr-x. root root system_u:object_r:shell_exec_t:s0 /bin/bash

# systemctl cat container-engine |grep -i execstart
ExecStartPre=/bin/bash -c 'export -p > /run/docker-bash-env'
ExecStart=/usr/bin/runc --systemd-cgroup run 'container-engine'

Test a: after adding label container_runtime_exec_t for /bin/bash, it could restart successfully
# chcon -t container_runtime_exec_t /bin/bash

Test b: after downgrading to container-selinux-2.36-1.gitff95335.el7.noarch, could restart successfully as well.

After adding label for 
Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Giuseppe Scrivano 2018-03-01 14:06:21 UTC
this might depend from:

https://bugzilla.redhat.com/show_bug.cgi?id=1544175

Comment 3 Steve Milner 2018-03-01 21:16:06 UTC
(In reply to Giuseppe Scrivano from comment #2)
> this might depend from:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1544175

Are you saying it's the same issue, it's similar to that issue, or that it may be a bug that came from issues fix?

Comment 5 Gan Huang 2018-03-02 03:32:12 UTC
I think Steve's statement is correct, this is indeed caused by the new package container-selinux. The thought of moving to installer component was that we might to need fix it via the installer if that is an intended change of container-selinux.

Comment 6 Giuseppe Scrivano 2018-03-02 08:27:27 UTC
the issue I linked was related to the fact that we wrongly labelled the root of the checkout, so everything started to behave in a strange way.

Could you try to remove the "SELinuxContext=system_u:system_r:container_runtime_t:s0" from the service file for the Docker system container?  Does it solve the issue?  Also, if it works, what SELinux label does the Docker process get?

Comment 7 Steve Milner 2018-03-02 15:48:19 UTC
I don't seem to have the same issue when installing manually. Gan, can you confirm what version of the container and what version of atomic cli is in use?

---

[root@rhel-7 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)
[root@rhel-7 ~]# ls -laZ /bin/bash
-rwxr-xr-x. root root system_u:object_r:shell_exec_t:s0 /bin/bash
[root@rhel-7 ~]# rpm -q container-selinux
container-selinux-2.48-1.el7.noarch
[root@rhel-7 ~]# atomic install --system --system-package=no --name ce registry.access.redhat.com/openshift3/container-engine:latest

<snip>

<install docker client>

[root@rhel-7 ~]# atomic images list
   REPOSITORY                                               TAG      IMAGE ID       CREATED            VIRTUAL SIZE   TYPE      
>  registry.access.redhat.com/openshift3/container-engine   latest   4693614da0bf   2018-03-02 10:39   126.8 MB       ostree  
[root@rhel-7 ~]# systemctl start ce
[root@rhel-7 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@rhel-7 ~]# ls -laZ /bin/bash
-rwxr-xr-x. root root system_u:object_r:shell_exec_t:s0 /bin/bash
[root@rhel-7 ~]# ls -laZ /var/lib/containers/atomic/ce.0/rootfs/
drwxr-xr-x. root root system_u:object_r:root_t:s0      .
drwxr-xr-x. root root system_u:object_r:root_t:s0      ..
lrwxrwxrwx. root root unconfined_u:object_r:container_share_t:s0 bin -> usr/bin
dr-xr-xr-x. root root unconfined_u:object_r:container_share_t:s0 boot
drwxr-xr-x. root root unconfined_u:object_r:container_share_t:s0 dev
drwxr-xr-x. root root unconfined_u:object_r:container_share_t:s0 etc
<snip>

Comment 8 Gan Huang 2018-03-05 01:35:46 UTC
Issue is gone after removing "SELinuxContext=system_u:system_r:container_runtime_t:s0".

# ps -eZ |grep docker
system_u:system_r:container_runtime_t:s0 13244 ? 00:00:00 dockerd-current
system_u:system_r:container_runtime_t:s0 13343 ? 00:00:00 docker-lvm-plug
system_u:system_r:container_runtime_t:s0 13349 ? 00:00:00 docker-novolume
system_u:system_r:container_runtime_t:s0 13359 ? 00:00:00 docker-containe

# rpm -q atomic
atomic-1.22.1-1.gitd36c015.el7.x86_64

# rpm -q container-selinux
container-selinux-2.48-1.el7.noarch

# atomic images list
   REPOSITORY                                                       TAG    IMAGE ID       CREATED            VIRTUAL SIZE   TYPE      
>  registry.reg-aws.openshift.com:443/openshift3/container-engine   v3.9   abcc99b5ecf6   2018-03-04 20:19   128.66 MB      ostree    


Thanks for all the help.

Comment 9 Giuseppe Scrivano 2018-03-05 08:54:46 UTC
Dan, could we make it that bash can run again with the SELinuxContext=system_u:system_r:container_runtime_t:s0 from the Docker system container .service file?

bash is used in an ExecStartPre= script.

Comment 10 Daniel Walsh 2018-03-05 13:45:26 UTC
What AVC's are  you seeing. 

Is this RHEL7.5 or RHEL7.4.5?

Comment 11 Steve Milner 2018-03-05 14:42:18 UTC
Dan,

This was found on RHEL 7.5 with container-selinux-2.48-1.el7.noarch. Previous version (container-selinux-2.36-1.gitff95335.el7.noarch) does not have this issue.

Gan,

Can you post the AVC denial that occured?

Comment 12 Gan Huang 2018-03-06 01:51:56 UTC
Correct, it's RHEL 7.5.

type=AVC msg=audit(1520301012.997:1228): avc:  denied  { entrypoint } for  pid=13235 comm="(bash)" path="/usr/bin/bash" dev="dm-0" ino=25216889 scontext=system_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1520301012.997:1228): arch=c000003e syscall=59 success=no exit=-13 a0=5587376070e0 a1=5587376c4c10 a2=5587376c4a20 a3=fffff000 items=0 ppid=1 pid=13235 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="(bash)" exe="/usr/lib/systemd/systemd" subj=system_u:system_r:init_t:s0 key=(null)
type=PROCTITLE msg=audit(1520301012.997:1228): proctitle="(bash)"
type=SERVICE_START msg=audit(1520301013.005:1229): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=container-engine comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

Comment 13 Gan Huang 2018-03-06 01:53:16 UTC
# systemctl cat container-engine
# /etc/systemd/system/container-engine.service
[Unit]
Description=Container Engine service
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
SELinuxContext=system_u:system_r:container_runtime_t:s0
ExecStartPre=/bin/bash -c 'export -p > /run/docker-bash-env'
ExecStart=/usr/bin/runc --systemd-cgroup run 'container-engine'
ExecStop=/usr/bin/runc --systemd-cgroup kill 'container-engine'
KillMode=process
Restart=on-abnormal
WorkingDirectory=/var/lib/containers/atomic/container-engine.0
RuntimeDirectory=docker
LimitNOFILE=1048576
LimitNPROC=1048576
...skipping...
KillMode=process
Restart=on-abnormal
WorkingDirectory=/var/lib/containers/atomic/container-engine.0
RuntimeDirectory=docker
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/container-engine.service.d/custom.conf
# Ansible managed

[Service]
[Unit]
Wants=iptables.service
After=iptables.service
lines 14-32/32 (END)
KillMode=process
Restart=on-abnormal
WorkingDirectory=/var/lib/containers/atomic/container-engine.0
RuntimeDirectory=docker
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/container-engine.service.d/custom.conf
# Ansible managed

[Service]
[Unit]
Wants=iptables.service
After=iptables.service

Comment 19 Daniel Walsh 2018-03-07 06:04:04 UTC
container-selinux-2.50 adds the entrypoint that is missing, so that the service file can do the transition.

Comment 21 Gan Huang 2018-03-08 06:42:33 UTC
Issue still persists:

type=AVC msg=audit(1520491233.795:1258): avc:  denied  { entrypoint } for  pid=13325 comm="(bash)" path="/usr/bin/bash" dev="dm-0" ino=25216953 scontext=system_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1520491233.795:1258): arch=c000003e syscall=59 success=no exit=-13 a0=562403599450 a1=562403590b60 a2=562403590ab0 a3=766e652d68736162 items=0 ppid=1 pid=13325 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="(bash)" exe="/usr/lib/systemd/systemd" subj=system_u:system_r:init_t:s0 key=(null)
type=PROCTITLE msg=audit(1520491233.795:1258): proctitle="(bash)"
type=SERVICE_START msg=audit(1520491233.802:1259): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=container-engine comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

# rpm -q container-selinux
container-selinux-2.50-1.el7.noarch

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Comment 22 Daniel Walsh 2018-03-08 07:58:56 UTC
Building container-selinux-2.51-1.el7.noarch

Comment 29 Johnny Liu 2018-03-12 06:10:21 UTC
According to comment 26 and comment 27, move this bug to verified.

Comment 31 errata-xmlrpc 2018-04-11 00:03:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1073