RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1692513 - unable to mount disk at `/var/lib/containers` via `systemd` unit when `container-selinux` policy installed
Summary: unable to mount disk at `/var/lib/containers` via `systemd` unit when `contai...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: container-selinux
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1693806
TreeView+ depends on / blocked
 
Reported: 2019-03-25 18:47 UTC by Ben Breard
Modified: 2020-11-14 11:35 UTC (History)
14 users (show)

Fixed In Version: container-selinux-2.94
Doc Type: Bug Fix
Doc Text:
Systemd needs to create the container directories with the correct labels.
Clone Of:
: 1693806 1695669 (view as bug list)
Environment:
Last Closed: 2019-11-05 21:01:58 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:3403 0 None None None 2019-11-05 21:02:34 UTC

Description Ben Breard 2019-03-25 18:47:14 UTC 
Description of problem:

SELinux is blocking the "Where" line of mount units from creating mount points

Version-Release number of selected component (if applicable):

$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="410.8.20190322.0"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION_ID="4.1"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 410.8.20190322.0"
ANSI_COLOR="0;31"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.1"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.1"
OSTREE_VERSION=410.8.20190322.0

(also present in Fedora CoreOS 3-22 build from F29)

How reproducible:

100%

Steps to Reproduce:
1. add a second disk and create a partition
2. Create the following mount unit that points to the correct disk:

$ systemctl cat var-lib-containers.mount 
# /etc/systemd/system/var-lib-containers.mount
[Mount]
What=/dev/sdb1
Where=/var/lib/containers
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target


3. systemctl daemon-reload ; systemctl start var-lib-containers.mount


Actual results:

The mount fails due to the mount point not being present. (but systemd should be able to create it) 

Expected results:

/var/lib/containers should be created, but SELInux is blocking it. setenforce=0 gets around this problem.

Additional info:
Mar 25 18:40:13 host.local systemd[1]: var-lib-containers.mount: Failed to check directory /var/lib/containers: No such file or directory
Mar 25 18:40:13 host.local systemd[1]: Mounting /var/lib/containers...
Mar 25 18:40:13 host.local kernel: audit: type=1400 audit(1553539213.983:5): avc:  denied  { create } for  pid=1 comm="systemd" name="containers" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
Mar 25 18:40:13 host.local mount[1536]: mount: /var/lib/containers: mount point does not exist.
Mar 25 18:40:14 host.local systemd[1]: var-lib-containers.mount: Mount process exited, code=exited status=32
Mar 25 18:40:14 host.local systemd[1]: var-lib-containers.mount: Failed with result 'exit-code'.
Mar 25 18:40:14 host.local systemd[1]: Failed to mount /var/lib/containers.
Comment 2 Micah Abbott 2019-03-26 15:11:10 UTC 
This doesn't happen for all mounts; for example, I created a similar mount unit for `/var/lib/foobar` and it succeeded:

```
$ rpm-ostree status
State: idle
Warning: failed to query journal: Cannot assign requested address (os error 99)
AutomaticUpdates: disabled
Deployments:
● pivot://docker-registry-default.cloud.registry.upshift.redhat.com/redhat-coreos/ootpa@sha256:84b54f4578bd04aa4e6fae2a7052a3b6df0bf5405dfbba1db44d59d282f3ee2f
              CustomOrigin: Provisioned from oscontainer
                   Version: 410.8.20190325.0 (2019-03-25T18:27:58Z)

$ sudo cat /etc/systemd/system/var-lib-foobar.mount
[Mount]
What=/dev/xvdb1
Where=/var/lib/foobar
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target

$ sudo systemctl daemon-reload

$ sudo ls -la /var/lib/foobar
ls: cannot access '/var/lib/foobar': No such file or directory

$ sudo systemctl start var-lib-foobar.mount
$ sudo systemctl status var-lib-foobar.mount
● var-lib-foobar.mount - /var/lib/foobar
   Loaded: loaded (/etc/systemd/system/var-lib-foobar.mount; disabled; vendor preset: disabled)
   Active: active (mounted) since Tue 2019-03-26 14:41:42 UTC; 10s ago
    Where: /var/lib/foobar
     What: /dev/xvdb1
    Tasks: 0 (limit: 24828)
   Memory: 72.0K
   CGroup: /system.slice/var-lib-foobar.mount

Mar 26 14:41:42 ip-172-18-7-88 systemd[1]: Mounting /var/lib/foobar...
Mar 26 14:41:42 ip-172-18-7-88 systemd[1]: Mounted /var/lib/foobar.

$ sudo ls -la /var/lib/foobar
total 4
drwxr-xr-x.  2 root root    6 Mar 26 14:40 .
drwxr-xr-x. 27 root root 4096 Mar 26 14:41 ..
```

When I tested the failing scenario on a traditional RHEL 8 system, it succeeded for me.  I noticed that the version of `selinux-policy` on that host was older than what was on the RHCOS host:

traditional RHEL 8 - selinux-policy-3.14.1-51.el8.noarch
RHCOS - selinux-policy-3.14.1-61.el8.noarch

After upgrading the traditional RHEL 8 host, everything continued to work as expected.

There is something specific about the `/var/lib/containers` mount point on RHCOS, but it's not obvious to me what it is yet.
Comment 3 Micah Abbott 2019-03-26 16:23:13 UTC 
I found that on the traditional RHEL 8 host, the `/var/lib/containers` directory is labeled differently than on the RHCOS host.

traditional RHEL 8 (after the mount unit):
```
$ ls -laZ /var/lib/containers
total 4
drwxr-xr-x.  2 root root system_u:object_r:unlabeled_t:s0    6 Mar 26 10:54 .
drwxr-xr-x. 35 root root system_u:object_r:var_lib_t:s0   4096 Mar 26 11:57 ..
```

RHCOS (defaults):

```
$ sudo ls -laZ /var/lib/containers
total 4
drwx------.  3 root root system_u:object_r:container_var_lib_t:s0   21 Mar 26 16:06 .
drwxr-xr-x. 27 root root system_u:object_r:var_lib_t:s0           4096 Mar 26 16:06 ..
drwx------.  8 root root system_u:object_r:container_var_lib_t:s0  136 Mar 26 16:06 storage
```

On the RHCOS host, `/var/lib/containers` is owned by the `cri-o` package.

Perhaps there is an SELinux policy difference since the traditional RHEL 8 host does not have `container-selinux` or `cri-o` installed.
Comment 4 Micah Abbott 2019-03-26 16:54:57 UTC 
So the traditional RHEL 8 host is missing the `container_var_lib_t` (and associated SELinux types) by default:

```
$ sudo seinfo -x --type=container_var_lib_t

Types: 0

$ sudo sesearch --dontaudit -t container_var_lib_t
container_var_lib_t is not a valid type attribute

$ sudo semanage fcontext -l | grep container_var_lib_t

$ rpm -qa | grep selinux | sort
libselinux-2.8-6.el8.x86_64
libselinux-utils-2.8-6.el8.x86_64
python3-libselinux-2.8-6.el8.x86_64
rpm-plugin-selinux-4.14.2-9.el8.x86_64
selinux-policy-3.14.1-61.el8.noarch
selinux-policy-targeted-3.14.1-61.el8.noarch
```

When the `container-selinux` package is installed on the traditional RHEL 8 host, those types are now populated:

```
$ sudo seinfo -x --type container_var_lib_t                     
                                                                        
Types: 1                                                               

$ sudo sesearch --dontaudit -t container_var_lib_t
dontaudit abrt_helper_t non_security_file_type:blk_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:chr_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:fifo_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:lnk_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:sock_file { append getattr ioctl lock read write };
dontaudit abrt_t file_type:lnk_file read;                                    
dontaudit abrt_t file_type:sock_file getattr;                     
dontaudit apmd_t file_type:fifo_file getattr;             
...

$  sudo semanage fcontext -l | grep container_var_lib_t
/exports(/.*)?                                     all files          system_u:object_r:container_var_lib_t:s0
/var/lib/containers(/.*)?                          all files          system_u:object_r:container_var_lib_t:s0
/var/lib/docker(/.*)?                              all files          system_u:object_r:container_var_lib_t:s0
/var/lib/docker-latest(/.*)?                       all files          system_u:object_r:container_var_lib_t:s0
/var/lib/lxc(/.*)?                                 all files          system_u:object_r:container_var_lib_t:s0
/var/lib/lxd(/.*)?                                 all files          system_u:object_r:container_var_lib_t:s0
/var/lib/ocid(/.*)?                                all files          system_u:object_r:container_var_lib_t:s0

$ rpm -qa | grep selinux | sort
container-selinux-2.75-1.git99e2cfd.module+el8+2769+577ad176.noarch
libselinux-2.8-6.el8.x86_64
libselinux-utils-2.8-6.el8.x86_64
python3-libselinux-2.8-6.el8.x86_64
rpm-plugin-selinux-4.14.2-9.el8.x86_64
selinux-policy-3.14.1-61.el8.noarch
selinux-policy-targeted-3.14.1-61.el8.noarch
```

With this policy installed, the traditional RHEL 8 host will suffer the same failure:

```
$ ls -laZ /var/lib/containers                  
ls: cannot access '/var/lib/containers': No such file or directory         

$ cat /etc/systemd/system/var-lib-containers.mount 
[Mount]
What=/dev/vdb1
Where=/var/lib/containers
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target

$ sudo systemctl daemon-reload                                                    
$ sudo systemctl start var-lib-containers.mount            
Job for var-lib-containers.mount failed.                                            
See "systemctl status var-lib-containers.mount" and "journalctl -xe" for details.            

Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Failed to check directory /var/lib/containers: No such file or directory
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: Mounting /var/lib/containers...
Mar 26 12:52:13 micah-rhel8-vm0326c mount[1149]: mount: /var/lib/containers: mount point does not exist.
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Mount process exited, code=exited status=32
Mar 26 12:52:13 micah-rhel8-vm0326c sudo[1145]: pam_unix(sudo:session): session closed for user root
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Failed with result 'exit-code'.
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: Failed to mount /var/lib/containers.
Mar 26 12:52:16 micah-rhel8-vm0326c dbus-daemon[643]: [system] Activating service name='org.fedoraproject.Setroubleshootd' requested by ':1.29' (uid=0 pid=611 comm="/usr/sbin/sedispatch " label="system_u:system_r:auditd_t:s0") (using servicehelper)
Mar 26 12:52:16 micah-rhel8-vm0326c dbus-daemon[643]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Mar 26 12:52:17 micah-rhel8-vm0326c setroubleshoot[1152]: SELinux is preventing systemd from create access on the directory containers. For complete SELinux messages run: sealert -l 85ca15e5-4c63-44b0-989a-e7535189c316
Mar 26 12:52:17 micah-rhel8-vm0326c platform-python[1152]: SELinux is preventing systemd from create access on the directory containers.

                                                           *****  Plugin catchall (100. confidence) suggests   **************************

                                                           If you believe that systemd should be allowed create access on the containers directory by default.
                                                           Then you should report this as a bug.
                                                           You can generate a local policy module to allow this access.
                                                           Do
                                                           allow this access for now by executing:
                                                           # ausearch -c 'systemd' --raw | audit2allow -M my-systemd
                                                           # semodule -X 300 -i my-systemd.pp

```


So I think this is ultimately an `selinux-policy` or `container-selinux` issue.  @bbreard WDYT?
Comment 5 Ben Breard 2019-03-26 17:03:14 UTC 
Brilliant analysis! I guess I'm leaning toward container-selinux being where this should land. 

This is blocking our means to recreate the docker-storage-setup utility via ignition.
Comment 6 Micah Abbott 2019-03-26 18:20:54 UTC 
FWIW, mounting manually is still possible:

```
$ sudo ls -laZ /var/lib/containers
ls: cannot access '/var/lib/containers': No such file or directory
$ sudo mkdir -p /var/lib/containers
$ sudo ls -laZ /var/lib/containers
total 4
drwxr-xr-x.  2 root root unconfined_u:object_r:container_var_lib_t:s0    6 Mar 26 14:18 .
drwxr-xr-x. 35 root root system_u:object_r:var_lib_t:s0               4096 Mar 26 14:18 ..
$ sudo mount /dev/vdb1 /var/lib/containers
$ findmnt /var/lib/containers
TARGET              SOURCE    FSTYPE OPTIONS
/var/lib/containers /dev/vdb1 xfs    rw,relatime,seclabel,attr2,inode64,noquota
```
Comment 7 Ben Breard 2019-03-26 18:23:58 UTC 
That is correct; I got the same result. I'm not sure how to create that directory first via ignition. I'm sure it's possible, but not obvious or user friendly.
Comment 9 Daniel Walsh 2019-03-28 11:50:05 UTC 
We can fix this in container-selinux, but it is really a fix in selinux-policy.

Fixed in container-selinux-2.94
Comment 16 Joy Pu 2019-09-29 06:37:22 UTC 
Test with podman-1.4.2-5.module+el8.1.0+4240+893c1ab8.x86_64 and seems it works as expect:
# systemctl status var-lib-containers.mount
● var-lib-containers.mount - /var/lib/containers
   Loaded: loaded (/etc/systemd/system/var-lib-containers.mount; disabled; vendor preset: disabled)
   Active: active (mounted) since Sun 2019-09-29 02:30:24 EDT; 11s ago
    Where: /var/lib/containers
     What: /dev/vdb1
    Tasks: 0 (limit: 5057)
   Memory: 12.0K
   CGroup: /system.slice/var-lib-containers.mount

Sep 29 02:30:24 ypu-rhel8.1 systemd[1]: Mounting /var/lib/containers...
Sep 29 02:30:24 ypu-rhel8.1 systemd[1]: Mounted /var/lib/containers.
[root@ypu-rhel8 ~]# findmnt /var/lib/containers   
TARGET              SOURCE    FSTYPE OPTIONS
/var/lib/containers /dev/vdb1 xfs    rw,relatime,seclabel,attr2,inode64,noquota
[root@ypu-rhel8 ~]# ls -laZ /var/lib/containers              
total 4
drwxr-xr-x.  2 root root system_u:object_r:unlabeled_t:s0    6 Sep 29 00:04 .
drwxr-xr-x. 37 root root system_u:object_r:var_lib_t:s0   4096 Sep 29 00:10 ..
Comment 18 errata-xmlrpc 2019-11-05 21:01:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3403
Note You need to log in before you can comment on or make changes to this bug.