Bug 1692513

This doesn't happen for all mounts; for example, I created a similar mount unit for `/var/lib/foobar` and it succeeded:

```
$ rpm-ostree status
State: idle
Warning: failed to query journal: Cannot assign requested address (os error 99)
AutomaticUpdates: disabled
Deployments:
● pivot://docker-registry-default.cloud.registry.upshift.redhat.com/redhat-coreos/ootpa@sha256:84b54f4578bd04aa4e6fae2a7052a3b6df0bf5405dfbba1db44d59d282f3ee2f
              CustomOrigin: Provisioned from oscontainer
                   Version: 410.8.20190325.0 (2019-03-25T18:27:58Z)

$ sudo cat /etc/systemd/system/var-lib-foobar.mount
[Mount]
What=/dev/xvdb1
Where=/var/lib/foobar
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target

$ sudo systemctl daemon-reload

$ sudo ls -la /var/lib/foobar
ls: cannot access '/var/lib/foobar': No such file or directory

$ sudo systemctl start var-lib-foobar.mount
$ sudo systemctl status var-lib-foobar.mount
● var-lib-foobar.mount - /var/lib/foobar
   Loaded: loaded (/etc/systemd/system/var-lib-foobar.mount; disabled; vendor preset: disabled)
   Active: active (mounted) since Tue 2019-03-26 14:41:42 UTC; 10s ago
    Where: /var/lib/foobar
     What: /dev/xvdb1
    Tasks: 0 (limit: 24828)
   Memory: 72.0K
   CGroup: /system.slice/var-lib-foobar.mount

Mar 26 14:41:42 ip-172-18-7-88 systemd[1]: Mounting /var/lib/foobar...
Mar 26 14:41:42 ip-172-18-7-88 systemd[1]: Mounted /var/lib/foobar.

$ sudo ls -la /var/lib/foobar
total 4
drwxr-xr-x.  2 root root    6 Mar 26 14:40 .
drwxr-xr-x. 27 root root 4096 Mar 26 14:41 ..
```

When I tested the failing scenario on a traditional RHEL 8 system, it succeeded for me.  I noticed that the version of `selinux-policy` on that host was older than what was on the RHCOS host:

traditional RHEL 8 - selinux-policy-3.14.1-51.el8.noarch
RHCOS - selinux-policy-3.14.1-61.el8.noarch

After upgrading the traditional RHEL 8 host, everything continued to work as expected.

There is something specific about the `/var/lib/containers` mount point on RHCOS, but it's not obvious to me what it is yet.

I found that on the traditional RHEL 8 host, the `/var/lib/containers` directory is labeled differently than on the RHCOS host.

traditional RHEL 8 (after the mount unit):
```
$ ls -laZ /var/lib/containers
total 4
drwxr-xr-x.  2 root root system_u:object_r:unlabeled_t:s0    6 Mar 26 10:54 .
drwxr-xr-x. 35 root root system_u:object_r:var_lib_t:s0   4096 Mar 26 11:57 ..
```

RHCOS (defaults):

```
$ sudo ls -laZ /var/lib/containers
total 4
drwx------.  3 root root system_u:object_r:container_var_lib_t:s0   21 Mar 26 16:06 .
drwxr-xr-x. 27 root root system_u:object_r:var_lib_t:s0           4096 Mar 26 16:06 ..
drwx------.  8 root root system_u:object_r:container_var_lib_t:s0  136 Mar 26 16:06 storage
```

On the RHCOS host, `/var/lib/containers` is owned by the `cri-o` package.

Perhaps there is an SELinux policy difference since the traditional RHEL 8 host does not have `container-selinux` or `cri-o` installed.

So the traditional RHEL 8 host is missing the `container_var_lib_t` (and associated SELinux types) by default:

```
$ sudo seinfo -x --type=container_var_lib_t

Types: 0

$ sudo sesearch --dontaudit -t container_var_lib_t
container_var_lib_t is not a valid type attribute

$ sudo semanage fcontext -l | grep container_var_lib_t

$ rpm -qa | grep selinux | sort
libselinux-2.8-6.el8.x86_64
libselinux-utils-2.8-6.el8.x86_64
python3-libselinux-2.8-6.el8.x86_64
rpm-plugin-selinux-4.14.2-9.el8.x86_64
selinux-policy-3.14.1-61.el8.noarch
selinux-policy-targeted-3.14.1-61.el8.noarch
```

When the `container-selinux` package is installed on the traditional RHEL 8 host, those types are now populated:

```
$ sudo seinfo -x --type container_var_lib_t                     
                                                                        
Types: 1                                                               

$ sudo sesearch --dontaudit -t container_var_lib_t
dontaudit abrt_helper_t non_security_file_type:blk_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:chr_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:fifo_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:lnk_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:sock_file { append getattr ioctl lock read write };
dontaudit abrt_t file_type:lnk_file read;                                    
dontaudit abrt_t file_type:sock_file getattr;                     
dontaudit apmd_t file_type:fifo_file getattr;             
...

$  sudo semanage fcontext -l | grep container_var_lib_t
/exports(/.*)?                                     all files          system_u:object_r:container_var_lib_t:s0
/var/lib/containers(/.*)?                          all files          system_u:object_r:container_var_lib_t:s0
/var/lib/docker(/.*)?                              all files          system_u:object_r:container_var_lib_t:s0
/var/lib/docker-latest(/.*)?                       all files          system_u:object_r:container_var_lib_t:s0
/var/lib/lxc(/.*)?                                 all files          system_u:object_r:container_var_lib_t:s0
/var/lib/lxd(/.*)?                                 all files          system_u:object_r:container_var_lib_t:s0
/var/lib/ocid(/.*)?                                all files          system_u:object_r:container_var_lib_t:s0

$ rpm -qa | grep selinux | sort
container-selinux-2.75-1.git99e2cfd.module+el8+2769+577ad176.noarch
libselinux-2.8-6.el8.x86_64
libselinux-utils-2.8-6.el8.x86_64
python3-libselinux-2.8-6.el8.x86_64
rpm-plugin-selinux-4.14.2-9.el8.x86_64
selinux-policy-3.14.1-61.el8.noarch
selinux-policy-targeted-3.14.1-61.el8.noarch
```

With this policy installed, the traditional RHEL 8 host will suffer the same failure:

```
$ ls -laZ /var/lib/containers                  
ls: cannot access '/var/lib/containers': No such file or directory         

$ cat /etc/systemd/system/var-lib-containers.mount 
[Mount]
What=/dev/vdb1
Where=/var/lib/containers
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target

$ sudo systemctl daemon-reload                                                    
$ sudo systemctl start var-lib-containers.mount            
Job for var-lib-containers.mount failed.                                            
See "systemctl status var-lib-containers.mount" and "journalctl -xe" for details.            

Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Failed to check directory /var/lib/containers: No such file or directory
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: Mounting /var/lib/containers...
Mar 26 12:52:13 micah-rhel8-vm0326c mount[1149]: mount: /var/lib/containers: mount point does not exist.
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Mount process exited, code=exited status=32
Mar 26 12:52:13 micah-rhel8-vm0326c sudo[1145]: pam_unix(sudo:session): session closed for user root
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Failed with result 'exit-code'.
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: Failed to mount /var/lib/containers.
Mar 26 12:52:16 micah-rhel8-vm0326c dbus-daemon[643]: [system] Activating service name='org.fedoraproject.Setroubleshootd' requested by ':1.29' (uid=0 pid=611 comm="/usr/sbin/sedispatch " label="system_u:system_r:auditd_t:s0") (using servicehelper)
Mar 26 12:52:16 micah-rhel8-vm0326c dbus-daemon[643]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Mar 26 12:52:17 micah-rhel8-vm0326c setroubleshoot[1152]: SELinux is preventing systemd from create access on the directory containers. For complete SELinux messages run: sealert -l 85ca15e5-4c63-44b0-989a-e7535189c316
Mar 26 12:52:17 micah-rhel8-vm0326c platform-python[1152]: SELinux is preventing systemd from create access on the directory containers.

                                                           *****  Plugin catchall (100. confidence) suggests   **************************

                                                           If you believe that systemd should be allowed create access on the containers directory by default.
                                                           Then you should report this as a bug.
                                                           You can generate a local policy module to allow this access.
                                                           Do
                                                           allow this access for now by executing:
                                                           # ausearch -c 'systemd' --raw | audit2allow -M my-systemd
                                                           # semodule -X 300 -i my-systemd.pp

```


So I think this is ultimately an `selinux-policy` or `container-selinux` issue.  @bbreard WDYT?

Brilliant analysis! I guess I'm leaning toward container-selinux being where this should land. 

This is blocking our means to recreate the docker-storage-setup utility via ignition.

FWIW, mounting manually is still possible:

```
$ sudo ls -laZ /var/lib/containers
ls: cannot access '/var/lib/containers': No such file or directory
$ sudo mkdir -p /var/lib/containers
$ sudo ls -laZ /var/lib/containers
total 4
drwxr-xr-x.  2 root root unconfined_u:object_r:container_var_lib_t:s0    6 Mar 26 14:18 .
drwxr-xr-x. 35 root root system_u:object_r:var_lib_t:s0               4096 Mar 26 14:18 ..
$ sudo mount /dev/vdb1 /var/lib/containers
$ findmnt /var/lib/containers
TARGET              SOURCE    FSTYPE OPTIONS
/var/lib/containers /dev/vdb1 xfs    rw,relatime,seclabel,attr2,inode64,noquota
```

That is correct; I got the same result. I'm not sure how to create that directory first via ignition. I'm sure it's possible, but not obvious or user friendly.

We can fix this in container-selinux, but it is really a fix in selinux-policy.

Fixed in container-selinux-2.94

Test with podman-1.4.2-5.module+el8.1.0+4240+893c1ab8.x86_64 and seems it works as expect:
# systemctl status var-lib-containers.mount
● var-lib-containers.mount - /var/lib/containers
   Loaded: loaded (/etc/systemd/system/var-lib-containers.mount; disabled; vendor preset: disabled)
   Active: active (mounted) since Sun 2019-09-29 02:30:24 EDT; 11s ago
    Where: /var/lib/containers
     What: /dev/vdb1
    Tasks: 0 (limit: 5057)
   Memory: 12.0K
   CGroup: /system.slice/var-lib-containers.mount

Sep 29 02:30:24 ypu-rhel8.1 systemd[1]: Mounting /var/lib/containers...
Sep 29 02:30:24 ypu-rhel8.1 systemd[1]: Mounted /var/lib/containers.
[root@ypu-rhel8 ~]# findmnt /var/lib/containers   
TARGET              SOURCE    FSTYPE OPTIONS
/var/lib/containers /dev/vdb1 xfs    rw,relatime,seclabel,attr2,inode64,noquota
[root@ypu-rhel8 ~]# ls -laZ /var/lib/containers              
total 4
drwxr-xr-x.  2 root root system_u:object_r:unlabeled_t:s0    6 Sep 29 00:04 .
drwxr-xr-x. 37 root root system_u:object_r:var_lib_t:s0   4096 Sep 29 00:10 ..

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3403