Bug 1564949

Summary: [3.9] No directory /var/lib/containers/overlay2
Product: OpenShift Container Platform Reporter: Gan Huang <ghuang>
Component: InstallerAssignee: Scott Dodson <sdodson>
Status: CLOSED ERRATA QA Contact: Gan Huang <ghuang>
Severity: high Docs Contact:
Priority: high    
Version: 3.9.0CC: aos-bugs, jialiu, jokerman, mmccomas, sdodson, wmeng, wsun
Target Milestone: ---Keywords: TestBlocker
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The installer improperly tried to set selinux context on a path that may not exist. This task was meant to workaround a problem in CRI-O that no longer exists and as such that task has been removed.
Story Points: ---
Clone Of: 1564840 Environment:
Last Closed: 2018-05-17 06:43:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1564840    
Bug Blocks: 1553186    

Description Gan Huang 2018-04-09 02:33:44 UTC
+++ This bug was initially created as a clone of Bug #1564840 +++

Description of problem:
No directory /var/lib/containers/overlay2 while fixing selinux permissions.

Version-Release number of the following components:
openshift-ansible-3.9.19-1.git.0.34f4090.el7.noarch.rpm

# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-693.11.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 1
Total Memory: 3.456 GiB
Name: qe-ghuang-master-etcd-1
Docker Root Dir: /var/lib/containers/docker
Debug Mode (client): false
Debug Mode (server): false

How reproducible:
always

Steps to Reproduce:
1. Trigger installation with rpm cri-o enabled
# cat inventory
<--snip-->
openshift_crio_use_rpm=true
<--snip-->


Actual results:
TASK [container_runtime : Fixup SELinux permissions for docker] ****************
Saturday 07 April 2018  23:12:27 -0400 (0:00:01.585)       0:02:00.428 ******** 
fatal: [qe-ghuang-master-etcd-1.0407-fvf.qe.rhcloud.com]: FAILED! => {"changed": true, "cmd": "semanage fcontext -a -e /var/lib/docker/overlay2 /var/lib/containers/overlay2\n restorecon -R -v /var/lib/containers/overlay2", "delta": "0:00:00.373619", "end": "2018-04-07 23:12:30.368412", "failed": true, "msg": "non-zero return code", "rc": 255, "start": "2018-04-07 23:12:29.994793", "stderr": "restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory", "stderr_lines": ["restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory"], "stdout": "", "stdout_lines": []}
fatal: [qe-ghuang-node-registry-router-1.0407-fvf.qe.rhcloud.com]: FAILED! => {"changed": true, "cmd": "semanage fcontext -a -e /var/lib/docker/overlay2 /var/lib/containers/overlay2\n restorecon -R -v /var/lib/containers/overlay2", "delta": "0:00:00.398757", "end": "2018-04-07 23:12:30.415676", "failed": true, "msg": "non-zero return code", "rc": 255, "start": "2018-04-07 23:12:30.016919", "stderr": "restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory", "stderr_lines": ["restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory"], "stdout": "", "stdout_lines": []}

Expected results:

Additional info:
# ll /var/lib/containers/overlay2
ls: cannot access /var/lib/containers/overlay2: No such file or directory

Comment 1 Gan Huang 2018-04-12 02:16:30 UTC
rpm cri-o test is blocked by this issue.

Comment 2 Scott Dodson 2018-04-17 19:44:52 UTC
I think this should be considered a blocker for the 3.9.z release, so I've produced a build with the fix in it so that we can unblock QE.

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=674444

At QE's discretion, I'd like to update the errata with this build after we've verified that this fixes CRI-O RPM installs. Note, that build is not yet in puddles until we decide that it should be in.

Comment 3 Gan Huang 2018-04-18 08:07:08 UTC
Installation is completed successfully in openshift-ansible-3.9.22-1.git.7.92620c6.el7.noarch.rpm. Also the S2I build succeeded.

Comment 4 Gan Huang 2018-04-18 10:04:33 UTC
> At QE's discretion, I'd like to update the errata with this build after
> we've verified that this fixes CRI-O RPM installs. Note, that build is not
> yet in puddles until we decide that it should be in.

Thanks, go head.

Comment 6 Gan Huang 2018-04-20 03:31:20 UTC
Fix is in openshift-ansible-3.9.24-1.git.0.d0289ea.el7

Per comment 3, moving to verified.

Comment 9 errata-xmlrpc 2018-05-17 06:43:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1566