Bug 1564840 - [3.10] No directory /var/lib/containers/overlay2
Summary: [3.10] No directory /var/lib/containers/overlay2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.10.0
Assignee: Scott Dodson
QA Contact: Gan Huang
URL:
Whiteboard:
Depends On:
Blocks: 1564949
TreeView+ depends on / blocked
 
Reported: 2018-04-08 06:51 UTC by Gan Huang
Modified: 2018-07-30 19:12 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The installer improperly tried to set selinux context on a path that may not exist. This task was meant to workaround a problem in CRI-O that no longer exists and as such that task has been removed.
Clone Of:
: 1564949 (view as bug list)
Environment:
Last Closed: 2018-07-30 19:12:20 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 None None None 2018-07-30 19:12:47 UTC

Description Gan Huang 2018-04-08 06:51:12 UTC
Description of problem:
No directory /var/lib/containers/overlay2 while fixing selinux permissions.

Version-Release number of the following components:
openshift-ansible-3.10.0-0.16.0.git.0.8925606.el7.noarch.rpm

# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-693.11.1.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 1
Total Memory: 3.456 GiB
Name: qe-ghuang-master-etcd-1
Docker Root Dir: /var/lib/containers/docker
Debug Mode (client): false
Debug Mode (server): false

How reproducible:
always

Steps to Reproduce:
1. Trigger installation with rpm cri-o enabled
# cat inventory
<--snip-->
openshift_crio_use_rpm=true
<--snip-->


Actual results:
TASK [container_runtime : Fixup SELinux permissions for docker] ****************
Saturday 07 April 2018  23:12:27 -0400 (0:00:01.585)       0:02:00.428 ******** 
fatal: [qe-ghuang-master-etcd-1.0407-fvf.qe.rhcloud.com]: FAILED! => {"changed": true, "cmd": "semanage fcontext -a -e /var/lib/docker/overlay2 /var/lib/containers/overlay2\n restorecon -R -v /var/lib/containers/overlay2", "delta": "0:00:00.373619", "end": "2018-04-07 23:12:30.368412", "failed": true, "msg": "non-zero return code", "rc": 255, "start": "2018-04-07 23:12:29.994793", "stderr": "restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory", "stderr_lines": ["restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory"], "stdout": "", "stdout_lines": []}
fatal: [qe-ghuang-node-registry-router-1.0407-fvf.qe.rhcloud.com]: FAILED! => {"changed": true, "cmd": "semanage fcontext -a -e /var/lib/docker/overlay2 /var/lib/containers/overlay2\n restorecon -R -v /var/lib/containers/overlay2", "delta": "0:00:00.398757", "end": "2018-04-07 23:12:30.415676", "failed": true, "msg": "non-zero return code", "rc": 255, "start": "2018-04-07 23:12:30.016919", "stderr": "restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory", "stderr_lines": ["restorecon:  lstat(/var/lib/containers/overlay2) failed:  No such file or directory"], "stdout": "", "stdout_lines": []}

Expected results:

Additional info:
# ll /var/lib/containers/overlay2
ls: cannot access /var/lib/containers/overlay2: No such file or directory

Comment 2 Scott Dodson 2018-04-13 02:38:33 UTC
Mrunal,

You added this[1][2], under which conditions would we expect /var/lib/containers/overlay2 to exist? Should we just check for the existence and not apply those changes if it doesn't exist?

1 - https://github.com/openshift/openshift-ansible/pull/7553
2 - https://github.com/openshift/openshift-ansible/pull/7466

Comment 3 Mrunal Patel 2018-04-13 19:58:06 UTC
Scott,
Yes, we can check for existence and apply only if the directory is present.

Comment 5 Michael Gugino 2018-04-17 19:52:40 UTC
I think it's the overlay2 directory in question, which should be context type 'container_share_t'

However, the overlay2 directory is not created via RPM, so whatever creates it must be responsible for setting the context.

It's not clear what we should do for these cases.

Comment 6 Gan Huang 2018-04-19 03:23:01 UTC
Fix is already in master branch.

Tested with master branch, succeeded to spin up cluster with cri-o.

Removing TestBlocker.

Comment 8 Gan Huang 2018-04-26 06:20:01 UTC
Verified in openshift-ansible-3.10.0-0.29.0.git.0.8973ee4.el7.noarch.rpm

Comment 10 errata-xmlrpc 2018-07-30 19:12:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816


Note You need to log in before you can comment on or make changes to this bug.