Bug 1491566

Summary: [3.6] Checks during installation fails, due to /var being split out on different filesystems.
Product: OpenShift Container Platform Reporter: Kim Borup <kborup>
Component: InstallerAssignee: Luke Meyer <lmeyer>
Status: CLOSED ERRATA QA Contact: Wenkai Shi <weshi>
Severity: low Docs Contact:
Priority: low    
Version: 3.6.0CC: aivaras.laimikis, aos-bugs, jokerman, jrosenta, mmccomas, mrobson
Target Milestone: ---   
Target Release: 3.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: In verifying sufficient disk space available under /var, the disk_availability check only counted storage mounted directly at /var (or / if /var is not a separate filesystem). Consequence: Extra storage mounted below /var, for instance in /var/lib/docker, was not counted toward the required available storage, and thus the check could fail erroneously. Fix: Storage mounted below a filesystem is included in its total availability. Result: The check should accurately account for availability of storage.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-14 21:01:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kim Borup 2017-09-14 07:36:21 UTC
Description of problem:

customer have setup the filesystems to have /var /var/log /var/lib/origin /var/lib/docker in order to split out where to put data. 

During the installation the checks for storage will fail on /var as /var is only 10G but /var/lib/origin are 80GB and so is /var/lib/docker, we can abvoid this check with openshift_disable_check. however in future settings we should check on /var/lib/docker and /var/lib/origin as that is most likely the places to get filled up. 


How reproducible:

All the time

Steps to Reproduce:
1. Split out filesystems on /var with /var being below 40G
2. Start install 
3.

Actual results:

Installation fails with Storage requirement  not met even though we have enough storage in underlying mounts. 

Expected results:

Installation succeed 

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Luke Meyer 2017-09-14 19:08:01 UTC
I agree it would make sense to look at subtree mounts for data that OCP uses and adjust the requirement for /var accordingly. At the moment the documented requirements simply specify /var so that's what the check follows (https://docs.openshift.org/latest/install_config/install/prerequisites.html#system-requirements).

It shouldn't be too hard to just include filesystems underneath /var in the total calculation for /var. I think that would make the most sense here. It would also be a good idea to add "sanity" thresholds in the check for specific subtrees; for instance if a /var/lib/docker volume fills up it doesn't really matter that there's plenty of room in /var, docker will still fail.

Comment 3 Luke Meyer 2017-10-19 18:29:11 UTC
https://github.com/openshift/openshift-ansible/pull/5819 is to backport to 3.6

Comment 4 openshift-github-bot 2017-10-27 23:45:35 UTC
Commit pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/4fb35642a47550021fed3e455bedc9b6fdf2b9c9
disk_availability check: include submount storage

Fixes bug 1491566
https://bugzilla.redhat.com/show_bug.cgi?id=1491566

In order to determine how much storage is under a path, include any
mounts that are below it in addition to the path itself.

Comment 6 Wenkai Shi 2017-11-24 04:40:21 UTC
Change status since no build in errata system.

Comment 9 Wenkai Shi 2017-11-28 05:20:06 UTC
Change status since no build in errata system.

Comment 12 Wenkai Shi 2017-12-01 09:53:02 UTC
Waiting for build existing in Errata System to verify this.

Comment 13 Wenkai Shi 2017-12-05 03:22:00 UTC
Verified with version openshift-ansible-3.6.173.0.83-1.git.0.84c5eff.el7, create 
split filesystems under /var with more than 40G size, health check passed.

# df -h 
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root     10G  1.4G  8.7G  14% /
devtmpfs                 1.8G     0  1.8G   0% /dev
tmpfs                    1.8G     0  1.8G   0% /dev/shm
tmpfs                    1.8G   17M  1.8G   1% /run
tmpfs                    1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/xvda2               497M  159M  339M  32% /boot
/dev/mapper/rhel-docker   40G   33M   40G   1% /var/lib/docker
tmpfs                    361M     0  361M   0% /run/user/0

Comment 16 errata-xmlrpc 2017-12-14 21:01:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438

Comment 17 Luke Meyer 2018-01-15 17:51:15 UTC
*** Bug 1506265 has been marked as a duplicate of this bug. ***