Bug 1491566 - [3.6] Checks during installation fails, due to /var being split out on different filesystems.
Summary: [3.6] Checks during installation fails, due to /var being split out on differ...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.6.z
Assignee: Luke Meyer
QA Contact: Wenkai Shi
: 1506265 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2017-09-14 07:36 UTC by Kim Borup
Modified: 2018-01-15 17:51 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: In verifying sufficient disk space available under /var, the disk_availability check only counted storage mounted directly at /var (or / if /var is not a separate filesystem). Consequence: Extra storage mounted below /var, for instance in /var/lib/docker, was not counted toward the required available storage, and thus the check could fail erroneously. Fix: Storage mounted below a filesystem is included in its total availability. Result: The check should accurately account for availability of storage.
Clone Of:
Last Closed: 2017-12-14 21:01:55 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3438 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-15 01:58:11 UTC

Description Kim Borup 2017-09-14 07:36:21 UTC
Description of problem:

customer have setup the filesystems to have /var /var/log /var/lib/origin /var/lib/docker in order to split out where to put data. 

During the installation the checks for storage will fail on /var as /var is only 10G but /var/lib/origin are 80GB and so is /var/lib/docker, we can abvoid this check with openshift_disable_check. however in future settings we should check on /var/lib/docker and /var/lib/origin as that is most likely the places to get filled up. 

How reproducible:

All the time

Steps to Reproduce:
1. Split out filesystems on /var with /var being below 40G
2. Start install 

Actual results:

Installation fails with Storage requirement  not met even though we have enough storage in underlying mounts. 

Expected results:

Installation succeed 

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Luke Meyer 2017-09-14 19:08:01 UTC
I agree it would make sense to look at subtree mounts for data that OCP uses and adjust the requirement for /var accordingly. At the moment the documented requirements simply specify /var so that's what the check follows (https://docs.openshift.org/latest/install_config/install/prerequisites.html#system-requirements).

It shouldn't be too hard to just include filesystems underneath /var in the total calculation for /var. I think that would make the most sense here. It would also be a good idea to add "sanity" thresholds in the check for specific subtrees; for instance if a /var/lib/docker volume fills up it doesn't really matter that there's plenty of room in /var, docker will still fail.

Comment 3 Luke Meyer 2017-10-19 18:29:11 UTC
https://github.com/openshift/openshift-ansible/pull/5819 is to backport to 3.6

Comment 4 openshift-github-bot 2017-10-27 23:45:35 UTC
Commit pushed to master at https://github.com/openshift/openshift-ansible

disk_availability check: include submount storage

Fixes bug 1491566

In order to determine how much storage is under a path, include any
mounts that are below it in addition to the path itself.

Comment 6 Wenkai Shi 2017-11-24 04:40:21 UTC
Change status since no build in errata system.

Comment 9 Wenkai Shi 2017-11-28 05:20:06 UTC
Change status since no build in errata system.

Comment 12 Wenkai Shi 2017-12-01 09:53:02 UTC
Waiting for build existing in Errata System to verify this.

Comment 13 Wenkai Shi 2017-12-05 03:22:00 UTC
Verified with version openshift-ansible-, create 
split filesystems under /var with more than 40G size, health check passed.

# df -h 
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root     10G  1.4G  8.7G  14% /
devtmpfs                 1.8G     0  1.8G   0% /dev
tmpfs                    1.8G     0  1.8G   0% /dev/shm
tmpfs                    1.8G   17M  1.8G   1% /run
tmpfs                    1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/xvda2               497M  159M  339M  32% /boot
/dev/mapper/rhel-docker   40G   33M   40G   1% /var/lib/docker
tmpfs                    361M     0  361M   0% /run/user/0

Comment 16 errata-xmlrpc 2017-12-14 21:01:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Comment 17 Luke Meyer 2018-01-15 17:51:15 UTC
*** Bug 1506265 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.