Bug 1265811 - oo-accept-node reports a quota failures when a loop device is used.
Summary: oo-accept-node reports a quota failures when a loop device is used.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 2.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Miciah Dashiel Butler Masters
QA Contact: Chao Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-23 20:11 UTC by Eric Rich
Modified: 2019-08-15 05:30 UTC (History)
7 users (show)

Fixed In Version: openshift-origin-node-util-1.38.6.2-1.el6op
Doc Type: Bug Fix
Doc Text:
Cause: In order to check that filesystem quotas were enabled, the oo-accept-node tool used the output of the df command to determine the device and mountpoint of the gear base directory. oo-accept-node then specified this mountpoint to the repquota command and expected to see the device in repquota's output. However, if the gear base directory is a loop mount, df prints the loop image filename whereas repquota prints the loop device. Thus oo-accept-node was looking for the wrong string in the repquota output. Consequence: If the gear base directory was a loop mount, oo-accept-node printed a spurious error indicating that quotas were disabled for the filesystem even if they were enabled. Fix: oo-accept-node now uses the mount command to determine whether the filesystem is a loop mount and, if it is, oo-accept-node now parses the filesystem options to determine the loop device for the filesystem. In addition, because using a loop mount degrades performance, oo-accept-node will print a warning if the gear base directory is a loop mount. Result: oo-accept-node now will print an error only if quotas really are disabled on the filesystem, and oo-accept-node will print a warning if the filesystem is a loop mount.
Clone Of:
Environment:
Last Closed: 2016-03-22 16:51:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:0489 0 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 2.2.9 security, bug fix, and enhancement update 2016-03-22 20:49:04 UTC

Description Eric Rich 2015-09-23 20:11:43 UTC
Description of problem:

When running [0] you fail to properly get the loop device path, and as such you get the following string (output):  

    [root@node2 openshift]# /sbin/quotaon -u -p /opt/openshift/test.fs
    quotaon: Specified path /opt/openshift/node.ext4 is not directory nor device.
    quotaon: No correct mountpoint specified.

The information used to populate, this output comes from [1] and more or less grabs '/opt/openshift/test.fs' for the oo_mount,  However this is not the mount device. The actual device is /dev/loop0. In short grabbing the output shown above results in a "string" miss match and a false positive reporting, from oo-accept-node. 

You can see from other commands (using the right device) that the correct string is reported. 

    [root@node2 openshift]# /sbin/quotaon -u -p /var/lib/openshift/
    user quota on /var/lib/openshift (/dev/loop0) is on

Again the issue here seems to be with using the command (below) to get the device (for loop back devices). 

    [root@node2 openshift]# /bin/df -P /var/lib/openshift/ | /usr/bin/tail -1
    /opt/openshift/test.fs      999112  2836    943848       1% /var/lib/openshift

Version-Release number of selected component (if applicable): 2.0 -> 2.2.6
How reproducible: 100% 

Steps to Reproduce:

# truncate -s 1G /opt/openshift/test.fs
# mkfs -t ext4 /opt/openshift/test.fs
# mount -oloop,usrquota /opt/openshift/test.fs /var/lib/openshift
# oo-accept-node

Actual results:
    FAIL: quotas are not enabled on /var/lib/openshift (/opt/openshift/node.ext4)

Expected results:
    oo-accept-node should pass   

Additional info:

[0] https://github.com/openshift/origin-server/blob/master/node-util/sbin/oo-accept-node#L549-L552
[1] https://github.com/openshift/origin-server/blob/master/node-util/sbin/oo-accept-node#L540-L552

Comment 4 Miciah Dashiel Butler Masters 2016-02-08 22:58:11 UTC
PR: https://github.com/openshift/origin-server/pull/6350

Comment 5 openshift-github-bot 2016-02-09 17:19:33 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/afe315643abeb4463ca57e043e1a8037f99b111a
oo-accept-node: check_quotas: cope with loop mount

Modify oo-accept-node's quota check to check whether the mount is using
a loop device mount and, if so, check the quota on the loop device.  In
addition, print a warning that using a loop mount may impact performance
negatively.

This commit fixes bug 1265811.

https://bugzilla.redhat.com/show_bug.cgi?id=1265811

Comment 8 Rory Thrasher 2016-02-22 22:13:28 UTC
QE,

Can we verify that oo-accept-node will now pass based on the original reproduction steps?

# truncate -s 1G /opt/openshift/test.fs
# mkfs -t ext4 /opt/openshift/test.fs
# mount -oloop,usrquota /opt/openshift/test.fs /var/lib/openshift
# oo-accept-node

Puddle: http://etherpad.corp.redhat.com/puddle-2-2-2016-02-19


Thank you

Comment 9 Anping Li 2016-02-29 09:48:50 UTC
Reported as 'quotas are not enabled' by oo-accept-node. but step 2)&4) should be quota is turn on.

1) [root@node1 ~]# mount
/opt/openshift/test.fs on /var/lib/openshift type ext4 (rw,usrquota,loop=/dev/loop0)


2) [root@node1 ~]# /sbin/quotaon -u -p /var/lib/openshift/
user quota on /var/lib/openshift (/dev/loop0) is on

3) [root@node1 ~]# oo-accept-node
WARNING: /var/lib/openshift is a loop mount (loop device: /dev/loop0)
).  Using a loop mount may reduce performance.
FAIL: quotas are not enabled on /var/lib/openshift (/dev/loop0)
)
1 ERRORS


4) [root@node1 ~]# /sbin/quotaon -u -p /var/lib/openshift/
user quota on /var/lib/openshift (/dev/loop0) is on

5) [root@node1 ~]# rpm -qf /usr/sbin/oo-accept-node
openshift-origin-node-util-1.38.6.1-1.el6op.noarch

Comment 10 Anping Li 2016-02-29 11:05:44 UTC
If the mount show loop0 as the suffix as 2), we will hit this issue.

1) /opt/openshift/test.fs on /var/lib/openshift type ext4 (rw,loop=/dev/loop0,usrquota)
2) /opt/openshift/test.fs on /var/lib/openshift type ext4 (rw,usrquota,loop=/dev/loop0)

I add one line in /etc/fstab, and run "mount -o remount /var/lib/openshift", and got the message as 2).

# tail -1 /etc/fstab 
/opt/openshift/test.fs /var/lib/openshift   ext4    defaults,usrquota     1 0

Comment 11 openshift-github-bot 2016-03-01 17:01:49 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/896aba36d358cda1bd7b1a5d229704d995c72d4d
oo-accept-node: check-quotas: fix mnt opts parsing

Improve the parsing of mount options from the `mount` command's output by
properly splitting on newlines and then removing the parentheses around the
mount options.

Before this commit, the pattern match `/loop=([^,]+)/` on the mount options
was capturing the right parenthesis and newline from the `mount` command's
output if the loop option was the last mount option.

This commit is related to bug 1265811.

https://bugzilla.redhat.com/show_bug.cgi?id=1265811

Comment 12 Rory Thrasher 2016-03-03 22:39:02 UTC
QE,

This should be fixed now.  Can we verify with the latest puddle? Thank you.

Puddle: http://etherpad.corp.redhat.com/puddle-2-2-2016-02-19

Comment 13 Johnny Liu 2016-03-04 03:35:29 UTC
Verified this bug with, and PASS.


# truncate -s 1G /opt/openshift/test.fs
# mkfs -t ext4 /opt/openshift/test.fs
# mount -oloop,usrquota /opt/openshift/test.fs /var/lib/openshift
# mount |grep openshift
/opt/openshift/test.fs on /var/lib/openshift type ext4 (rw,loop=/dev/loop1,usrquota)
# quotaon /var/lib/openshift/
# /sbin/quotaon -u -p /var/lib/openshift/
user quota on /var/lib/openshift (/dev/loop1) is on
# oo-accept-node
WARNING: /var/lib/openshift is a loop mount (loop device: /dev/loop1).  Using a loop mount may reduce performance.
PASS


# truncate -s 1G /opt/openshift/test.fs
# mkfs -t ext4 /opt/openshift/test.fs
# mount -oloop,usrquota /opt/openshift/test.fs /var/lib/openshift
# mount |grep openshift
/opt/openshift/test.fs on /var/lib/openshift type ext4 (rw,loop=/dev/loop1,usrquota)
Add one line to /etc/fstab
/opt/openshift/test.fs /var/lib/openshift   ext4    defaults,usrquota     1 0
# mount -o remount /var/lib/openshift
# mount |grep openshift
/opt/openshift/test.fs on /var/lib/openshift type ext4 (rw,usrquota,loop=/dev/loop1)
# quotaon /var/lib/openshift/
# /sbin/quotaon -u -p /var/lib/openshift/
user quota on /var/lib/openshift (/dev/loop1) is on
# oo-accept-node 
WARNING: /var/lib/openshift is a loop mount (loop device: /dev/loop1).  Using a loop mount may reduce performance.
PASS

Comment 15 errata-xmlrpc 2016-03-22 16:51:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-0489.html


Note You need to log in before you can comment on or make changes to this bug.