Bug 1772179

Summary: Unmount error reported frequently from buildah output in openshift builds
Product: OpenShift Container Platform Reporter: Clayton Coleman <ccoleman>
Component: BuildAssignee: Nalin Dahyabhai <nalin>
Status: CLOSED ERRATA QA Contact: wewang <wewang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: adam.kaplan, aos-bugs, dwalsh, hfukumot, jnovy, jokerman, mhayashi, mpatel, nalin, tsweeney, wzheng
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: buildah-1.11.6-2.el8 Doc Type: Bug Fix
Doc Text:
When preparing to execute a RUN instruction, the OpenShift builder creates a root filesystem using the most recently commited layer, and adds several "bind" mounts to provide a proper runtime environment. After the command specified in the RUN instruction completed, the builder previously attempted to unmount each of those mountpoints in turn, and if it encountered any errors while doing so, the error would be logged. The builder has been updated to unmount only the top-level directory, and to do so using the MNT_DETACH kernel flag, which has the kernel unmount those "bind" mounts. As a result, those errors are no longer encountered, and no longer need to be reported.
Story Points: ---
Clone Of:
: 1782409 (view as bug list) Environment:
Last Closed: 2020-05-04 11:15:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1782409    

Description Clayton Coleman 2019-11-13 20:37:23 UTC
time="2019-11-08T17:35:19Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah790569738/mnt/rootfs\": error checking if \"/tmp/buildah790569738/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2019-11-08T17:35:19Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah790569738/mnt/rootfs\": error checking if \"/tmp/buildah790569738/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"

This is coming up very often in output of builds, it appears to be a race condition or always happening, and fills up logs.

Needs to be fixed because it makes builds look like they are broken.

Noticed in:

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-compact-4.3/18

[Feature:Builds][Conformance] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [Suite:openshift/conformance/parallel/minima

Comment 3 Nalin Dahyabhai 2019-11-25 14:25:23 UTC
(In reply to Tom Sweeney from comment #2)
> Nalin wasn't this in another BZ that you've been working?  Any updates?

It looks like we can actually skip a lot of the work that we were trying to do here (and by doing so, avoiding the errors we encountered while trying to do it).  buildah #1996 has a proposed fix.

Comment 4 Tom Sweeney 2019-12-04 18:23:05 UTC
I'm going to put this back to assign.  The PR with the fix is not yet passing tests (https://github.com/containers/buildah/pull/1996).  Once merged we can close this out.

Comment 6 Jindrich Novy 2019-12-06 16:39:21 UTC
buildah is now updated to buildah-1.11.6-2.el8 + PR 1996 to fix this.

Comment 8 Nalin Dahyabhai 2019-12-09 16:00:38 UTC
Pretty sure we need to vendor this into openshift/builder, which uses buildah as a library, to have this take effect there.

Comment 13 Adam Kaplan 2019-12-11 15:03:03 UTC
Moving this to builds, will be the "master" 4.4 issue. Will clone for 4.3 - note that given where we are in the release process this may be released in a z-stream.

Comment 15 wewang 2019-12-12 08:40:33 UTC
Checked version as below, which include the pr, no Unmount error info
4.4.0-0.ci-2019-12-12-062440


Jobs for: [Feature:Builds][Conformance] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [Suite:openshift/conformance/parallel/minima

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.4/281

Logs:
https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.4/281/artifacts/e2e-gcp/container-logs/test.log

Comment 17 Adam Kaplan 2020-01-22 15:42:39 UTC
*** Bug 1755670 has been marked as a duplicate of this bug. ***

Comment 19 errata-xmlrpc 2020-05-04 11:15:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581