Bug 1772179 - Unmount error reported frequently from buildah output in openshift builds
Summary: Unmount error reported frequently from buildah output in openshift builds
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Nalin Dahyabhai
QA Contact: wewang
URL:
Whiteboard:
: 1755670 (view as bug list)
Depends On:
Blocks: 1782409
TreeView+ depends on / blocked
 
Reported: 2019-11-13 20:37 UTC by Clayton Coleman
Modified: 2020-05-04 11:16 UTC (History)
11 users (show)

Fixed In Version: buildah-1.11.6-2.el8
Doc Type: Bug Fix
Doc Text:
When preparing to execute a RUN instruction, the OpenShift builder creates a root filesystem using the most recently commited layer, and adds several "bind" mounts to provide a proper runtime environment. After the command specified in the RUN instruction completed, the builder previously attempted to unmount each of those mountpoints in turn, and if it encountered any errors while doing so, the error would be logged. The builder has been updated to unmount only the top-level directory, and to do so using the MNT_DETACH kernel flag, which has the kernel unmount those "bind" mounts. As a result, those errors are no longer encountered, and no longer need to be reported.
Clone Of:
: 1782409 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:15:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift builder pull 119 None closed Bug 1772179: bump github.com/containers/{storage,buildah} 2020-10-05 09:54:50 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:16:06 UTC

Description Clayton Coleman 2019-11-13 20:37:23 UTC
time="2019-11-08T17:35:19Z" level=warning msg="pkg/chroot: error unmounting \"/tmp/buildah790569738/mnt/rootfs\": error checking if \"/tmp/buildah790569738/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"
time="2019-11-08T17:35:19Z" level=warning msg="pkg/bind: error unmounting \"/tmp/buildah790569738/mnt/rootfs\": error checking if \"/tmp/buildah790569738/mnt/rootfs/sys/fs/cgroup/hugetlb\" is mounted: no such file or directory"

This is coming up very often in output of builds, it appears to be a race condition or always happening, and fills up logs.

Needs to be fixed because it makes builds look like they are broken.

Noticed in:

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-compact-4.3/18

[Feature:Builds][Conformance] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [Suite:openshift/conformance/parallel/minima

Comment 3 Nalin Dahyabhai 2019-11-25 14:25:23 UTC
(In reply to Tom Sweeney from comment #2)
> Nalin wasn't this in another BZ that you've been working?  Any updates?

It looks like we can actually skip a lot of the work that we were trying to do here (and by doing so, avoiding the errors we encountered while trying to do it).  buildah #1996 has a proposed fix.

Comment 4 Tom Sweeney 2019-12-04 18:23:05 UTC
I'm going to put this back to assign.  The PR with the fix is not yet passing tests (https://github.com/containers/buildah/pull/1996).  Once merged we can close this out.

Comment 6 Jindrich Novy 2019-12-06 16:39:21 UTC
buildah is now updated to buildah-1.11.6-2.el8 + PR 1996 to fix this.

Comment 8 Nalin Dahyabhai 2019-12-09 16:00:38 UTC
Pretty sure we need to vendor this into openshift/builder, which uses buildah as a library, to have this take effect there.

Comment 13 Adam Kaplan 2019-12-11 15:03:03 UTC
Moving this to builds, will be the "master" 4.4 issue. Will clone for 4.3 - note that given where we are in the release process this may be released in a z-stream.

Comment 15 wewang 2019-12-12 08:40:33 UTC
Checked version as below, which include the pr, no Unmount error info
4.4.0-0.ci-2019-12-12-062440


Jobs for: [Feature:Builds][Conformance] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [Suite:openshift/conformance/parallel/minima

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.4/281

Logs:
https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-4.4/281/artifacts/e2e-gcp/container-logs/test.log

Comment 17 Adam Kaplan 2020-01-22 15:42:39 UTC
*** Bug 1755670 has been marked as a duplicate of this bug. ***

Comment 19 errata-xmlrpc 2020-05-04 11:15:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.