Bug 1937243 - Multistage dockerfile builds failing occasionally with "Error shutting down storage: A layer is mounted: layer is in use by a container"
Summary: Multistage dockerfile builds failing occasionally with "Error shutting down s...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: ---
Assignee: Nalin Dahyabhai
QA Contact: Jitendar Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-10 08:59 UTC by Lucas López Montero
Modified: 2024-06-14 00:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-23 13:53:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Problem reproduction files (109.78 KB, application/x-xz)
2021-03-10 08:59 UTC, Lucas López Montero
no flags Details

Description Lucas López Montero 2021-03-10 08:59:35 UTC
Created attachment 1762222 [details]
Problem reproduction files

Description of problem:

Sometimes, when OpenShift is used to build multistage Dockerfiles, it fails with the following error: 

STEP 8: ENV "MAVEN_ARGS_APPEND"="-am -pl microprofile-config -Pbootable-jar-openshift -Djkube.skip=true -Dversion.server.bom=22.0.1.Final -Dversion.microprofile.bom=22.0.1.Final" "MAVEN_OPTS"="-XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"
Error shutting down storage: A layer is mounted: layer is in use by a container
error: build error: error building at STEP "RUN mvn clean package ${MAVEN_ARGS_APPEND}": error while running runtime: exit status 1



Version-Release number of selected component (if applicable): 4.6.18.


How reproducible: intermittently.


Steps to Reproduce:
1. Create a BuildConfig file which uses a multistage Dockerfile.
2. Start the build and check whether the error mentioned shows up.

As the problem is intermittent, the previous steps can be repeated until the problem happens.


Actual results:

Build fails with the error mentioned.


Expected results:

Build is successful.



Additional info:

Find attached the logs of the build pods and the YAML definition of the build used by myself to reproduce the issue.

Regarding the "successful" logs, there is a failure, but it should not be relevant because it happens after "STEP 8".

Comment 3 Adam Kaplan 2021-03-10 16:05:23 UTC
It appears that both builds cited fail for real reasons during the build execution - in one build a maven download had its connection reset, in another a unit test failed because it reached a timeout.

Keeping severity at "medium" - the "Error shutting down storage" seems to be unrelated to the build execution, but is obviously a source of concern to end users. We may be able to reproduce by running a Docker strategy build that fails on purpose.

Comment 4 Lucas López Montero 2021-03-12 14:48:11 UTC
Understood. Thank you very much for having checked this, Adam.

Following KCS article published to share the knowledge: https://access.redhat.com/solutions/5880671. Feel free to review it and contact me if you find something wrong.

Comment 8 Hugo Melendez 2021-09-13 21:43:11 UTC
I have the same issue using

FROM registry.access.redhat.com/ubi8/openjdk-11:1.3 AS builder

as the builder image. After that, you usually do WORKDIR to some dir of your preference for the RUNTIME image to copy artifacts from. But if we remove the WORKDIR statement, everythings works fine.

On the other hand, if I do:

      FROM registry.access.redhat.com/ubi8/openjdk-11:1.3 AS builder

      USER root
      WORKDIR source

      RUN curl -o ${ARTIFACT_NAME}.jar ${ARTIFACT_URL}
works perfectly

Comment 9 Adam Kaplan 2021-09-23 13:53:17 UTC
This bug is being closed because while it represents a valid problem, the user impact is too low relative to the scope of change to resolve it. Rather than leaving such bugs open and providing a false indication that they will be addressed, we are opting to close it. If additional context is provided which increases the importance of this bug, or a proposed fix can be more readily implemented, it can be reopened to be evaluated further.

Comment 10 Thomas Suedbroecker 2021-11-20 20:46:46 UTC
Hi @Adam,

"the user impact is too low relative to the scope of change to resolve it" ... Why?
how is this measured, when would it have been relevant? We have the same problem. What is the exact guideline to work around this inferior bug effectively?

Thanks,
Thomas

Comment 12 Marek Paterczyk 2022-01-24 14:11:46 UTC
We run into the same issue and +1 to getting it fixed. As you can see below, the "Error shutting down storage: A layer is mounted: layer is in use by a container" error is replacing the real error making it difficult to debug.

Our build using BuildConfig on an OpenShift 4.8 cluster:

STEP 26: RUN yum -y install java-1.8.0-openjdk telnet curl unzip rsync
Error shutting down storage: A layer is mounted: layer is in use by a container

The same build run locally using podman:

[2/2] STEP 2/18: RUN yum -y install java-1.8.0-openjdk telnet curl unzip rsync
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Red Hat Universal Base Image 8 (RPMs) - BaseOS  808 kB/s | 794 kB     00:00    
Red Hat Universal Base Image 8 (RPMs) - AppStre 2.7 MB/s | 2.6 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - CodeRea  19 kB/s |  15 kB     00:00    
No match for argument: telnet
Package curl-7.61.1-22.el8.x86_64 is already installed.
Error: Unable to find a match: telnet
Error: error building at STEP "RUN yum -y install java-1.8.0-openjdk telnet curl unzip rsync": error while running runtime: exit status 1


Note You need to log in before you can comment on or make changes to this bug.