Bug 1427691 - Build fails with Driver devicemapper failed to remove root filesystem - Device Busy
Summary: Build fails with Driver devicemapper failed to remove root filesystem - Devi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.4.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 3.4.z
Assignee: Ben Parees
QA Contact: Vikas Laad
URL:
Whiteboard:
Depends On: 1391665
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-28 21:41 UTC by Ben Parees
Modified: 2020-05-14 15:41 UTC (History)
27 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Source to image builds expect image commits to take no longer than 2 minutes. Consequence: Commits which take longer than 2 minutes result in a timeout and a failed build. Fix: The timeout will be removed so image commits can take indeterminate lengths of time. Result: Commits which take an excessive amount of time will not result in a failed build.
Clone Of: 1391665
Environment:
Last Closed: 2017-03-15 20:04:02 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0512 0 normal SHIPPED_LIVE OpenShift Container Platform 3.4.1.10, 3.3.1.17, and 3.2.1.28 bug fix update 2017-03-16 00:01:17 UTC

Comment 2 Ben Parees 2017-03-01 16:42:29 UTC 
backport to 3.4.x: https://github.com/openshift/origin/pull/13174
Comment 3 Ben Parees 2017-03-06 15:41:16 UTC 
correct backport to 3.4.x pr: https://github.com/openshift/ose/pull/645
Comment 4 Troy Dawson 2017-03-07 19:57:27 UTC 
This has been merged into ocp and is in OCP v3.4.1.10 or newer.
Comment 6 Vikas Laad 2017-03-08 20:11:53 UTC 
I am verifying the fix, here is what I am seeing when running out of iops

root@ip-172-31-54-81: ~ # oc logs -n proj5       cakephp-mysql-example-40-build 
Cloning "https://github.com/redhat-performance/cakephp-ex.git" ...
        Commit: 0014ddebb91bc7dff3a1dabfbd7b51da762a6677 (made changes to enable database example)
        Author: ofthecure <robdean.smith>
        Date:   Mon Apr 25 14:33:06 2016 -0400
DEPRECATED: Use .s2i/bin instead of .sti/bin


error: build error: Cannot connect to the Docker daemon. Is the docker daemon running on this host?

Other error is pod stuck in RunContainerError and description of pod has

  12m           5m              4       {kubelet ip-172-31-31-237.us-west-2.compute.internal}   spec.containers{sti-build}      Warning         Failed          Failed to create docker container "sti-build" of pod "cakephp-mysql-example-40-build_proj4(16192de4-0439-11e7-8ba8-024b76e8e62f)" with error: operation timeout: context deadline exceeded
  12m           5m              4       {kubelet ip-172-31-31-237.us-west-2.compute.internal}                                   Warning         FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "sti-build" with RunContainerError: "runContainer: operation timeout: context deadline exceeded"

I think these are expected ? please confirm.
Comment 7 Ben Parees 2017-03-09 03:06:17 UTC 
yeah i'd say that's not surprising if you're out of iops.  But if you want to actually recreate the bug at hand, you need to run out of iops during the commit, not before.  The easiest way to make that happen might be to write an assemble script that does a sleep at the end.  Once the assemble script hits the sleep, then trigger whatever you are doing to exhaust/limit the iops.
Comment 9 Vikas Laad 2017-03-10 13:49:52 UTC 
Completed testing on 3.4.1.10, did not see this issue.
Comment 11 errata-xmlrpc 2017-03-15 20:04:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0512
Note You need to log in before you can comment on or make changes to this bug.