Bug 1416411 - [preview] build error: building xyz failed when committing the image due to error: Cannot connect to the Docker daemon
Summary: [preview] build error: building xyz failed when committing the image due to e...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 3.x
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Jhon Honce
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-25 12:59 UTC by Steve Speicher
Modified: 2020-08-13 08:50 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-14 16:42:50 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Steve Speicher 2017-01-25 12:59:28 UTC
Description of problem:
When running some builds (this has happened to me twice in about 20 builds), I get the error messages:

error: Execution of post execute step failed
warning: Failed to remove container "5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74": Error response from daemon: {"message":"Driver devicemapper failed to remove root filesystem 5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74: Device is Busy"}
error: build error: building sspeiche-pipeline1/nodejs-mongodb-example-19:8eb738d6 failed when committing the image due to error: Cannot connect to the Docker daemon. Is the docker daemon running on this host?


Version-Release number of selected component (if applicable):
https://console.preview.openshift.com

OpenShift Master:
v3.4.0.39 (online version 3.4.0.13)
Kubernetes Master:
v1.4.0+776c994

How reproducible:
Do nodejs-ex builds, repeatedly until it fails.


Additional info:

Tail of build log:

error: Execution of post execute step failed
warning: Failed to remove container "5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74": Error response from daemon: {"message":"Driver devicemapper failed to remove root filesystem 5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74: Device is Busy"}
error: build error: building sspeiche-pipeline1/nodejs-mongodb-example-19:8eb738d6 failed when committing the image due to error: Cannot connect to the Docker daemon. Is the docker daemon running on this host?

$ oc describe build/nodejs-mongodb-example-19
Name:		nodejs-mongodb-example-19
Namespace:	sspeiche-pipeline1
Created:	54 minutes ago
Labels:		app=jenkins-pipeline-example
		buildconfig=nodejs-mongodb-example
		openshift.io/build-config.name=nodejs-mongodb-example
		openshift.io/build.start-policy=Serial
		template=application-template-sample-pipeline
Annotations:	openshift.io/build-config.name=nodejs-mongodb-example
		openshift.io/build.number=19
		openshift.io/build.pod-name=nodejs-mongodb-example-19-build
		openshift.io/jenkins-build-uri=https://jenkins-sspeiche-pipeline1.44fs.preview.openshiftapps.com/job/sspeiche-pipeline1-sample-pipeline/19/

Status:		Failed
Started:	Wed, 25 Jan 2017 07:06:29 EST
Duration:	2m52s
Build Config:	nodejs-mongodb-example
Build Pod:	nodejs-mongodb-example-19-build

Strategy:		Source
URL:			https://github.com/openshift/nodejs-ex.git
Commit:			7f7c9dd (Merge pull request #90 from coreydaley/update_templates_to_use_secrets)
Author/Committer:	Ben Parees / GitHub
From Image:		DockerImage registry.access.redhat.com/rhscl/nodejs-4-rhel7@sha256:7fdfc09047439b9af110112fa3d289187af6a978addcb9d0e48b018c07728310
Output to:		ImageStreamTag nodejs-mongodb-example:latest
Post Commit Hook:	["/bin/sh", "-ic", "npm test"]
Push Secret:		builder-dockercfg-npnbu

Build trigger cause:	Jenkins job URI: https://jenkins-sspeiche-pipeline1.44fs.preview.openshiftapps.com/job/sspeiche-pipeline1-sample-pipeline/19/

Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath			Type		Reason		Message
  ---------	--------	-----	----					-------------			--------	------		-------
  49m		49m		1	{default-scheduler }							Normal		Scheduled	Successfully assigned nodejs-mongodb-example-19-build to ip-172-31-2-65.ec2.internal
  48m		48m		1	{kubelet ip-172-31-2-65.ec2.internal}	spec.containers{sti-build}	Normal		Pulling		pulling image "registry.ops.openshift.com/openshift3/ose-sti-builder:v3.4.0.39"
  48m		48m		1	{kubelet ip-172-31-2-65.ec2.internal}	spec.containers{sti-build}	Normal		Pulled		Successfully pulled image "registry.ops.openshift.com/openshift3/ose-sti-builder:v3.4.0.39"
  48m		48m		1	{kubelet ip-172-31-2-65.ec2.internal}	spec.containers{sti-build}	Normal		Created		Created container with docker id 617668b66d64; Security:[seccomp=unconfined]
  48m		48m		1	{kubelet ip-172-31-2-65.ec2.internal}	spec.containers{sti-build}	Normal		Started		Started container with docker id 617668b66d64
  45m		45m		1	{kubelet ip-172-31-2-65.ec2.internal}	spec.containers{sti-build}	Normal		Killing		Killing container with docker id 617668b66d64: Need to kill pod.

Comment 1 Cesar Wong 2017-01-25 14:41:21 UTC
This looks like a previous issue we'd seen with failing to remove the container. But in addition to that, it looks like the Docker daemon died.

Comment 2 Jhon Honce 2017-02-03 23:10:46 UTC
Is it possible to obtain the docker logs from a failed build with the -D option active?

Comment 3 Steve Speicher 2017-02-05 14:18:26 UTC
I don't have access to this, we'd need dakini or someone else from ops to get this.

Comment 8 Steve Speicher 2017-05-08 13:20:04 UTC
I have not seen this yet on a 3.5 system, then again I haven't used 3.5 that much. I will kick off a repeatable build cycle to try to reproduce.


Note You need to log in before you can comment on or make changes to this bug.