Description of problem: When running some builds (this has happened to me twice in about 20 builds), I get the error messages: error: Execution of post execute step failed warning: Failed to remove container "5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74": Error response from daemon: {"message":"Driver devicemapper failed to remove root filesystem 5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74: Device is Busy"} error: build error: building sspeiche-pipeline1/nodejs-mongodb-example-19:8eb738d6 failed when committing the image due to error: Cannot connect to the Docker daemon. Is the docker daemon running on this host? Version-Release number of selected component (if applicable): https://console.preview.openshift.com OpenShift Master: v3.4.0.39 (online version 3.4.0.13) Kubernetes Master: v1.4.0+776c994 How reproducible: Do nodejs-ex builds, repeatedly until it fails. Additional info: Tail of build log: error: Execution of post execute step failed warning: Failed to remove container "5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74": Error response from daemon: {"message":"Driver devicemapper failed to remove root filesystem 5c10a44ab7f93e393152502b7704ba5d0ea446a6987458160c80636b004c9d74: Device is Busy"} error: build error: building sspeiche-pipeline1/nodejs-mongodb-example-19:8eb738d6 failed when committing the image due to error: Cannot connect to the Docker daemon. Is the docker daemon running on this host? $ oc describe build/nodejs-mongodb-example-19 Name: nodejs-mongodb-example-19 Namespace: sspeiche-pipeline1 Created: 54 minutes ago Labels: app=jenkins-pipeline-example buildconfig=nodejs-mongodb-example openshift.io/build-config.name=nodejs-mongodb-example openshift.io/build.start-policy=Serial template=application-template-sample-pipeline Annotations: openshift.io/build-config.name=nodejs-mongodb-example openshift.io/build.number=19 openshift.io/build.pod-name=nodejs-mongodb-example-19-build openshift.io/jenkins-build-uri=https://jenkins-sspeiche-pipeline1.44fs.preview.openshiftapps.com/job/sspeiche-pipeline1-sample-pipeline/19/ Status: Failed Started: Wed, 25 Jan 2017 07:06:29 EST Duration: 2m52s Build Config: nodejs-mongodb-example Build Pod: nodejs-mongodb-example-19-build Strategy: Source URL: https://github.com/openshift/nodejs-ex.git Commit: 7f7c9dd (Merge pull request #90 from coreydaley/update_templates_to_use_secrets) Author/Committer: Ben Parees / GitHub From Image: DockerImage registry.access.redhat.com/rhscl/nodejs-4-rhel7@sha256:7fdfc09047439b9af110112fa3d289187af6a978addcb9d0e48b018c07728310 Output to: ImageStreamTag nodejs-mongodb-example:latest Post Commit Hook: ["/bin/sh", "-ic", "npm test"] Push Secret: builder-dockercfg-npnbu Build trigger cause: Jenkins job URI: https://jenkins-sspeiche-pipeline1.44fs.preview.openshiftapps.com/job/sspeiche-pipeline1-sample-pipeline/19/ Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 49m 49m 1 {default-scheduler } Normal Scheduled Successfully assigned nodejs-mongodb-example-19-build to ip-172-31-2-65.ec2.internal 48m 48m 1 {kubelet ip-172-31-2-65.ec2.internal} spec.containers{sti-build} Normal Pulling pulling image "registry.ops.openshift.com/openshift3/ose-sti-builder:v3.4.0.39" 48m 48m 1 {kubelet ip-172-31-2-65.ec2.internal} spec.containers{sti-build} Normal Pulled Successfully pulled image "registry.ops.openshift.com/openshift3/ose-sti-builder:v3.4.0.39" 48m 48m 1 {kubelet ip-172-31-2-65.ec2.internal} spec.containers{sti-build} Normal Created Created container with docker id 617668b66d64; Security:[seccomp=unconfined] 48m 48m 1 {kubelet ip-172-31-2-65.ec2.internal} spec.containers{sti-build} Normal Started Started container with docker id 617668b66d64 45m 45m 1 {kubelet ip-172-31-2-65.ec2.internal} spec.containers{sti-build} Normal Killing Killing container with docker id 617668b66d64: Need to kill pod.
This looks like a previous issue we'd seen with failing to remove the container. But in addition to that, it looks like the Docker daemon died.
Is it possible to obtain the docker logs from a failed build with the -D option active?
I don't have access to this, we'd need dakini or someone else from ops to get this.
I have not seen this yet on a 3.5 system, then again I haven't used 3.5 that much. I will kick off a repeatable build cycle to try to reproduce.