Bug 1356993

Summary:	Logs spam with: Failed to remove dead container
Product:	OpenShift Container Platform	Reporter:	Jaspreet Kaur <jkaur>
Component:	Node	Assignee:	Andy Goldstein <agoldste>
Status:	CLOSED DUPLICATE	QA Contact:	DeShuai Ma <dma>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.2.0	CC:	aos-bugs, jkaur, jokerman, mmccomas, mpatel
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-09 16:54:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jaspreet Kaur 2016-07-15 12:54:54 UTC

Description of problem: Log shows the  following error messages and seems no way to cure this

OpenShift Enterrpise 3.2.0
kubernetes v1.2.0-36-g4a3f9c5
Docker 1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-44.el7.x86_64
 Go version:      go1.4.2
 Git commit:      7ffc8ee-unsupported
 Built:           Fri Jun 17 15:27:21 2016
 OS/Arch:         linux/amd64

We also recognized this error with the previous used version of Docker (1.9.1)

W0713 12:04:43.839617 3002 container_gc.go:116] Failed to remove dead container "/k8s_POD.dca2255f_logging-es-nwxb4of1-4-vb527_logging_7b209970-450a-11e6-b06d-0050568005a0_ef8eb772": API error (500): Unable to remove filesystem for 22a44fff76f5ed507b6c0fe0b868b47378d5d85aa6a15668904e7316cf081f4b: remove /var/lib/docker/containers/22a44fff76f5ed507b6c0fe0b868b47378d5d85aa6a15668904e7316cf081f4b/shm: device or resource busy
W0713 12:04:39.251165 3001 container_gc.go:116] Failed to remove dead container "/k8s_POD.35f671ae_hawkular-cassandra-1-7e0a6_openshift-infra_f8b58d7c-44e1-11e6-b06d-0050568005a0_244c6cc5": API error (500): Unable to remove filesystem for 69feb93945c86235bed61f432812dc4c6b89dbde66081a0634164e450d5a155b: remove /var/lib/docker/containers/69feb93945c86235bed61f432812dc4c6b89dbde66081a0634164e450d5a155b/shm: device or resource busy
W0713 12:04:39.256088 3001 container_gc.go:116] Failed to remove dead container "/k8s_POD.dca2255f_logging-es-shwbecn8-4-546om_logging_53835ea7-450a-11e6-b06d-0050568005a0_561128b0": API error (500): Unable to remove filesystem for 64725f70747cdf63b99ccbe64e73242c14b94c3a5176f304786b94699520d84f: remove /var/lib/docker/containers/64725f70747cdf63b99ccbe64e73242c14b94c3a5176f304786b94699520d84f/shm: device or resource busy
W0713 12:02:34.400125 2918 container_gc.go:116] Failed to remove dead container "/k8s_POD.70d31fcc_fluentd-dispatcher-3-vvfn9_logging_f762f870-4736-11e6-b06d-0050568005a0_a02fed88": API error (500): Unable to remove filesystem for 7e87d4e5385fb05fe76ade729be4ac2c5e46a0eb898bbe649316d461d6536661: remove /var/lib/docker/containers/7e87d4e5385fb05fe76ade729be4ac2c5e46a0eb898bbe649316d461d6536661/shm: device or resource busy
W0713 12:00:05.406024 3036 container_gc.go:116] Failed to remove dead container "/k8s_POD.95ff44b1_ose-sample-1-et15a_sample-beta-common-1_d9883201-473e-11e6-b06d-0050568005a0_ec75656f": API error (500): Unable to remove filesystem for c3701cd106d4fe25bb0dfe29b1d4ea378c0d2e3dcca7e648c92a4dbfb4d33954: remove /var/lib/docker/containers/c3701cd106d4fe25bb0dfe29b1d4ea378c0d2e3dcca7e648c92a4dbfb4d33954/shm: device or resource busy
W0713 14:05:09.435140 13647 container_gc.go:116] Failed to remove dead container "/k8s_POD.3abbfd0b_docker-registry-8-i80c0_default_2c159247-48ef-11e6-becc-00505680470f_b3bbe920": API error (500): Unable to remove filesystem for 19ef9dc6a30e8b5c0aa44564ffb0cf320ceff72893dc015c35d13f60f9369361: remove /var/lib/docker/containers/19ef9dc6a30e8b5c0aa44564ffb0cf320ceff72893dc015c35d13f60f9369361/shm: device or resource busy



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results: There is no easy way to cure the situation

Expected results: Need an easy way to prevent the message that spam the logs


Additional info:

Comment 1 Andy Goldstein 2016-08-04 10:43:05 UTC

Mrunal, do you have any tips for debugging this?

Comment 2 Mrunal Patel 2016-08-04 17:42:51 UTC

Can we get the output of findmnt in the host mount namespace? I suspect that these mounts are getting propagated to another mount namespace which is holding up the removal. It may be a kernel bug.

Comment 3 Andy Goldstein 2016-08-05 20:57:00 UTC

Jaspreet, could we please get the info Mrunal requested from the customer?

Comment 4 Andy Goldstein 2016-08-09 16:54:03 UTC


*** This bug has been marked as a duplicate of bug 1357121 ***

Comment 5 Red Hat Bugzilla 2023-09-14 03:28:09 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days