Description of problem: Log shows the following error messages and seems no way to cure this OpenShift Enterrpise 3.2.0 kubernetes v1.2.0-36-g4a3f9c5 Docker 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-44.el7.x86_64 Go version: go1.4.2 Git commit: 7ffc8ee-unsupported Built: Fri Jun 17 15:27:21 2016 OS/Arch: linux/amd64 We also recognized this error with the previous used version of Docker (1.9.1) W0713 12:04:43.839617 3002 container_gc.go:116] Failed to remove dead container "/k8s_POD.dca2255f_logging-es-nwxb4of1-4-vb527_logging_7b209970-450a-11e6-b06d-0050568005a0_ef8eb772": API error (500): Unable to remove filesystem for 22a44fff76f5ed507b6c0fe0b868b47378d5d85aa6a15668904e7316cf081f4b: remove /var/lib/docker/containers/22a44fff76f5ed507b6c0fe0b868b47378d5d85aa6a15668904e7316cf081f4b/shm: device or resource busy W0713 12:04:39.251165 3001 container_gc.go:116] Failed to remove dead container "/k8s_POD.35f671ae_hawkular-cassandra-1-7e0a6_openshift-infra_f8b58d7c-44e1-11e6-b06d-0050568005a0_244c6cc5": API error (500): Unable to remove filesystem for 69feb93945c86235bed61f432812dc4c6b89dbde66081a0634164e450d5a155b: remove /var/lib/docker/containers/69feb93945c86235bed61f432812dc4c6b89dbde66081a0634164e450d5a155b/shm: device or resource busy W0713 12:04:39.256088 3001 container_gc.go:116] Failed to remove dead container "/k8s_POD.dca2255f_logging-es-shwbecn8-4-546om_logging_53835ea7-450a-11e6-b06d-0050568005a0_561128b0": API error (500): Unable to remove filesystem for 64725f70747cdf63b99ccbe64e73242c14b94c3a5176f304786b94699520d84f: remove /var/lib/docker/containers/64725f70747cdf63b99ccbe64e73242c14b94c3a5176f304786b94699520d84f/shm: device or resource busy W0713 12:02:34.400125 2918 container_gc.go:116] Failed to remove dead container "/k8s_POD.70d31fcc_fluentd-dispatcher-3-vvfn9_logging_f762f870-4736-11e6-b06d-0050568005a0_a02fed88": API error (500): Unable to remove filesystem for 7e87d4e5385fb05fe76ade729be4ac2c5e46a0eb898bbe649316d461d6536661: remove /var/lib/docker/containers/7e87d4e5385fb05fe76ade729be4ac2c5e46a0eb898bbe649316d461d6536661/shm: device or resource busy W0713 12:00:05.406024 3036 container_gc.go:116] Failed to remove dead container "/k8s_POD.95ff44b1_ose-sample-1-et15a_sample-beta-common-1_d9883201-473e-11e6-b06d-0050568005a0_ec75656f": API error (500): Unable to remove filesystem for c3701cd106d4fe25bb0dfe29b1d4ea378c0d2e3dcca7e648c92a4dbfb4d33954: remove /var/lib/docker/containers/c3701cd106d4fe25bb0dfe29b1d4ea378c0d2e3dcca7e648c92a4dbfb4d33954/shm: device or resource busy W0713 14:05:09.435140 13647 container_gc.go:116] Failed to remove dead container "/k8s_POD.3abbfd0b_docker-registry-8-i80c0_default_2c159247-48ef-11e6-becc-00505680470f_b3bbe920": API error (500): Unable to remove filesystem for 19ef9dc6a30e8b5c0aa44564ffb0cf320ceff72893dc015c35d13f60f9369361: remove /var/lib/docker/containers/19ef9dc6a30e8b5c0aa44564ffb0cf320ceff72893dc015c35d13f60f9369361/shm: device or resource busy Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: There is no easy way to cure the situation Expected results: Need an easy way to prevent the message that spam the logs Additional info:
Mrunal, do you have any tips for debugging this?
Can we get the output of findmnt in the host mount namespace? I suspect that these mounts are getting propagated to another mount namespace which is holding up the removal. It may be a kernel bug.
Jaspreet, could we please get the info Mrunal requested from the customer?
*** This bug has been marked as a duplicate of bug 1357121 ***
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days