Description of problem: journalctl -flu atomic-openshift-node.service | grep -e _wrapped_wrapped_ displays repeated messages for example, Jun 06 11:54:17 netdev35 atomic-openshift-node[115713]: I0606 11:54:17.416698 115713 reconciler.go:201] UnmountVolume operation started for volume "kubernetes.io/configmap/f083fcee-4608-11e7-b1a5-ecf4bbd7ae44-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config-volume.deleting~461540041.deleting~889175940.deleting~469023339.deleting~783960268.deleting~000819238.deleting~209408262.deleting~397218250.deleting~603280394" (spec.Name: "wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config-volume.deleting~461540041.deleting~889175940.deleting~469023339.deleting~783960268.deleting~000819238.deleting~209408262.deleting~397218250.deleting~603280394") from pod "f083fcee-4608-11e7-b1a5-ecf4bbd7ae44" (UID: "f083fcee-4608-11e7-b1a5-ecf4bbd7ae44"). Jun 06 11:54:17 netdev35 atomic-openshift-node[115713]: I0606 11:54:17.421298 115713 reconciler.go:201] UnmountVolume operation started for volume "kubernetes.io/configmap/f083fcee-4608-11e7-b1a5-ecf4bbd7ae44-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config-volume.deleting~151677536.deleting~030149897.deleting~096579521.deleting~512783192.deleting~123077078.deleting~118263454.deleting~001451200.deleting~668627464" (spec.Name: "wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config-volume.deleting~151677536.deleting~030149897.deleting~096579521.deleting~512783192.deleting~123077078.deleting~118263454.deleting~001451200.deleting~668627464") from pod "f083fcee-4608-11e7-b1a5-ecf4bbd7ae44" (UID: "f083fcee-4608-11e7-b1a5-ecf4bbd7ae44"). Jun 06 11:54:17 netdev35 atomic-openshift-node[115713]: I0606 11:54:17.425387 115713 reconciler.go:201] UnmountVolume operation started for volume "kubernetes.io/configmap/f083fcee-4608-11e7-b1a5-ecf4bbd7ae44-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config-volume.deleting~878887437.deleting~626538613.deleting~626857022.deleting~967767454.deleting~704035685.deleting~443312250.deleting~898141921" (spec.Name: "wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config-volume.deleting~878887437.deleting~626538613.deleting~626857022.deleting~967767454.deleting~704035685.deleting~443312250.deleting~898141921") from pod "f083fcee-4608-11e7-b1a5-ecf4bbd7ae44" (UID: "f083fcee-4608-11e7-b1a5-ecf4bbd7ae44"). Version-Release number of selected component (if applicable): software built from latest origin clone. How reproducible: At present can't get it to stop. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
Phil - Can you provide more information including the dump of PVs and PVCs and any pods?
There are no PV/PVC in the setup # oc get po NAME READY STATUS RESTARTS AGE docker-registry-4-2hdmp 1/1 Running 0 1h hello-rc-c9m05 1/1 Running 0 4d
Looks like https://github.com/kubernetes/kubernetes/issues/45935 Fix coming to 1.7 https://github.com/kubernetes/kubernetes/commit/c6bfa43c58d7323563a128b3189e0f7d2359cfc1
This was built with: # go version go version go1.8.1 linux/amd64
how dangerous is the backport of https://github.com/kubernetes/kubernetes/pull/43878 to 3.6? Officially we will only ever build 3.6 with golang 1.7 but Phil won't be the only dev to do so...
Rebuilt with go 1.7.4 still getting the journal messages. Is there some way to clean things up?
The bad "deleting~...deleting~ folders to clean up should be under /var/lib/origin/pods/$POD_UID/volumes/kubernetes.io~config-map
correction: /var/lib/origin/openshift.local.volumes/pods/$POD_UID/volumes/kubernetes.io~config-map
This was caused by go version mismatch, manual cleanup of directories is required but resolves the issue.