Created attachment 1433157 [details]
pod termination doc
Description of problem:
Fluentd pods stuck in terminating state.
Version-Release number of selected component (if applicable):
No, sorry, I mean, please ask the customer to watch for this to happen again, and when it does, have them view the list of mounted volumes on the node. They'll be looking to see if there is a tmpfs volume mounted in the "busy" directory that the node can't remove. In the above example, they'd do something like "mount | grep 4631-11e8-8ed1-005056ab75a6" and they'd expect to see a volume mounted at /var/lib/origin/openshift.local.volumes/pods/89e072ff-4631-11e8-8ed1-005056ab75a6/volumes/kubernetes.io~secret/aggregated-logging-fluentd-token-n5ppt"
One other thing I forgot to mention is that we have a temporary work-around they can use to get the pod to terminate. If they manually umount the mounted path, the pod should terminate. We'll work on figuring out why it is failing to unmount on its own.