Description of problem: If for some reason you need to force-delete a pod, it may happen that the volume remains mounted in the node. Then diskmaker cannot clean it and it will fail with messages like: Deleting PV block volume "local-pv-96cf56b9" device hostpath "/mnt/local-storage/general/dm-name-autopart-lv_9", mountpath "/mnt/local-storage/general/dm-name-autopart-lv_9" Cleanup pv "local-pv-96cf56b9": StdoutBuf - "Calling mkfs" Cleanup pv "local-pv-96cf56b9": StdoutBuf - "mke2fs 1.45.6 (20-Mar-2020)" Cleanup pv "local-pv-96cf56b9": StdoutBuf - "/mnt/local-storage/general/dm-name-autopart-lv_9 is apparently in use by the system; will not make a filesystem here" In the node we can see this: $ lsblk <output_omitted> autopart-lv_9 235:8 0 1G 0 lvm /var/lib/kubelet/pods/8dfac272-50dd-445a-9267-ed4c0f527e44/volumes/kubernetes.io~local-volume/local-pv-96cf56b9 <output_omitted> And if we look at the pv state: oc get pv local-pv-96cf56b9 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-96cf56b9 1Gi RWO Delete Available general 114m That means that a PVC can bound to this PV and pod will fail to work since the lv is still mounted and was not cleaned up. Version-Release number of selected component (if applicable): 4.8.13 How reproducible: Under certain conditions Steps to Reproduce: 1. Create a pod that mounts a volume exposed by LSO 2. Force delete the pod 3. Check if volume is still mounted in the node 4. Check pv state Actual results: PV is available Expected results: PV is not available
@kir any news on this? if the fix was included in 4.6 should not be already included in 4.8?
It looks like the runc fix mentioned earlier is unrelated. From my perspective, what happens here is there are some processes still left in the cgroup, this is the reason why it can't be removed. Volumes are out of runc scope, but the volume might be left mounted for the same reason why cgroup can't be removed -- there are some processes using it. I'd use lsof or similar tools to investigate further.
If a file system cannot be unmounted due to a process using it, there isn't much that can be done by the file system or below - it would have to be resolved by the process releasing the FS first, then unmounting, etc. It is possible that it is storage related if: 1) the storage is blocking/hung and the process that is preventing the unmount cannot proceed. This could happen, for example, if a device-mapper device was suspended (I don't immediately see an indication of this). 2) A FS or storage tool /is/ the process preventing unmount. It could be hung due to locking or other software bug. I find this very unlikely, as most often these tools do not utilize the storage they are administering. Comment 9 seems like the best suggestion - identify the process(es) that are running which are preventing the unmount. There are nastier hacks which could be performed (like remapping the PV to 'error'), but that would be worst case, I think.
I now suspect this is a dup of https://bugzilla.redhat.com/show_bug.cgi?id=2003206. according to https://bugzilla.redhat.com/show_bug.cgi?id=2014083#c18 we're hitting a deadlock with stop. I could verify if we could gather the goroutine stacks of a running instance: https://github.com/cri-o/cri-o/blob/main/tutorials/debugging.md#printing-go-routines
setting need info for information requested in https://bugzilla.redhat.com/show_bug.cgi?id=2014083#c41
a clarification comment: if we get into this situation where a pod can't be cleaned up, I'll need someone to ssh to the node and run commands described in https://github.com/cri-o/cri-o/blob/main/tutorials/debugging.md#printing-go-routines and post the file written to /tmp. That will help me determine whether the issue we're seeing is the same as the suspected duplicate.
(In reply to Peter Hunt from comment #44) > a clarification comment: if we get into this situation where a pod can't be > cleaned up, I'll need someone to ssh to the node and run commands described > in > https://github.com/cri-o/cri-o/blob/main/tutorials/debugging.md#printing-go- > routines and post the file written to /tmp. That will help me determine > whether the issue we're seeing is the same as the suspected duplicate. Hi Peter, I will contact customer and get you that stack trace asap. The case linked to this bz was put on hold due to the possible duplicate with namespace termination mentioned. br, Chris
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days