Description of problem: The LVM stores the LVM metadata on the disk and by default it works with all the devices under /dev/. So it scans every device in /dev/ looking for the LVM metadata and will show the LVM devices obtained from these metadata. When using Ceph rbd storage class, we have a loop device linked to the Persistent Volume. ~~~ /dev/loop2: [0006]:51168221 (/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev/233e0255-280d-4676-a868-8857ec4ad282) ~~~ Although the LVM was created from the VM, the loop device is directly linked to rbd device and hence the LVM in the node is able to find the LVM metadata created by the VM from these loop devices. ~~~ worker-0 ~]# pvs PV VG Fmt Attr PSize PFree /dev/loop2 vg1 lvm2 a-- <30.00g <29.00g ~~~ And the event activation is by default true in LVM which will auto-activate these discovered LVs. ~~~ Dec 28 11:34:50 worker-0.ocp4.shiftvirt.com lvm[27124]: pvscan[27124] PV /dev/loop2 online, VG vg1 is complete. Dec 28 11:34:50 worker-0.ocp4.shiftvirt.com lvm[27124]: pvscan[27124] VG vg1 run autoactivation. Dec 28 11:34:50 worker-0.ocp4.shiftvirt.com lvm[27124]: pvscan[27124] PVID blLzdH-Ww48-4HqC-qYF5-dkSf-ArRd-jyiGxc read from /dev/loop2 last written to /dev/vdc Dec 28 11:34:50 worker-0.ocp4.shiftvirt.com lvm[27124]: 1 logical volume(s) in volume group "vg1" now active ~~~ This will create a dm device on top of the loop device. ~~~ dmsetup ls --tree vg1-lv1 (253:0) `- (7:2) ~~~ Now during the VM power down when it tries to unmap the Persistent Volume, it will fail because here we have dm device active here on the linked loop device preventing the umount. ~~~ Dec 28 11:49:06 worker-0.ocp4.shiftvirt.com hyperkube[2200]: E1228 11:49:06.274235 2200 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com^0001-0011-openshift-storage-0000000000000002-82a7d8eb-869b-11ed-8332-0a580a81016d podName:ae3b79e4-e551-4338-a8b4-f2f18e8c1469 nodeName:}" failed. No retries permitted until 2022-12-28 11:49:38.274206068 +0000 UTC m=+2048.314118469 (durationBeforeRetry 32s). Error: UnmapVolume.UnmapBlockVolume failed for volume "disk-integral-boar" (UniqueName: "kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com^0001-0011-openshift-storage-0000000000000002-82a7d8eb-869b-11ed-8332-0a580a81016d") pod "ae3b79e4-e551-4338-a8b4-f2f18e8c1469" (UID: "ae3b79e4-e551-4338-a8b4-f2f18e8c1469") : blkUtil.DetachFileDevice failed. globalUnmapPath:/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev, podUID: ae3b79e4-e551-4338-a8b4-f2f18e8c1469, bindMount: true: failed to unmount linkPath /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev/ae3b79e4-e551-4338-a8b4-f2f18e8c1469: unmount failed: exit status 32 Dec 28 11:49:06 worker-0.ocp4.shiftvirt.com hyperkube[2200]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev/ae3b79e4-e551-4338-a8b4-f2f18e8c1469: target is busy. ~~~ This causes the virt-launcher pod to be stuck in "Terminating" status indefinitely. Manual umount will also fail: ~~~ # umount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev/ae3b79e4-e551-4338-a8b4-f2f18e8c1469 umount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev/ae3b79e4-e551-4338-a8b4-f2f18e8c1469: target is busy. ~~~ Removing the dm device will allow unmounting and will remove the virt-launcher pod. ~~~ # dmsetup remove vg1-lv1 # umount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d74dc943-7166-4eb9-9d3e-d6e6d220e0f0/dev/ae3b79e4-e551-4338-a8b4-f2f18e8c1469 ~~~ Version-Release number of selected component (if applicable): OpenShift Virtualization 4.11.1 How reproducible: 100% Steps to Reproduce: 1. Create an LVM on the ceph rbd disk from the virtual machine. 2. On the node where the VM is running, run LVM commands. The LVM created from the VM will be visible in the node and the LV will be active. 3. Shutdown this VM. The virt-launcher pod will be stuck in terminating status. Actual results: LVM created from the VMs are getting activated in the OCP nodes Expected results: The LVM created from the VM shouldn't be visible on the OCP node. Additional info:
Thanks, Nijin. I believe that this issue should be solved in OpenShift proper, not just in CNV. I would affect any pod that is binding to a PVC with LV. Hence the link to https://issues.redhat.com/browse/OCPBUGS-5223
Tested with local-storage on: Linux worker-1.virt.toca.local 5.14.0-285.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 7 17:32:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux CentOS Stream CoreOS release 4.13 OCP 4.13.0-rc.0 kubevirt-hyperconverged-operator.4.13.0-1896 I don't see any improvement. Pod stuck in terminating, its still trying to umount it in a loop and failing $ oc get pods -n default NAME READY STATUS RESTARTS AGE virt-launcher-rhel9-wh2o1vdw68gr4bhp-kz7tg 0/1 Terminating 0 9m23s Mar 28 23:11:32 worker-1.virt.toca.local crio[2365]: time="2023-03-28 23:11:32.707072029Z" level=info msg="Stopped container e783e187800aadc8b6b06967b2655363537853fd2f240df7f3f6d8da48b4a483: default/virt-launcher-rhel9-wh2o1vdw68gr4bhp-kz7tg/compute" id=85fd53b5-2c42-437f-a199-1a18b744d999 name=/runtime.v1.RuntimeService/StopContainer Mar 28 23:11:33 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:11:33 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:11:34 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:11:36 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:11:40 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:11:48 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:12:04 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:12:36 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Mar 28 23:13:40 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. ... Mar 28 23:19:47 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. Everything still visible and active on the host: [root@worker-1 core]# pvs WARNING: Not using device /dev/sda1 for PV auGYDr-3O4I-xReX-3VUL-ZZ4T-Xv6o-gin63z. WARNING: PV auGYDr-3O4I-xReX-3VUL-ZZ4T-Xv6o-gin63z prefers device /dev/loop1 because device is used by LV. PV VG Fmt Attr PSize PFree /dev/loop1 guestvg lvm2 a-- <50.00g <30.00g [root@worker-1 core]# vgs WARNING: Not using device /dev/sda1 for PV auGYDr-3O4I-xReX-3VUL-ZZ4T-Xv6o-gin63z. WARNING: PV auGYDr-3O4I-xReX-3VUL-ZZ4T-Xv6o-gin63z prefers device /dev/loop1 because device is used by LV. VG #PV #LV #SN Attr VSize VFree guestvg 1 1 0 wz--n- <50.00g <30.00g [root@worker-1 core]# lvs WARNING: Not using device /dev/sda1 for PV auGYDr-3O4I-xReX-3VUL-ZZ4T-Xv6o-gin63z. WARNING: PV auGYDr-3O4I-xReX-3VUL-ZZ4T-Xv6o-gin63z prefers device /dev/loop1 because device is used by LV. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert guestlv guestvg -wi-a----- 20.00g The current build doesnt seem to be using devicesfile. [root@worker-1 core]# lvmdevices Devices file does not exist. This needs to be properly configured out of the box, or the same nasty issues seen on RHV may happen here.
30 minutes later, still the same. Needed manual cleanup ... Mar 28 23:40:07 worker-1.virt.toca.local kubenswrapper[2398]: Output: umount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/local-pv-86cb7cb0/5e12e03b-f89c-4ed3-b79e-65ca952891d8: target is busy. $ dmsetup table $ dmsetup remove guestgv-guestlv Mar 28 23:42:09 worker-1.virt.toca.local kubenswrapper[2398]: I0328 23:42:09.951024 2398 reconciler_common.go:295] "Volume detached for volume \"local-pv-86cb7cb0\" (UniqueName: \"kubernetes.io/local-volume/local-pv-86cb7cb0\") on node \"worker-1.virt.toca.local\" DevicePath \"/dev/sda1\"" $ oc get pods -n default No resources found in default namespace.
I have been working to reproduce this on a recently deployed 4.11.0-nightly cluster. In the past I could reproduce easily but now the VM is able to shut down cleanly. If you have updated your cluster (including the Nodes themselves) could you try to reproduce again to see if the issue might have been resolved by a RHCOS OS update? It would help to get the precise version of RHCOS that is running. Find the node where the VM was running and do the following: oc debug node/<node-name> chroot /host rpm-ostree status You should get output like the following: # rpm-ostree status State: idle Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f9d64c6756bacfb5f7b44261fd9108921a264d99e2aa5dddfd45548b684a2f1 CustomOrigin: Managed by machine-config-operator Version: 411.86.202307101926-0 (2023-07-10T19:30:23Z)
waiting on ocp bug https://issues.redhat.com/browse/OCPBUGS-5223 to be verified.
See OCPBUGS-5223 for verification details.
verified on OpenShift Virtualization 4.14.0.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Virtualization 4.14.1 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:7704