Bug 2024328

Since the deletion is unexpected and occurs in the default configuration, marking this urgent/urgent

The issue has probably existed in our code from day one which supprises me how are we hearing about it just now?
Any way in RHV when you delete a VM then all the attached disks are deleted as well by default since we are using templates to provision our VMs and we want the bootable disk to be removed but not the non bootable disks(PVs) to be deleted, we needed to detach the disks from the VM before deleting it.
The attached PR solves the issue and we will need to backport it as far as we can since it can cause data loss

> The issue has probably existed in our code from day one which supprises me how are we hearing about it just now?

I've asked about the test suite that should have caught this, but the machine delete test don't include additional storage. We'll work with the cloud team to make it more robust, which will benefit all virtualization platforms.

I's not clear that this previously “undefined” or ambiguous behavior. Since we found a similar issue on VMware, is it left to each virtual infrastructure provider to figure out how to handle this use case? Or is it well defined, and I don't understand it yet?

(In reply to Peter Lauterbach from comment #4)
> > The issue has probably existed in our code from day one which supprises me how are we hearing about it just now?
> 
> I've asked about the test suite that should have caught this, but the
> machine delete test don't include additional storage. We'll work with the
> cloud team to make it more robust, which will benefit all virtualization
> platforms.
> 
> I's not clear that this previously “undefined” or ambiguous behavior. Since
> we found a similar issue on VMware, is it left to each virtual
> infrastructure provider to figure out how to handle this use case? Or is it
> well defined, and I don't understand it yet?

We will need to dive a bit into the why to understand what is going on here.
The normal flow should be that when a node is marked for deletion then it is drained from all pods, and only when the node is fully drained then delete the node.
The draining process should take care of detaching the disk before the VM is deleted, since the pod that is using the PVC is not found on the node yet then the disk should be detached.
On my dev and that was the case, but I haven't tried any edge cases like network failure between the node and the cluster on deletion.
The PR should handle making sure we don't delete stuff that shouldn't be deleted, but we need to understand if and when a Machine will be deleted when draining is not done - to eliminate other bugs that may hide behind this issue.

(In reply to Peter Lauterbach from comment #4)
> > The issue has probably existed in our code from day one which supprises me how are we hearing about it just now?
> 
> I've asked about the test suite that should have caught this, but the
> machine delete test don't include additional storage. We'll work with the
> cloud team to make it more robust, which will benefit all virtualization
> platforms.
> 
> I's not clear that this previously “undefined” or ambiguous behavior. Since
> we found a similar issue on VMware, is it left to each virtual
> infrastructure provider to figure out how to handle this use case? Or is it
> well defined, and I don't understand it yet?

I just opened BZ #2026179 with the same problem, as the machineset scale event deleted all data used for ODF. My suggestion is to avoid using the "cascade delete all storage" option when removing a VM. Instead, explicitly detach ALL storage from the VM before deleting. Once the VM is deleted, lookup the osDisk that was created when the VM template was instantiated, and remove it. I'm not sure if a better option would be to have something that looks for "hanging" disk volumees by comparing each disk volume belonging to the OCP cluster to the list of PVs and those unattached and not part of a PV reference, can be deleted once all VMs are removed.  This would allow for the odd template where more than one disk is used as part of the VM template.

Bottom line is that we need to verify each volume attached to the VM against the existing PV definitions. If there is a match, the disk-volume cannot be deleted. HOW (the actual process) of doing that is probably not as important - the above will work in all cases, and if it fails it leaves unused disk-volumes behind, instead of removing too much data. That seems as a safer option.

rhv: 4.4.9.5
ocp: 4.10.0-0.nightly-2021-11-29-191648


steps:
1) create pvc
2) create deployment -> pod was created
3) verify that pod run on machine
4) delete machine and verify that pod move to other machine

1) create pvc
2) create deployment -> pod was created
3) verify that pod run on machine
4) disconnect network from vm that pod run on
5) delete the vm that you disconnect the network
6) verify that pod move to different machine

actual:
pod appear in different machine and PVC doesn't delete

How far back into OCP z-stream will this be backported, I do not see any linked bz.  Definitely both OCP 4.9 and OCP 4.8 are needed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056