Bug 1825294

Summary: VM delete hung after VMI was manually deleted during delete
Product: Container Native Virtualization (CNV) Reporter: Matthew Booth <mbooth>
Component: VirtualizationAssignee: sgott
Status: CLOSED NOTABUG QA Contact: Israel Pinto <ipinto>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.2.0CC: cnv-qe-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-17 15:52:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
kubectl describe vm osp-director
none
kubectl get vm osp-director -o yaml
none
virt-controller logs none

Description Matthew Booth 2020-04-17 15:23:19 UTC 
Description of problem:
I was deleting several VMs with:

kubectl kustomize | kubectl delete -f -

The local object definition contains only the VM, so I assume the specific operation which hung was a VM delete. I'll attach the VM definition to this bug.

The delete of the osp-director VM was taking a long time. As I have grace period set to 600 seconds I decided to delete the VMI manually before the VM delete completed. The VMI delete completed almost immediately. I confirmed that the VMI object was gone, and the VM object had running: false.

However, the VM delete did not complete. events contains:
67m         Normal    Killing                      pod/virt-launcher-osp-director-l7mg8          Stopping container compute
66m         Warning   Unhealthy                    pod/virt-launcher-osp-director-l7mg8          Readiness probe errored: rpc error: code = Unknown desc = container is not created or running

Attempting to delete the VM again similarly hung.

I will attach any relevant logs I can find.

Version-Release number of selected component (if applicable):
Not entirely sure where to get this, but it was installed via Operator Hub.

kubevirt.kubevirt.io/kubevirt-kubevirt-hyperconverged says:
  observedKubeVirtRegistry: docker.io/kubevirt
  observedKubeVirtVersion: v0.26.0
  operatorVersion: v0.0.0-master+$Format:%h$
  phase: Deployed
Comment 1 Matthew Booth 2020-04-17 15:25:48 UTC 
Created attachment 1679684 [details]
kubectl describe vm osp-director
Comment 2 Matthew Booth 2020-04-17 15:27:33 UTC 
Created attachment 1679685 [details]
kubectl get vm osp-director -o yaml
Comment 3 Matthew Booth 2020-04-17 15:39:29 UTC 
Created attachment 1679688 [details]
virt-controller logs
Comment 4 Matthew Booth 2020-04-17 15:52:01 UTC 
The issue here was that the VM had:

    finalizers:
    - k8s.v1.cni.cncf.io/kubeMacPool

However, I had stopped this service due to bug 1816971. Deleting this finalizer caused the VM to be immediately deleted.