Bug 1825294 - VM delete hung after VMI was manually deleted during delete
Summary: VM delete hung after VMI was manually deleted during delete
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 2.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: sgott
QA Contact: Israel Pinto
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-17 15:23 UTC by Matthew Booth
Modified: 2020-04-17 15:52 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-17 15:52:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
kubectl describe vm osp-director (2.58 KB, text/plain)
2020-04-17 15:25 UTC, Matthew Booth
no flags Details
kubectl get vm osp-director -o yaml (3.88 KB, text/plain)
2020-04-17 15:27 UTC, Matthew Booth
no flags Details
virt-controller logs (17.96 KB, text/plain)
2020-04-17 15:39 UTC, Matthew Booth
no flags Details

Description Matthew Booth 2020-04-17 15:23:19 UTC
Description of problem:
I was deleting several VMs with:

kubectl kustomize | kubectl delete -f -

The local object definition contains only the VM, so I assume the specific operation which hung was a VM delete. I'll attach the VM definition to this bug.

The delete of the osp-director VM was taking a long time. As I have grace period set to 600 seconds I decided to delete the VMI manually before the VM delete completed. The VMI delete completed almost immediately. I confirmed that the VMI object was gone, and the VM object had running: false.

However, the VM delete did not complete. events contains:
67m         Normal    Killing                      pod/virt-launcher-osp-director-l7mg8          Stopping container compute
66m         Warning   Unhealthy                    pod/virt-launcher-osp-director-l7mg8          Readiness probe errored: rpc error: code = Unknown desc = container is not created or running

Attempting to delete the VM again similarly hung.

I will attach any relevant logs I can find.

Version-Release number of selected component (if applicable):
Not entirely sure where to get this, but it was installed via Operator Hub.

kubevirt.kubevirt.io/kubevirt-kubevirt-hyperconverged says:
  observedKubeVirtRegistry: docker.io/kubevirt
  observedKubeVirtVersion: v0.26.0
  operatorVersion: v0.0.0-master+$Format:%h$
  phase: Deployed

Comment 1 Matthew Booth 2020-04-17 15:25:48 UTC
Created attachment 1679684 [details]
kubectl describe vm osp-director

Comment 2 Matthew Booth 2020-04-17 15:27:33 UTC
Created attachment 1679685 [details]
kubectl get vm osp-director -o yaml

Comment 3 Matthew Booth 2020-04-17 15:39:29 UTC
Created attachment 1679688 [details]
virt-controller logs

Comment 4 Matthew Booth 2020-04-17 15:52:01 UTC
The issue here was that the VM had:

    finalizers:
    - k8s.v1.cni.cncf.io/kubeMacPool

However, I had stopped this service due to bug 1816971. Deleting this finalizer caused the VM to be immediately deleted.


Note You need to log in before you can comment on or make changes to this bug.