Bug 1893790 - VM stuck in pending state
Summary: VM stuck in pending state
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 2.5.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: lpivarc
QA Contact: Israel Pinto
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-02 15:59 UTC by Israel Pinto
Modified: 2021-07-27 14:21 UTC (History)
5 users (show)

Fixed In Version: hco-bundle-registry-container-v4.8.0-347 virt-operator-container-v4.8.0-58
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 14:20:49 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
vm spec (2.20 KB, text/plain)
2020-11-02 16:01 UTC, Israel Pinto
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:2920 0 None None None 2021-07-27 14:21:52 UTC

Description Israel Pinto 2020-11-02 15:59:08 UTC
Description of problem:
While create vm with data volume template with wrong attribute for dv and vm.spec.running=ture 
See attached spec. 
VM is in pending status since dv is in pending,
I can't get out of the pending state, i try the following:
1. Stop vm with virtctl  
2. Edit running status from running:true to running:false. 
3. Delete importer pod

There no way of keeping the VM and get out of the pending state. 


Version-Release number of selected component (if applicable):
CNV-2.5

How reproducible:
100%

Steps to Reproduce:
oc create -f <attached vm yaml>

Actual results:
VM is stuck on pending. 

Expected results:
Keep the vm and not delete it.

Comment 1 Israel Pinto 2020-11-02 16:01:53 UTC
Created attachment 1725890 [details]
vm spec

Comment 2 Guohua Ouyang 2020-11-19 09:19:05 UTC
Could not reproduce the issue by using the provided yaml to create VM on latest CNV 2.5.

Comment 3 sgott 2020-12-04 14:25:32 UTC
Israel, is this still an issue?

Comment 6 sgott 2021-01-27 16:02:54 UTC
Does this still occur on the current release?

Comment 7 Israel Pinto 2021-02-26 17:09:19 UTC
(In reply to sgott from comment #6)
> Does this still occur on the current release?

Reproduce with:
#oc get clusterversions.config.openshift.io 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-02-25-151800   True        False         10h     Cluster version is 4.8.0-0.nightly-2021-02-25-151800
#oc get csv -n openshift-cnv
NAME                                      DISPLAY                    VERSION   REPLACES                                  PHASE
kubevirt-hyperconverged-operator.v4.8.0   OpenShift Virtualization   4.8.0     kubevirt-hyperconverged-operator.v2.5.3   Succeeded

Steps:
Create VM with the attached VM spec. 
VMI is in pending state since import is failed
I can't stop the VMI with virtctl, only delete it. 
See outputs: 
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get vmi 
NAME     AGE     PHASE     IP    NODENAME
f31-v2   2m12s   Pending         
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get pods
NAME                    READY   STATUS             RESTARTS   AGE
importer-f31-rootdisk   0/1     CrashLoopBackOff   2          67s
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get pods
NAME                    READY   STATUS             RESTARTS   AGE
importer-f31-rootdisk   0/1     CrashLoopBackOff   2          71s
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get pods
NAME                    READY   STATUS    RESTARTS   AGE
importer-f31-rootdisk   1/1     Running   3          75s
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get pods
NAME                    READY   STATUS   RESTARTS   AGE
importer-f31-rootdisk   0/1     Error    3          82s
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get pods
NAME                    READY   STATUS   RESTARTS   AGE
importer-f31-rootdisk   0/1     Error    3          86s
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get pods
NAME                    READY   STATUS             RESTARTS   AGE
importer-f31-rootdisk   0/1     CrashLoopBackOff   3          92s
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get vmi 
NAME     AGE     PHASE     IP    NODENAME
f31-v2   2m49s   Pending         
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ virtctl stop f31-v2
VM f31-v2 was scheduled to stop
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get vmi 
NAME     AGE     PHASE     IP    NODENAME
f31-v2   3m10s   Pending         
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get vmi 
NAME     AGE     PHASE     IP    NODENAME
f31-v2   3m13s   Pending         
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc delete vm f31-v2
virtualmachine.kubevirt.io "f31-v2" deleted
[cnv-qe-jenkins@virt01-4vvbc-executor ~]$ oc get vmi 
No resources found in default namespace.

Comment 8 lpivarc 2021-03-30 13:22:01 UTC
https://github.com/kubevirt/kubevirt/pull/5349 should fix this.

Comment 9 Shaul Garbourg 2021-04-20 08:35:29 UTC
PR https://github.com/kubevirt/kubevirt/pull/5349 was merged to master, moving to MODIFIED

Comment 10 Israel Pinto 2021-05-26 06:55:19 UTC
Verify with the vm spec (image url no exits)

---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  annotations:
    description: Fedora 31 image to run kubevirtci k8s-1.19 in it.
    kubevirt.io/latest-observed-api-version: v1alpha3
    kubevirt.io/storage-observed-api-version: v1alpha3
    name.os.template.kubevirt.io/silverblue32: Fedora 31 or higher
  name: f31-v2
  labels:
    app: f31-v2
spec:
  dataVolumeTemplates:
      -

       metadata:
         name: f31-rootdisk
       spec:
         pvc:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 10Gi
           volumeMode: Filesystem
         source:
           http:
             url: "https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.6.x86_64.qcow2"
       status: {}
  running: true
  template:
    metadata:
      labels:
        flavor.template.kubevirt.io/medium: 'true'
        kubevirt.io/domain: f31
        kubevirt.io/size: medium
        os.template.kubevirt.io/silverblue32: 'true'
        vm.kubevirt.io/name: f31
        workload.template.kubevirt.io/server: 'true'
    spec:
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
            - bootOrder: 1
              disk:
                bus: virtio
              name: rootdisk
          interfaces:
            - masquerade: {}
              model: virtio
              name: nic-0
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: pc-q35-rhel8.2.0
        resources:
          requests:
            memory: 4Gi
      evictionStrategy: LiveMigrate
      hostname: f31
      networks:
        - name: nic-0
          pod: {}
      terminationGracePeriodSeconds: 180
      volumes:
        - dataVolume:
            name: f31-rootdisk
          name: rootdisk
status: {}


flow:
# oc apply -f ~/cnv_yamls/pending_bug.yaml -n israel-bugs                                                                                                   (9:49)(:|✔)
virtualmachine.kubevirt.io/f31-v2 created
# oc get vm,vmi,dv  -o wide -n israel-bugs                                                                                                                  (9:50)(:|✔)
NAME                                AGE   VOLUME   CREATED
virtualmachine.kubevirt.io/f31-v2   3s             true

NAME                                        AGE   PHASE     IP    NODENAME   LIVE-MIGRATABLE   PAUSED
virtualmachineinstance.kubevirt.io/f31-v2   3s    Pending                                      

NAME                                      PHASE      PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/f31-rootdisk   PVCBound   N/A                   4s
# oc get vm,vmi,dv  -o wide -n israel-bugs                                                                                                                  (9:50)(:|✔)
NAME                                AGE   VOLUME   CREATED
virtualmachine.kubevirt.io/f31-v2   10s            true

NAME                                        AGE   PHASE     IP    NODENAME   LIVE-MIGRATABLE   PAUSED
virtualmachineinstance.kubevirt.io/f31-v2   10s   Pending                                      

NAME                                      PHASE              PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/f31-rootdisk   ImportInProgress   N/A        1          11s
# oc get vm,vmi,dv,pods  -o wide -n israel-bugs                                                                                                             (9:50)(:|✔)
NAME                                AGE   VOLUME   CREATED
virtualmachine.kubevirt.io/f31-v2   17s            true

NAME                                        AGE   PHASE     IP    NODENAME   LIVE-MIGRATABLE   PAUSED
virtualmachineinstance.kubevirt.io/f31-v2   17s   Pending                                      

NAME                                      PHASE              PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/f31-rootdisk   ImportInProgress   N/A        1          18s

NAME                        READY   STATUS             RESTARTS   AGE   IP             NODE                                      NOMINATED NODE   READINESS GATES
pod/importer-f31-rootdisk   0/1     CrashLoopBackOff   1          12s   10.129.3.153   cnv-qe-13.cnvqe.lab.eng.rdu2.redhat.com   <none>           <none>
# oc get vm,vmi,dv,pods  -o wide -n israel-bugs                                                                                                             (9:50)(:|✔)
NAME                                AGE   VOLUME   CREATED
virtualmachine.kubevirt.io/f31-v2   22s            true

NAME                                        AGE   PHASE     IP    NODENAME   LIVE-MIGRATABLE   PAUSED
virtualmachineinstance.kubevirt.io/f31-v2   22s   Pending                                      

NAME                                      PHASE              PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/f31-rootdisk   ImportInProgress   N/A        1          23s

NAME                        READY   STATUS             RESTARTS   AGE   IP             NODE                                      NOMINATED NODE   READINESS GATES
pod/importer-f31-rootdisk   0/1     CrashLoopBackOff   1          17s   10.129.3.153   cnv-qe-13.cnvqe.lab.eng.rdu2.redhat.com   <none>           <none>
# virtctl stop f31-v2 -n israel-bugs                                                                                                                        (9:50)(:|✔)
VM f31-v2 was scheduled to stop
# oc get vm,vmi,dv  -o wide -n israel-bugs                                                                                                                  (9:50)(:|✔)
NAME                                AGE   VOLUME   CREATED
virtualmachine.kubevirt.io/f31-v2   39s            

NAME                                      PHASE              PROGRESS   RESTARTS   AGE
datavolume.cdi.kubevirt.io/f31-rootdisk   ImportInProgress   N/A        2          40s
# oc delete vm f31-v2 -n israel-bugs                                                                                                                        (9:51)(:|✔)
virtualmachine.kubevirt.io "f31-v2" deleted


VM been stopped with virtctl, we can get out of pending state. 
moving to verfiy.

Comment 14 errata-xmlrpc 2021-07-27 14:20:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2920


Note You need to log in before you can comment on or make changes to this bug.