This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 2241637 - [4.14] DV status is 'ImportScheduled' when importer-prime pod FailedMount, VMI status is 'Scheduling'
Summary: [4.14] DV status is 'ImportScheduled' when importer-prime pod FailedMount, VM...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 4.14.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.14.2
Assignee: Álvaro Romero
QA Contact: Jenia Peimer
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-10-01 14:54 UTC by Jenia Peimer
Modified: 2024-02-19 07:29 UTC (History)
2 users (show)

Fixed In Version: v4.14.1.rhel9-22
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-12-14 16:15:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt containerized-data-importer pull 2928 0 None open Improve DataVolume status reporting with populators 2023-10-16 09:48:31 UTC
Red Hat Issue Tracker   CNV-33620 0 None None None 2023-12-14 16:15:53 UTC

Description Jenia Peimer 2023-10-01 14:54:14 UTC
Description of problem:
When we create a VM on a node without HPP pod,
importer-prime pod shows FailedMount in the events, but 
DV's status is 'ImportScheduled', which is misleading. 
It leads to VM status 'Provisioning' and VMI status 'Scheduling', 
although there's a clear issue with the storage.
DV should report 'Pending' status or some other error status, 
and VMI should be 'Pending'.

Version-Release number of selected component (if applicable):
4.14, 4.15

How reproducible:
Always

Steps to Reproduce:
1. Add nodeSelector to hostpath-provisioner CR:

$ oc edit hostpathprovisioner hostpath-provisioner

kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  workload:
    nodeSelector:
      kubernetes.io/hostname: c01-jp415-57lmz-worker-0-24q58

2. Wait for only 1 'hostpath-provisioner-csi' pod to be Running.

3. Create a VM on another node:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-cirros-source-hpp
spec:
  dataVolumeTemplates:
  - metadata:
      name: cirros-dv-source-hpp
    spec:
      storage:
        resources:
          requests:
            storage: 1Gi
        storageClassName: hostpath-csi-basic
      source:
        http:
          url: <cirros.qcow2>
  running: true
  template:
    spec:
      nodeSalactor: 
        kubernetes.io/hostname: c01-jp415-57lmz-worker-0-vrtp2
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: datavolumev-hpp
        machine:
          type: ""
        resources:
          requests:
            memory: 100M
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: cirros-dv-source-hpp
        name: datavolumev-hpp


Actual results:

DV is ImportScheduled
PVC is Pending
PVC-prime is Pending
VM is Provisioning
VMI is Scheduling
importer-prime pod has FailedMount Event
virt-launcher pod doesn't have any Events

$ oc get dv 
NAME                   PHASE             PROGRESS   RESTARTS   AGE
cirros-dv-source-hpp   ImportScheduled   N/A                   4m55s

$ oc get pvc 
NAME                                         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
cirros-dv-source-hpp                         Pending                                      hostpath-csi-basic   5m2s
prime-3148ae76-3873-42cf-91c7-5a86ac2324ea   Pending                                      hostpath-csi-basic   5m1s

$ oc get vm 
NAME                   AGE    STATUS         READY
vm-cirros-source-hpp   5m7s   Provisioning   False

$ oc get vmi 
NAME                   AGE     PHASE        IP    NODENAME   READY
vm-cirros-source-hpp   5m11s   Scheduling                    False

$ oc get pods
NAME                                                  READY   STATUS              RESTARTS   AGE
importer-prime-3148ae76-3873-42cf-91c7-5a86ac2324ea   0/1     ContainerCreating   0          5m21s
virt-launcher-vm-cirros-source-hpp-5tj99              0/1     Pending             0          5m21s

$ oc describe pod importer-prime-3148ae76-3873-42cf-91c7-5a86ac2324ea | grep Events -A 10
Events:
  Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  41s (x26 over 5m56s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[cdi-data-vol], unattached volumes=[], failed to process volumes=[cdi-data-vol]: error processing PVC default/prime-3148ae76-3873-42cf-91c7-5a86ac2324ea: PVC is not bound

$ oc describe pod virt-launcher-vm-cirros-source-hpp-5tj99  | grep Events -A 10
Events:          <none>


Expected results:

DV is Pending or some other error state
PVC is Pending (as it is)
PVC-prime is Pending (as it is)
VM is WaitingForVolumeBinding or some other state, indicating there's an issue 
VMI is Pending
virt-launcher pod has an Event, indicating there's an issue


How it was in 4.13, before the populators:

$ oc get dv 
NAME      PHASE                  PROGRESS   RESTARTS   AGE
dv-5717   WaitForFirstConsumer   N/A                   84m

$ oc get pvc 
NAME      STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
dv-5717   Pending                                      hostpath-csi-basic   84m

$ oc get vm 
NAME                         AGE   STATUS                    READY
vm-5717-1696165923-4705606   84m   WaitingForVolumeBinding   False

$ oc get vmi
NAME                         AGE   PHASE     IP    NODENAME   READY
vm-5717-1696165923-4705606   84m   Pending                    False

$ oc get pods
NAME                                             READY   STATUS    RESTARTS   AGE
virt-launcher-vm-5717-1696165923-4705606-r8fsj   0/1     Pending   0          84m

$ oc describe pods virt-launcher-vm-5717-1696165923-4705606-r8fsj | grep Events -A 10
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  5m21s (x8 over 76m)  default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition


Additional info:

Comment 3 Jenia Peimer 2023-12-04 14:23:27 UTC
So with the fix, VM is 'WaitingForVolumeBinding' - as expected, VMI is 'Scheduling' (but we would expect it to be 'Pending')

$ oc get vm -A
NAMESPACE                     NAME                        AGE     STATUS                    READY
hpp-test-hpp-node-placement   vm-5717-1701699225-103285   5m34s   WaitingForVolumeBinding   False

$ oc get vmi -A
NAMESPACE                     NAME                        AGE     PHASE        IP    NODENAME   READY
hpp-test-hpp-node-placement   vm-5717-1701699225-103285   5m38s   Scheduling                    False

$ oc get dv -n hpp-test-hpp-node-placement
NAME      PHASE               PROGRESS   RESTARTS   AGE
dv-5717   PendingPopulation   N/A                   5m47s

$ oc get pvc -n hpp-test-hpp-node-placement
NAME                                         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
dv-5717                                      Pending                                      hostpath-csi-basic   5m55s
prime-c610ad92-b21d-4703-b219-111d11fd5f18   Pending                                      hostpath-csi-basic   5m55s

$ oc get pods -n hpp-test-hpp-node-placement
NAME                                                  READY   STATUS              RESTARTS   AGE
importer-prime-c610ad92-b21d-4703-b219-111d11fd5f18   0/1     ContainerCreating   0          6m13s
virt-launcher-vm-5717-1701699225-103285-68xbp         0/1     Pending             0          6m13s

$ oc describe pods -n hpp-test-hpp-node-placement importer-prime-c610ad92-b21d-4703-b219-111d11fd5f18  | grep Events -A 10
Events:
  Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  84s (x26 over 6m37s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[cdi-data-vol], unattached volumes=[], failed to process volumes=[cdi-data-vol]: error processing PVC hpp-test-hpp-node-placement/prime-c610ad92-b21d-4703-b219-111d11fd5f18: PVC is not bound


Note You need to log in before you can comment on or make changes to this bug.