Bug 1918294 - VM created from template when OCS is default SC fails to start on "source volumeMode (Block) and target volumeMode (Filesystem) do not match"
Summary: VM created from template when OCS is default SC fails to start on "source vol...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: SSP
Version: 2.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.10.0
Assignee: Dominik Holler
QA Contact: Ruth Netser
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-20 12:00 UTC by Ruth Netser
Modified: 2022-03-16 15:51 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-16 15:50:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2022:0947 0 None None None 2022-03-16 15:51:05 UTC

Description Ruth Netser 2021-01-20 12:00:26 UTC
Description of problem:
When OCS is set as default storage class and a VM is created from template (i.e using a golden image as source PVC), VM fails to start.
This is the default flow for VM from golden images using the CLI.

Version-Release number of selected component (if applicable):
virt-cdi-operator-container-v2.6.0-19

How reproducible:
100%

Steps to Reproduce:
1. Set OCS as default SC
2. Create an OCS golden image DV under openshift-virtualization-os-images ns 
3. Create a VM from common templates

Actual results:
VM fails to start:
  Warning  FailedDataVolumeCreate  3m27s (x20 over 58m)  virtualmachine-controller  Error creating DataVolume vm-from-golden-image-1611140157-9784682: admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request:  source volumeMode (Block) and target volumeMode (Filesystem) do not match

- When a VM is created from common templates, only accessModes is provided in the template under dataVolumeTemplates.
The Storage class and volumeMode should be automatically filled based on cluster defaults (deafult SC and kubevirt-storage-class-defaults); in this case OCS configuration.

  dataVolumeTemplates:
  - apiVersion: cdi.kubevirt.io/v1beta1
    kind: DataVolume
    metadata:
      name: vm-from-golden-image-1611140157-9784682
    spec:
      pvc:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 30Gi
      source:
        pvc:
          name: fedora33
          namespace: openshift-virtualization-os-images


Expected results:
VM should be created and started successfully.

Additional info:

=============== Source DV ==============================
$ oc describe dv -n openshift-virtualization-os-images fedora33
Name:         fedora33
Namespace:    openshift-virtualization-os-images
Labels:       <none>
Annotations:  <none>
API Version:  cdi.kubevirt.io/v1beta1
Kind:         DataVolume
Metadata:
  Creation Timestamp:  2021-01-20T10:49:18Z
  Generation:          198
  Managed Fields:
    API Version:  cdi.kubevirt.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:contentType:
        f:pvc:
          .:
          f:accessModes:
          f:resources:
            .:
            f:requests:
              .:
              f:storage:
          f:storageClassName:
          f:volumeMode:
        f:source:
          .:
          f:http:
            .:
            f:url:
    Manager:      OpenAPI-Generator
    Operation:    Update
    Time:         2021-01-20T10:49:18Z
    API Version:  cdi.kubevirt.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:phase:
        f:progress:
    Manager:         virt-cdi-controller
    Operation:       Update
    Time:            2021-01-20T10:49:18Z
  Resource Version:  1172823
  Self Link:         /apis/cdi.kubevirt.io/v1beta1/namespaces/openshift-virtualization-os-images/datavolumes/fedora33
  UID:               c45cc67c-b35a-46b9-a2d8-6b9cc841f39d
Spec:
  Content Type:  kubevirt
  Pvc:
    Access Modes:
      ReadWriteMany
    Resources:
      Requests:
        Storage:         10Gi
    Storage Class Name:  ocs-storagecluster-ceph-rbd
    Volume Mode:         Block
  Source:
    Http:
      URL:  http://cnv-qe-server.rhevdev.lab.eng.rdu2.redhat.com/files/cnv-tests/fedora-images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2
Status:
  Conditions:
    Last Heartbeat Time:   2021-01-20T10:49:19Z
    Last Transition Time:  2021-01-20T10:49:19Z
    Message:               PVC fedora33 Bound
    Reason:                Bound
    Status:                True
    Type:                  Bound
    Last Heartbeat Time:   2021-01-20T10:55:56Z
    Last Transition Time:  2021-01-20T10:55:56Z
    Status:                True
    Type:                  Ready
    Last Heartbeat Time:   2021-01-20T10:55:56Z
    Last Transition Time:  2021-01-20T10:55:56Z
    Message:               Import Complete
    Reason:                Completed
    Status:                False
    Type:                  Running
  Phase:                   Succeeded
  Progress:                100.0%
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  Pending           66m   datavolume-controller  PVC fedora33 Pending
  Normal  Bound             66m   datavolume-controller  PVC fedora33 Bound
  Normal  ImportScheduled   66m   datavolume-controller  Import into fedora33 scheduled
  Normal  ImportInProgress  66m   datavolume-controller  Import into fedora33 in progress
  Normal  ImportSucceeded   60m   datavolume-controller  Successfully imported into PVC fedora33


================ Storage class config =============================
$ oc get sc
NAME                                    PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-ceph                         manila.csi.openstack.org                Delete          Immediate              false                  24h
hostpath-provisioner                    kubevirt.io/hostpath-provisioner        Delete          WaitForFirstConsumer   false                  23h
local-block                             kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  23h
nfs                                     kubernetes.io/no-provisioner            Delete          Immediate              false                  23h
ocs-storagecluster-ceph-rbd (default)   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   23h
ocs-storagecluster-ceph-rgw             openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  23h
ocs-storagecluster-cephfs               openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   23h
openshift-storage.noobaa.io             openshift-storage.noobaa.io/obc         Delete          Immediate              false                  7h6m
standard                                kubernetes.io/cinder                    Delete          WaitForFirstConsumer   true                   24h
standard-csi                            cinder.csi.openstack.org                Delete          WaitForFirstConsumer   true                   24h

====================== kubevirt-storage-class-defaults =======================
$ oc get cm -n openshift-cnv kubevirt-storage-class-defaults -oyaml
apiVersion: v1
data:
  accessMode: ReadWriteOnce
  local-sc.accessMode: ReadWriteOnce
  local-sc.volumeMode: Filesystem
  ocs-storagecluster-ceph-rbd.accessMode: ReadWriteMany
  ocs-storagecluster-ceph-rbd.volumeMode: Block
  volumeMode: Filesystem
kind: ConfigMap


================ VM yaml =============================
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  annotations:
    vm.kubevirt.io/flavor: tiny
    vm.kubevirt.io/os: fedora
    vm.kubevirt.io/validations: "[\n  {\n    \"name\": \"minimal-required-memory\"\
      ,\n    \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n   \
      \ \"rule\": \"integer\",\n    \"message\": \"This VM requires more memory.\"\
      ,\n    \"min\": 1073741824\n  }\n]\n"
    vm.kubevirt.io/workload: server
  labels:
    app: vm-from-golden-image-1611140157-9784682
    vm.kubevirt.io/template: fedora-server-tiny
    vm.kubevirt.io/template.namespace: openshift
    vm.kubevirt.io/template.revision: '1'
    vm.kubevirt.io/template.version: v0.13.1
  name: vm-from-golden-image-1611140157-9784682
spec:
  dataVolumeTemplates:
  - apiVersion: cdi.kubevirt.io/v1beta1
    kind: DataVolume
    metadata:
      name: vm-from-golden-image-1611140157-9784682
    spec:
      pvc:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 30Gi
      source:
        pvc:
          name: fedora33
          namespace: openshift-virtualization-os-images
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/domain: vm-from-golden-image-1611140157-9784682
        kubevirt.io/size: tiny
        kubevirt.io/vm: vm-from-golden-image-1611140157-9784682
    spec:
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
          - disk:
              bus: virtio
            name: vm-from-golden-image-1611140157-9784682
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - masquerade: {}
            name: default
          networkInterfaceMultiqueue: true
          rng: {}
        machine:
          type: pc-q35-rhel8.3.0
        resources:
          requests:
            memory: 1Gi
      evictionStrategy: LiveMigrate
      networks:
      - name: default
        pod: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - dataVolume:
          name: vm-from-golden-image-1611140157-9784682
        name: vm-from-golden-image-1611140157-9784682
      - cloudInitNoCloud:
          userData: '#cloud-config

            user: fedora

            password: fedora

            chpasswd: { expire: False }'
        name: cloudinitdisk

================ VM describe =============================
$ oc describe vm -n templates-golden-images-test-common-templates-data-volumes
Name:         vm-from-golden-image-1611140157-9784682
Namespace:    templates-golden-images-test-common-templates-data-volumes
Labels:       app=vm-from-golden-image-1611140157-9784682
              vm.kubevirt.io/template=fedora-server-tiny
              vm.kubevirt.io/template.namespace=openshift
              vm.kubevirt.io/template.revision=1
              vm.kubevirt.io/template.version=v0.13.1
Annotations:  kubevirt.io/latest-observed-api-version: v1alpha3
              kubevirt.io/storage-observed-api-version: v1alpha3
              vm.kubevirt.io/flavor: tiny
              vm.kubevirt.io/os: fedora
              vm.kubevirt.io/validations:
                [
                  {
                    "name": "minimal-required-memory",
                    "path": "jsonpath::.spec.domain.resources.requests.memory",
                    "rule": "integer",
                    "message": "This VM requires more memory.",
                    "min": 1073741824
                  }
                ]
              vm.kubevirt.io/workload: server
API Version:  kubevirt.io/v1alpha3
Kind:         VirtualMachine
Metadata:
  Creation Timestamp:  2021-01-20T10:55:58Z
  Finalizers:
    k8s.v1.cni.cncf.io/kubeMacPool
  Generation:  2
  Managed Fields:
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:vm.kubevirt.io/flavor:
          f:vm.kubevirt.io/os:
          f:vm.kubevirt.io/validations:
          f:vm.kubevirt.io/workload:
        f:labels:
          .:
          f:app:
          f:vm.kubevirt.io/template:
          f:vm.kubevirt.io/template.namespace:
          f:vm.kubevirt.io/template.revision:
          f:vm.kubevirt.io/template.version:
      f:spec:
        .:
        f:dataVolumeTemplates:
        f:template:
          .:
          f:metadata:
            .:
            f:labels:
              .:
              f:kubevirt.io/domain:
              f:kubevirt.io/size:
              f:kubevirt.io/vm:
          f:spec:
            .:
            f:domain:
              .:
              f:cpu:
                .:
                f:cores:
                f:sockets:
                f:threads:
              f:devices:
                .:
                f:disks:
                f:interfaces:
                f:networkInterfaceMultiqueue:
                f:rng:
              f:machine:
                .:
                f:type:
              f:resources:
                .:
                f:requests:
                  .:
                  f:memory:
            f:evictionStrategy:
            f:networks:
            f:terminationGracePeriodSeconds:
            f:volumes:
    Manager:      OpenAPI-Generator
    Operation:    Update
    Time:         2021-01-20T10:55:58Z
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:running:
    Manager:      virt-api
    Operation:    Update
    Time:         2021-01-20T10:55:59Z
    API Version:  kubevirt.io/v1alpha3
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubevirt.io/latest-observed-api-version:
          f:kubevirt.io/storage-observed-api-version:
      f:status:
        .:
        f:conditions:
        f:volumeSnapshotStatuses:
    Manager:         virt-controller
    Operation:       Update
    Time:            2021-01-20T10:55:59Z
  Resource Version:  1172874
  Self Link:         /apis/kubevirt.io/v1alpha3/namespaces/templates-golden-images-test-common-templates-data-volumes/virtualmachines/vm-from-golden-image-1611140157-9784682
  UID:               ca850194-4bf6-452a-a902-2f6cd54e58ae
Spec:
  Data Volume Templates:
    API Version:  cdi.kubevirt.io/v1beta1
    Kind:         DataVolume
    Metadata:
      Creation Timestamp:  <nil>
      Name:                vm-from-golden-image-1611140157-9784682
    Spec:
      Pvc:
        Access Modes:
          ReadWriteMany
        Resources:
          Requests:
            Storage:  30Gi
      Source:
        Pvc:
          Name:       fedora33
          Namespace:  openshift-virtualization-os-images
  Running:            true
  Template:
    Metadata:
      Creation Timestamp:  <nil>
      Labels:
        kubevirt.io/domain:  vm-from-golden-image-1611140157-9784682
        kubevirt.io/size:    tiny
        kubevirt.io/vm:      vm-from-golden-image-1611140157-9784682
    Spec:
      Domain:
        Cpu:
          Cores:    1
          Sockets:  1
          Threads:  1
        Devices:
          Disks:
            Disk:
              Bus:  virtio
            Name:   vm-from-golden-image-1611140157-9784682
            Disk:
              Bus:  virtio
            Name:   cloudinitdisk
          Interfaces:
            Mac Address:  02:7e:45:00:00:68
            Masquerade:
            Name:                        default
          Network Interface Multiqueue:  true
          Rng:
        Machine:
          Type:  pc-q35-rhel8.3.0
        Resources:
          Requests:
            Memory:       1Gi
      Eviction Strategy:  LiveMigrate
      Networks:
        Name:  default
        Pod:
      Termination Grace Period Seconds:  180
      Volumes:
        Data Volume:
          Name:  vm-from-golden-image-1611140157-9784682
        Name:    vm-from-golden-image-1611140157-9784682
        Cloud Init No Cloud:
          User Data:  #cloud-config
user: fedora
password: fedora
chpasswd: { expire: False }
        Name:  cloudinitdisk
Status:
  Conditions:
    Last Probe Time:       <nil>
    Last Transition Time:  2021-01-20T10:55:59Z
    Message:               Failed to create DataVolume: admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request:  source volumeMode (Block) and target volumeMode (Filesystem) do not match
    Reason:                FailedCreate
    Status:                True
    Type:                  Failure
  Volume Snapshot Statuses:
    Enabled:  false
    Name:     vm-from-golden-image-1611140157-9784682
    Reason:   DV 'templates-golden-images-test-common-templates-data-volumes/vm-from-golden-image-1611140157-9784682' not found
    Enabled:  false
    Name:     cloudinitdisk
    Reason:   Volume type does not suport snapshots
Events:
  Type     Reason                  Age                  From                       Message
  ----     ------                  ----                 ----                       -------
  Warning  FailedDataVolumeCreate  104s (x18 over 23m)  virtualmachine-controller  Error creating DataVolume vm-from-golden-image-1611140157-9784682: admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request:  source volumeMode (Block) and target volumeMode (Filesystem) do not match



=============================================

Comment 1 Natalie Gavrielov 2021-01-20 16:09:33 UTC
Omer, when cloning a disk the volume mode of the source should be the same for the target. In this case, how was the target created with the wrong volume mode?

Comment 2 Omer Yahud 2021-01-21 07:57:41 UTC
(In reply to Natalie Gavrielov from comment #1)
> Omer, when cloning a disk the volume mode of the source should be the same
> for the target. In this case, how was the target created with the wrong
> volume mode?

I do not know, the templates themselves do not specify the volumeMode (as you can see in the describe output, no volumeMode is specified for the target DV), they leave it for CDI to default using the kubevirt-storage-class-defaults configmap?

Comment 3 Adam Litke 2021-01-21 17:06:02 UTC
CDI does not (yet) manage this value.  It is a feature that we are adding soon but currently no volume mode means Filesystem.  The kubevirt-storage-class-defaults ConfigMap is used only by the UI when creating VM Disks.

Comment 9 Yaacov Zamir 2021-01-25 10:23:48 UTC
> @tjelinek Can you confirm that the UI overrides the volumeMode, storageClass and accessMode of the target PVC, 
so adding template parameters to those values should not break the VM creation flow?

yes, the UI overrides the volumeMode, storageClass and accessMode of the target PVC

The UI:
a - If source is PVC, try to use same volumeMode, storageClass as source PVC
b - o/w, try to use the suggested volumeMode and accessMode suggested for the storageClass in the cluster config map

Comment 12 Ruth Netser 2021-12-16 12:21:20 UTC
Verified with CNV 4.10.0 (SSP v4.10.0-21)
.storage api is used in templates.
Can create a VM from template without specifying any storage parameters (as long as the cluster has a default storage class configured).

Comment 17 errata-xmlrpc 2022-03-16 15:50:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0947


Note You need to log in before you can comment on or make changes to this bug.