Description of problem: When OCS is set as default storage class and a VM is created from template (i.e using a golden image as source PVC), VM fails to start. This is the default flow for VM from golden images using the CLI. Version-Release number of selected component (if applicable): virt-cdi-operator-container-v2.6.0-19 How reproducible: 100% Steps to Reproduce: 1. Set OCS as default SC 2. Create an OCS golden image DV under openshift-virtualization-os-images ns 3. Create a VM from common templates Actual results: VM fails to start: Warning FailedDataVolumeCreate 3m27s (x20 over 58m) virtualmachine-controller Error creating DataVolume vm-from-golden-image-1611140157-9784682: admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request: source volumeMode (Block) and target volumeMode (Filesystem) do not match - When a VM is created from common templates, only accessModes is provided in the template under dataVolumeTemplates. The Storage class and volumeMode should be automatically filled based on cluster defaults (deafult SC and kubevirt-storage-class-defaults); in this case OCS configuration. dataVolumeTemplates: - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: vm-from-golden-image-1611140157-9784682 spec: pvc: accessModes: - ReadWriteMany resources: requests: storage: 30Gi source: pvc: name: fedora33 namespace: openshift-virtualization-os-images Expected results: VM should be created and started successfully. Additional info: =============== Source DV ============================== $ oc describe dv -n openshift-virtualization-os-images fedora33 Name: fedora33 Namespace: openshift-virtualization-os-images Labels: <none> Annotations: <none> API Version: cdi.kubevirt.io/v1beta1 Kind: DataVolume Metadata: Creation Timestamp: 2021-01-20T10:49:18Z Generation: 198 Managed Fields: API Version: cdi.kubevirt.io/v1beta1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:contentType: f:pvc: .: f:accessModes: f:resources: .: f:requests: .: f:storage: f:storageClassName: f:volumeMode: f:source: .: f:http: .: f:url: Manager: OpenAPI-Generator Operation: Update Time: 2021-01-20T10:49:18Z API Version: cdi.kubevirt.io/v1beta1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:conditions: f:phase: f:progress: Manager: virt-cdi-controller Operation: Update Time: 2021-01-20T10:49:18Z Resource Version: 1172823 Self Link: /apis/cdi.kubevirt.io/v1beta1/namespaces/openshift-virtualization-os-images/datavolumes/fedora33 UID: c45cc67c-b35a-46b9-a2d8-6b9cc841f39d Spec: Content Type: kubevirt Pvc: Access Modes: ReadWriteMany Resources: Requests: Storage: 10Gi Storage Class Name: ocs-storagecluster-ceph-rbd Volume Mode: Block Source: Http: URL: http://cnv-qe-server.rhevdev.lab.eng.rdu2.redhat.com/files/cnv-tests/fedora-images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 Status: Conditions: Last Heartbeat Time: 2021-01-20T10:49:19Z Last Transition Time: 2021-01-20T10:49:19Z Message: PVC fedora33 Bound Reason: Bound Status: True Type: Bound Last Heartbeat Time: 2021-01-20T10:55:56Z Last Transition Time: 2021-01-20T10:55:56Z Status: True Type: Ready Last Heartbeat Time: 2021-01-20T10:55:56Z Last Transition Time: 2021-01-20T10:55:56Z Message: Import Complete Reason: Completed Status: False Type: Running Phase: Succeeded Progress: 100.0% Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pending 66m datavolume-controller PVC fedora33 Pending Normal Bound 66m datavolume-controller PVC fedora33 Bound Normal ImportScheduled 66m datavolume-controller Import into fedora33 scheduled Normal ImportInProgress 66m datavolume-controller Import into fedora33 in progress Normal ImportSucceeded 60m datavolume-controller Successfully imported into PVC fedora33 ================ Storage class config ============================= $ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE csi-manila-ceph manila.csi.openstack.org Delete Immediate false 24h hostpath-provisioner kubevirt.io/hostpath-provisioner Delete WaitForFirstConsumer false 23h local-block kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 23h nfs kubernetes.io/no-provisioner Delete Immediate false 23h ocs-storagecluster-ceph-rbd (default) openshift-storage.rbd.csi.ceph.com Delete Immediate true 23h ocs-storagecluster-ceph-rgw openshift-storage.ceph.rook.io/bucket Delete Immediate false 23h ocs-storagecluster-cephfs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 23h openshift-storage.noobaa.io openshift-storage.noobaa.io/obc Delete Immediate false 7h6m standard kubernetes.io/cinder Delete WaitForFirstConsumer true 24h standard-csi cinder.csi.openstack.org Delete WaitForFirstConsumer true 24h ====================== kubevirt-storage-class-defaults ======================= $ oc get cm -n openshift-cnv kubevirt-storage-class-defaults -oyaml apiVersion: v1 data: accessMode: ReadWriteOnce local-sc.accessMode: ReadWriteOnce local-sc.volumeMode: Filesystem ocs-storagecluster-ceph-rbd.accessMode: ReadWriteMany ocs-storagecluster-ceph-rbd.volumeMode: Block volumeMode: Filesystem kind: ConfigMap ================ VM yaml ============================= apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: annotations: vm.kubevirt.io/flavor: tiny vm.kubevirt.io/os: fedora vm.kubevirt.io/validations: "[\n {\n \"name\": \"minimal-required-memory\"\ ,\n \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n \ \ \"rule\": \"integer\",\n \"message\": \"This VM requires more memory.\"\ ,\n \"min\": 1073741824\n }\n]\n" vm.kubevirt.io/workload: server labels: app: vm-from-golden-image-1611140157-9784682 vm.kubevirt.io/template: fedora-server-tiny vm.kubevirt.io/template.namespace: openshift vm.kubevirt.io/template.revision: '1' vm.kubevirt.io/template.version: v0.13.1 name: vm-from-golden-image-1611140157-9784682 spec: dataVolumeTemplates: - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: vm-from-golden-image-1611140157-9784682 spec: pvc: accessModes: - ReadWriteMany resources: requests: storage: 30Gi source: pvc: name: fedora33 namespace: openshift-virtualization-os-images running: false template: metadata: labels: kubevirt.io/domain: vm-from-golden-image-1611140157-9784682 kubevirt.io/size: tiny kubevirt.io/vm: vm-from-golden-image-1611140157-9784682 spec: domain: cpu: cores: 1 sockets: 1 threads: 1 devices: disks: - disk: bus: virtio name: vm-from-golden-image-1611140157-9784682 - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default networkInterfaceMultiqueue: true rng: {} machine: type: pc-q35-rhel8.3.0 resources: requests: memory: 1Gi evictionStrategy: LiveMigrate networks: - name: default pod: {} terminationGracePeriodSeconds: 180 volumes: - dataVolume: name: vm-from-golden-image-1611140157-9784682 name: vm-from-golden-image-1611140157-9784682 - cloudInitNoCloud: userData: '#cloud-config user: fedora password: fedora chpasswd: { expire: False }' name: cloudinitdisk ================ VM describe ============================= $ oc describe vm -n templates-golden-images-test-common-templates-data-volumes Name: vm-from-golden-image-1611140157-9784682 Namespace: templates-golden-images-test-common-templates-data-volumes Labels: app=vm-from-golden-image-1611140157-9784682 vm.kubevirt.io/template=fedora-server-tiny vm.kubevirt.io/template.namespace=openshift vm.kubevirt.io/template.revision=1 vm.kubevirt.io/template.version=v0.13.1 Annotations: kubevirt.io/latest-observed-api-version: v1alpha3 kubevirt.io/storage-observed-api-version: v1alpha3 vm.kubevirt.io/flavor: tiny vm.kubevirt.io/os: fedora vm.kubevirt.io/validations: [ { "name": "minimal-required-memory", "path": "jsonpath::.spec.domain.resources.requests.memory", "rule": "integer", "message": "This VM requires more memory.", "min": 1073741824 } ] vm.kubevirt.io/workload: server API Version: kubevirt.io/v1alpha3 Kind: VirtualMachine Metadata: Creation Timestamp: 2021-01-20T10:55:58Z Finalizers: k8s.v1.cni.cncf.io/kubeMacPool Generation: 2 Managed Fields: API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:vm.kubevirt.io/flavor: f:vm.kubevirt.io/os: f:vm.kubevirt.io/validations: f:vm.kubevirt.io/workload: f:labels: .: f:app: f:vm.kubevirt.io/template: f:vm.kubevirt.io/template.namespace: f:vm.kubevirt.io/template.revision: f:vm.kubevirt.io/template.version: f:spec: .: f:dataVolumeTemplates: f:template: .: f:metadata: .: f:labels: .: f:kubevirt.io/domain: f:kubevirt.io/size: f:kubevirt.io/vm: f:spec: .: f:domain: .: f:cpu: .: f:cores: f:sockets: f:threads: f:devices: .: f:disks: f:interfaces: f:networkInterfaceMultiqueue: f:rng: f:machine: .: f:type: f:resources: .: f:requests: .: f:memory: f:evictionStrategy: f:networks: f:terminationGracePeriodSeconds: f:volumes: Manager: OpenAPI-Generator Operation: Update Time: 2021-01-20T10:55:58Z API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:spec: f:running: Manager: virt-api Operation: Update Time: 2021-01-20T10:55:59Z API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: f:kubevirt.io/latest-observed-api-version: f:kubevirt.io/storage-observed-api-version: f:status: .: f:conditions: f:volumeSnapshotStatuses: Manager: virt-controller Operation: Update Time: 2021-01-20T10:55:59Z Resource Version: 1172874 Self Link: /apis/kubevirt.io/v1alpha3/namespaces/templates-golden-images-test-common-templates-data-volumes/virtualmachines/vm-from-golden-image-1611140157-9784682 UID: ca850194-4bf6-452a-a902-2f6cd54e58ae Spec: Data Volume Templates: API Version: cdi.kubevirt.io/v1beta1 Kind: DataVolume Metadata: Creation Timestamp: <nil> Name: vm-from-golden-image-1611140157-9784682 Spec: Pvc: Access Modes: ReadWriteMany Resources: Requests: Storage: 30Gi Source: Pvc: Name: fedora33 Namespace: openshift-virtualization-os-images Running: true Template: Metadata: Creation Timestamp: <nil> Labels: kubevirt.io/domain: vm-from-golden-image-1611140157-9784682 kubevirt.io/size: tiny kubevirt.io/vm: vm-from-golden-image-1611140157-9784682 Spec: Domain: Cpu: Cores: 1 Sockets: 1 Threads: 1 Devices: Disks: Disk: Bus: virtio Name: vm-from-golden-image-1611140157-9784682 Disk: Bus: virtio Name: cloudinitdisk Interfaces: Mac Address: 02:7e:45:00:00:68 Masquerade: Name: default Network Interface Multiqueue: true Rng: Machine: Type: pc-q35-rhel8.3.0 Resources: Requests: Memory: 1Gi Eviction Strategy: LiveMigrate Networks: Name: default Pod: Termination Grace Period Seconds: 180 Volumes: Data Volume: Name: vm-from-golden-image-1611140157-9784682 Name: vm-from-golden-image-1611140157-9784682 Cloud Init No Cloud: User Data: #cloud-config user: fedora password: fedora chpasswd: { expire: False } Name: cloudinitdisk Status: Conditions: Last Probe Time: <nil> Last Transition Time: 2021-01-20T10:55:59Z Message: Failed to create DataVolume: admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request: source volumeMode (Block) and target volumeMode (Filesystem) do not match Reason: FailedCreate Status: True Type: Failure Volume Snapshot Statuses: Enabled: false Name: vm-from-golden-image-1611140157-9784682 Reason: DV 'templates-golden-images-test-common-templates-data-volumes/vm-from-golden-image-1611140157-9784682' not found Enabled: false Name: cloudinitdisk Reason: Volume type does not suport snapshots Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedDataVolumeCreate 104s (x18 over 23m) virtualmachine-controller Error creating DataVolume vm-from-golden-image-1611140157-9784682: admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request: source volumeMode (Block) and target volumeMode (Filesystem) do not match =============================================
Omer, when cloning a disk the volume mode of the source should be the same for the target. In this case, how was the target created with the wrong volume mode?
(In reply to Natalie Gavrielov from comment #1) > Omer, when cloning a disk the volume mode of the source should be the same > for the target. In this case, how was the target created with the wrong > volume mode? I do not know, the templates themselves do not specify the volumeMode (as you can see in the describe output, no volumeMode is specified for the target DV), they leave it for CDI to default using the kubevirt-storage-class-defaults configmap?
CDI does not (yet) manage this value. It is a feature that we are adding soon but currently no volume mode means Filesystem. The kubevirt-storage-class-defaults ConfigMap is used only by the UI when creating VM Disks.
> @tjelinek Can you confirm that the UI overrides the volumeMode, storageClass and accessMode of the target PVC, so adding template parameters to those values should not break the VM creation flow? yes, the UI overrides the volumeMode, storageClass and accessMode of the target PVC The UI: a - If source is PVC, try to use same volumeMode, storageClass as source PVC b - o/w, try to use the suggested volumeMode and accessMode suggested for the storageClass in the cluster config map
Verified with CNV 4.10.0 (SSP v4.10.0-21) .storage api is used in templates. Can create a VM from template without specifying any storage parameters (as long as the cluster has a default storage class configured).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947