Bug 2162333

Summary:

PVC created using non default storage class on fresh cluster

Product:

Container Native Virtualization (CNV)

Reporter:

Roni Kishner <rkishner>

Component:

Storage

Assignee:

Adam Litke <alitke>

Status:

CLOSED ERRATA

QA Contact:

Yan Du <yadu>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

4.13.0

CC:

dollierp, gouyang, jpeimer, yadu

Target Milestone:

---

Target Release:

4.13.0

Hardware:

All

OS:

All

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2023-05-18 02:56:40 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
storage classes definitions	none

Description Roni Kishner 2023-01-19 11:28:27 UTC

Description of problem:
When creating a fresh 4.13 cluster, the PVCs created from DataImportCron are being created using the "standard-csi" storage class instead of the default storage class


Version-Release number of selected component (if applicable):
v4.13.0-13

How reproducible:
100% (seen on 5 different clusters creation)

Steps to Reproduce:
1. Create a new cluster with CNV
2. oc get pvc -n openshift-virtualization-os-images

Actual results:
The PVCs STORAGECLASS column show "standard-csi"

Expected results:
The PVCs STORAGECLASS column show the default storage class (hostpath-csi-basic in this case)


Additional info:
This might be because of PVC being created before the storage class default is set, or the CNV installation deploy script changed?
need to understand what changed from 4.12

Comment 1 Yan Du 2023-02-06 02:15:08 UTC

Hi, Denis
Could you please help to check whether the installation deploy script changed in 4.13?

Comment 2 Denis Ollier 2023-02-06 08:56:50 UTC

Hi,

AFAIK, our installation script didn't change in 4.13.

Comment 3 Roni Kishner 2023-02-10 07:40:24 UTC

Created attachment 1943261 [details]
storage classes definitions

This looks like a deeper problem then a new cluster. I just created a new VM from a template without defining the storage class and it used "standard-csi" instead of the default storage class which is "hostpath-csi-basic".
attaching a list of both storage classes definition

Comment 4 Roni Kishner 2023-02-12 14:01:15 UTC

(In reply to Roni Kishner from comment #3)
> Created attachment 1943261 [details]
> storage classes definitions
> 
> This looks like a deeper problem then a new cluster. I just created a new VM
> from a template without defining the storage class and it used
> "standard-csi" instead of the default storage class which is
> "hostpath-csi-basic".
> attaching a list of both storage classes definition

Ignore the comment, it was my mistake, we create a new DV/PVC from dataSource but the template still looks at the original PVC created so it used a wrong storageclass

Comment 5 Jenia Peimer 2023-02-13 11:06:25 UTC

TL DR: It's because of the cluster setup flow (maybe we need to unset 'default' of 'standard-csi' earlier in the flow)


From my findings of 4.13 deployment flow:

Once we deployed OCP, but not yet deployed CNV, 'standard-csi' is a default storage:

[cloud-user@ocp-psi-executor ~]$ oc get sc 
NAME                     PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-ceph          manila.csi.openstack.org   Delete          Immediate              false                  60m
standard-csi (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   61m

Then we deploy CNV and data import cron DVs start to import:

[cloud-user@ocp-psi-executor ~]$ oc get pvc -n openshift-virtualization-os-images
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
centos-stream8-2f16c067b974   Bound    pvc-52e914ea-5d25-4fc7-8b3e-678bd01dc171   30Gi       RWO            standard-csi   84s
centos-stream9-23d7d288eb34   Bound    pvc-661ed6ba-6915-4d23-a6d2-d76e58d7991c   30Gi       RWO            standard-csi   86s
centos7-680e9b4e0fba          Bound    pvc-6caab12b-621c-428c-b228-cdb861021d43   30Gi       RWO            standard-csi   85s
fedora-56ccabc01cbe           Bound    pvc-8ea4e2b4-2fa3-48ed-b130-b416856aa7ba   30Gi       RWO            standard-csi   81s
rhel8-e004563ff8cf            Bound    pvc-9ead3860-e602-40fa-937b-ee18f3dd6496   30Gi       RWO            standard-csi   87s
rhel9-2a70e7a1fadc            Bound    pvc-1a415f8f-5119-4a82-ae9f-1fe6a09888b0   30Gi       RWO            standard-csi   87s

And that exact moment the storage classes are:

[cloud-user@ocp-psi-executor ~]$ oc get sc -w
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-ceph               manila.csi.openstack.org                Delete          Immediate              false                  89m
local-block-hpp               kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  21m
local-block-ocs               kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  21m
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   11m
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   11m
standard-csi (default)        cinder.csi.openstack.org                Delete          WaitForFirstConsumer   true                   90m


'standard-csi' is a default, 
'hostpath-csi-basic' is not yet deployed,
so data import cron DVs were imported as expected and used the default storage class.


In the fresh 4.12 cluster, it also picked the 'standard-csi' storage class:
OCP-4.12.3 + CNV-v4.12.1-39

[cloud-user@ocp-psi-executor ~]$ oc get pvc -n openshift-virtualization-os-images 
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
centos-stream8-2f16c067b974   Bound    pvc-0ed566c0-d997-44e8-b89f-16bd9203a29a   30Gi       RWO            standard-csi   17m
centos-stream9-23d7d288eb34   Bound    pvc-1feca034-833c-4586-b711-d56a8d6d61c6   30Gi       RWO            standard-csi   17m
centos7-680e9b4e0fba          Bound    pvc-0169663c-2d98-479b-98af-7b68de90c1db   30Gi       RWO            standard-csi   17m
fedora-56ccabc01cbe           Bound    pvc-6c7b51c3-f939-4b30-876a-6c689d237a25   30Gi       RWO            standard-csi   17m
rhel8-e004563ff8cf            Bound    pvc-22d6ebb7-c023-48ed-b0c8-f59cb99901e0   30Gi       RWO            standard-csi   17m
rhel9-2a70e7a1fadc            Bound    pvc-ed6dd791-1500-4a60-a05f-c43c16e1e33c   30Gi       RWO            standard-csi   17m

Comment 6 Yan Du 2023-02-14 03:32:20 UTC

In our pervious(old) cluster, I can see it is using hostpath-csi-basic as it is the default storage class. seems now all the clusters are picking up 'standard-csi' storage class.

$ oc get pvc -n openshift-virtualization-os-images 
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
centos-stream8-2f16c067b974   Bound    pvc-cf184104-840b-498f-98b1-3c5e7ff8ee5e   149Gi      RWO            hostpath-csi-basic   3d15h
centos-stream9-23d7d288eb34   Bound    pvc-847f3fd6-be90-4905-a026-c860d75abc29   149Gi      RWO            hostpath-csi-basic   3d15h
centos7-680e9b4e0fba          Bound    pvc-880795ce-a452-4dc9-a118-4ee0e0b0ccd1   149Gi      RWO            hostpath-csi-basic   3d15h
fedora-56ccabc01cbe           Bound    pvc-a48e1355-6cc4-4ef0-a0a7-b99670bb26ad   149Gi      RWO            hostpath-csi-basic   3d15h
rhel8-e004563ff8cf            Bound    pvc-f9854fc3-1049-4c4d-a2d0-c8c27770c57a   149Gi      RWO            hostpath-csi-basic   3d15h
rhel9-2a70e7a1fadc            Bound    pvc-9c0e0425-d279-4f4d-aaad-e88265908ef7   149Gi      RWO            hostpath-csi-basic   3d15h

Denis, Could we unset the standard-csi as default storage class during deployment? or maybe we can adjust the deployment flow? 
Could we set the ocs-storagecluster-ceph-rbd (which CNV suggests to use) as the default storage class?

Comment 9 Yan Du 2023-02-27 10:22:06 UTC

Test on CNV v4.13.0.rhel9-1623

$ oc get pvc -n openshift-virtualization-os-images
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
centos-stream8-2f16c067b974   Bound    pvc-03f9e46c-533a-4227-a739-f9e01e3b4b37   149Gi      RWO            hostpath-csi-basic   17m
centos-stream9-4e7ee7558c01   Bound    pvc-cbce5d78-9d69-4998-b7cf-d92c3613bc41   149Gi      RWO            hostpath-csi-basic   17m
centos7-680e9b4e0fba          Bound    pvc-a8536fef-6bf7-4c37-96ee-bf4cdb79689a   149Gi      RWO            hostpath-csi-basic   17m
fedora-56ccabc01cbe           Bound    pvc-aa7bfcee-0187-4125-974b-2d5c148d3275   149Gi      RWO            hostpath-csi-basic   17m
rhel8-b0bacbb4e426            Bound    pvc-3a62a6da-e011-4f9c-a66b-11923f489af8   149Gi      RWO            hostpath-csi-basic   17m
rhel9-2a70e7a1fadc            Bound    pvc-6432008e-62b7-456d-b3d8-1dbae51279e8   149Gi      RWO            hostpath-csi-basic   17m

Comment 12 errata-xmlrpc 2023-05-18 02:56:40 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.13.0 Images security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3205