Bug 2162333 - PVC created using non default storage class on fresh cluster
Summary: PVC created using non default storage class on fresh cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 4.13.0
Hardware: All
OS: All
unspecified
high
Target Milestone: ---
: 4.13.0
Assignee: Adam Litke
QA Contact: Yan Du
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-01-19 11:28 UTC by Roni Kishner
Modified: 2023-05-18 02:57 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-18 02:56:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
storage classes definitions (940 bytes, text/plain)
2023-02-10 07:40 UTC, Roni Kishner
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CNV-24507 0 None None None 2023-01-19 11:29:51 UTC
Red Hat Product Errata RHSA-2023:3205 0 None None None 2023-05-18 02:57:04 UTC

Description Roni Kishner 2023-01-19 11:28:27 UTC
Description of problem:
When creating a fresh 4.13 cluster, the PVCs created from DataImportCron are being created using the "standard-csi" storage class instead of the default storage class


Version-Release number of selected component (if applicable):
v4.13.0-13

How reproducible:
100% (seen on 5 different clusters creation)

Steps to Reproduce:
1. Create a new cluster with CNV
2. oc get pvc -n openshift-virtualization-os-images

Actual results:
The PVCs STORAGECLASS column show "standard-csi"

Expected results:
The PVCs STORAGECLASS column show the default storage class (hostpath-csi-basic in this case)


Additional info:
This might be because of PVC being created before the storage class default is set, or the CNV installation deploy script changed?
need to understand what changed from 4.12

Comment 1 Yan Du 2023-02-06 02:15:08 UTC
Hi, Denis
Could you please help to check whether the installation deploy script changed in 4.13?

Comment 2 Denis Ollier 2023-02-06 08:56:50 UTC
Hi,

AFAIK, our installation script didn't change in 4.13.

Comment 3 Roni Kishner 2023-02-10 07:40:24 UTC
Created attachment 1943261 [details]
storage classes definitions

This looks like a deeper problem then a new cluster. I just created a new VM from a template without defining the storage class and it used "standard-csi" instead of the default storage class which is "hostpath-csi-basic".
attaching a list of both storage classes definition

Comment 4 Roni Kishner 2023-02-12 14:01:15 UTC
(In reply to Roni Kishner from comment #3)
> Created attachment 1943261 [details]
> storage classes definitions
> 
> This looks like a deeper problem then a new cluster. I just created a new VM
> from a template without defining the storage class and it used
> "standard-csi" instead of the default storage class which is
> "hostpath-csi-basic".
> attaching a list of both storage classes definition

Ignore the comment, it was my mistake, we create a new DV/PVC from dataSource but the template still looks at the original PVC created so it used a wrong storageclass

Comment 5 Jenia Peimer 2023-02-13 11:06:25 UTC
TL DR: It's because of the cluster setup flow (maybe we need to unset 'default' of 'standard-csi' earlier in the flow)


From my findings of 4.13 deployment flow:

Once we deployed OCP, but not yet deployed CNV, 'standard-csi' is a default storage:

[cloud-user@ocp-psi-executor ~]$ oc get sc 
NAME                     PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-ceph          manila.csi.openstack.org   Delete          Immediate              false                  60m
standard-csi (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   61m

Then we deploy CNV and data import cron DVs start to import:

[cloud-user@ocp-psi-executor ~]$ oc get pvc -n openshift-virtualization-os-images
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
centos-stream8-2f16c067b974   Bound    pvc-52e914ea-5d25-4fc7-8b3e-678bd01dc171   30Gi       RWO            standard-csi   84s
centos-stream9-23d7d288eb34   Bound    pvc-661ed6ba-6915-4d23-a6d2-d76e58d7991c   30Gi       RWO            standard-csi   86s
centos7-680e9b4e0fba          Bound    pvc-6caab12b-621c-428c-b228-cdb861021d43   30Gi       RWO            standard-csi   85s
fedora-56ccabc01cbe           Bound    pvc-8ea4e2b4-2fa3-48ed-b130-b416856aa7ba   30Gi       RWO            standard-csi   81s
rhel8-e004563ff8cf            Bound    pvc-9ead3860-e602-40fa-937b-ee18f3dd6496   30Gi       RWO            standard-csi   87s
rhel9-2a70e7a1fadc            Bound    pvc-1a415f8f-5119-4a82-ae9f-1fe6a09888b0   30Gi       RWO            standard-csi   87s

And that exact moment the storage classes are:

[cloud-user@ocp-psi-executor ~]$ oc get sc -w
NAME                          PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
csi-manila-ceph               manila.csi.openstack.org                Delete          Immediate              false                  89m
local-block-hpp               kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  21m
local-block-ocs               kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  21m
ocs-storagecluster-ceph-rbd   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   11m
ocs-storagecluster-cephfs     openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   11m
standard-csi (default)        cinder.csi.openstack.org                Delete          WaitForFirstConsumer   true                   90m


'standard-csi' is a default, 
'hostpath-csi-basic' is not yet deployed,
so data import cron DVs were imported as expected and used the default storage class.


In the fresh 4.12 cluster, it also picked the 'standard-csi' storage class:
OCP-4.12.3 + CNV-v4.12.1-39

[cloud-user@ocp-psi-executor ~]$ oc get pvc -n openshift-virtualization-os-images 
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
centos-stream8-2f16c067b974   Bound    pvc-0ed566c0-d997-44e8-b89f-16bd9203a29a   30Gi       RWO            standard-csi   17m
centos-stream9-23d7d288eb34   Bound    pvc-1feca034-833c-4586-b711-d56a8d6d61c6   30Gi       RWO            standard-csi   17m
centos7-680e9b4e0fba          Bound    pvc-0169663c-2d98-479b-98af-7b68de90c1db   30Gi       RWO            standard-csi   17m
fedora-56ccabc01cbe           Bound    pvc-6c7b51c3-f939-4b30-876a-6c689d237a25   30Gi       RWO            standard-csi   17m
rhel8-e004563ff8cf            Bound    pvc-22d6ebb7-c023-48ed-b0c8-f59cb99901e0   30Gi       RWO            standard-csi   17m
rhel9-2a70e7a1fadc            Bound    pvc-ed6dd791-1500-4a60-a05f-c43c16e1e33c   30Gi       RWO            standard-csi   17m

Comment 6 Yan Du 2023-02-14 03:32:20 UTC
In our pervious(old) cluster, I can see it is using hostpath-csi-basic as it is the default storage class. seems now all the clusters are picking up 'standard-csi' storage class.

$ oc get pvc -n openshift-virtualization-os-images 
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
centos-stream8-2f16c067b974   Bound    pvc-cf184104-840b-498f-98b1-3c5e7ff8ee5e   149Gi      RWO            hostpath-csi-basic   3d15h
centos-stream9-23d7d288eb34   Bound    pvc-847f3fd6-be90-4905-a026-c860d75abc29   149Gi      RWO            hostpath-csi-basic   3d15h
centos7-680e9b4e0fba          Bound    pvc-880795ce-a452-4dc9-a118-4ee0e0b0ccd1   149Gi      RWO            hostpath-csi-basic   3d15h
fedora-56ccabc01cbe           Bound    pvc-a48e1355-6cc4-4ef0-a0a7-b99670bb26ad   149Gi      RWO            hostpath-csi-basic   3d15h
rhel8-e004563ff8cf            Bound    pvc-f9854fc3-1049-4c4d-a2d0-c8c27770c57a   149Gi      RWO            hostpath-csi-basic   3d15h
rhel9-2a70e7a1fadc            Bound    pvc-9c0e0425-d279-4f4d-aaad-e88265908ef7   149Gi      RWO            hostpath-csi-basic   3d15h

Denis, Could we unset the standard-csi as default storage class during deployment? or maybe we can adjust the deployment flow? 
Could we set the ocs-storagecluster-ceph-rbd (which CNV suggests to use) as the default storage class?

Comment 9 Yan Du 2023-02-27 10:22:06 UTC
Test on CNV v4.13.0.rhel9-1623

$ oc get pvc -n openshift-virtualization-os-images
NAME                          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
centos-stream8-2f16c067b974   Bound    pvc-03f9e46c-533a-4227-a739-f9e01e3b4b37   149Gi      RWO            hostpath-csi-basic   17m
centos-stream9-4e7ee7558c01   Bound    pvc-cbce5d78-9d69-4998-b7cf-d92c3613bc41   149Gi      RWO            hostpath-csi-basic   17m
centos7-680e9b4e0fba          Bound    pvc-a8536fef-6bf7-4c37-96ee-bf4cdb79689a   149Gi      RWO            hostpath-csi-basic   17m
fedora-56ccabc01cbe           Bound    pvc-aa7bfcee-0187-4125-974b-2d5c148d3275   149Gi      RWO            hostpath-csi-basic   17m
rhel8-b0bacbb4e426            Bound    pvc-3a62a6da-e011-4f9c-a66b-11923f489af8   149Gi      RWO            hostpath-csi-basic   17m
rhel9-2a70e7a1fadc            Bound    pvc-6432008e-62b7-456d-b3d8-1dbae51279e8   149Gi      RWO            hostpath-csi-basic   17m

Comment 12 errata-xmlrpc 2023-05-18 02:56:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.13.0 Images security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3205


Note You need to log in before you can comment on or make changes to this bug.