Bug 2124469

Description of problem (please be detailed as possible and provide log
snippests):

Redhat deploy a OCS 4.8 cluster.
After initial deployment "FlexibleAutoscaling: true" need to be set on StorageCluster in openshift-storage namespace, and that require a redeployment.
After this redeployment pvc for ocs-storagecluster-cephfs does not work and pvc stay in pending state instead of Bound.




Version of all relevant components (if applicable):

OCS 4.8, same issue has been reproduced in OCS 4.7



Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

This issue had an impact for multiple day.
pvc for  was ocs-storagecluster-cephfs was not working anymore.




Is there any workaround available to the best of your knowledge?

The following github rook issue describe the situation : 
https://github.com/rook/rook/issues/6183


Following comment from the previous github link give us clue and solution : 
Muyan0828 commented on Mar 4, 2021

"I think the root case is that your filesystem was recreate, cephfs-csi stores a bool variable in memory for each cluster to mark whether a subvolumegroup has been created, cephfs-csi corresponds to a cluster via the clusterID field in StorageClass, which is set to the namespace of the cephcluster in rook, so when the CephFilesystem is rebuilt in the same namespace and StorageClass is recreated, in cephfs-csi the variable for the cluster is true, so there is no attempt to recreate the subvolumegroup"

So the following was try on a rook-ceph-operator pod : 

ceph fs subvolumegroup ls ocs-storagecluster-cephfilesystem
[]

Which give us not the same result as on a working OCS.
We expected the following result : 

ceph fs subvolumegroup ls ocs-storagecluster-cephfilesystem
[
    {
        "name": "_deleting"
    },
    {
        "name": "csi"  
    }
]

Restarting the following pod solve the situation : 

oc delete pod -l app=csi-cephfsplugin-provisioner
oc delete pod -l app=csi-cephfsplugin

At this point PVC can be "bound".



Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

1


Can this issue reproducible?

This issue has been reproduced with different scenario on OCS 4.7.
Removing the "cephfs" filesystem manually using the following procedure ( not recommanded for debugging purpose ) : 

ceph fs volume rm ocs-storagecluster-cephfilesystem --yes-i-really-mean-it
ceph  fs volume create ocs-storagecluster-cephfilesystem

Or removing StorageCluster in openshift-storage and recreate it produce the issue.





Can this issue reproduce from the UI?

Not to my knowledge.




If this is a regression, please provide more details to justify this:

This is not a regression from my understanding.
It is more a behaviour.
The operator in the "openshift-storage" might need to be removed to not fall into the same scenario.




Steps to Reproduce:

The following procedure was used : 

https://gitlab.consulting.redhat.com/emea-telco-practice/telefonica-next/mano-cluster/-/tree/maindc1/05.OCS

====
oc apply -f 1.LSO.yml
oc apply -f 2.labelNodes.yaml
oc apply -f 3.localVolumeSet.yml

---------------> LSO Operator and local volume are provisioned.

oc apply -f 4.OCS.yml

--------------> OCS Operator deployed.

oc apply -f 5.storageCluster.yml

-----------> OCS cluster deployed



At this exact step everything was working as expected.
Then the StorageCluster in openshift-storage namespace was removed

oc delete -f 5.storageCluster.yml

----------- > OCS cluster deleted, BUT, LSO and OCS operators are installed and running.

vi 5.storageCluster.yml

-----------> Enabled FlexibleAutoscaling: true

oc apply -f 5.storageCluster.yml

-----------> Redeployed OCS cluster with felxibleAustoSacling true.



Actual results:

pvc in pending state for ocs-storagecluster-cephfs not able to be bound

oc get pvc
NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                AGE
mypvc   Pending                                      ocs-storagecluster-cephfs   10s

oc logs -n openshift-storage  csi-cephfsplugin-provisioner-544d647bc4-svtwz csi-provisioner|less

I0902 13:05:32.877114       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"test", Name:"mypvc", UID:"7e2a57bc-ee1f-49d9-8951-66fdaa2a4f74", APIVersion:"v1", ResourceVersion:"359756", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ocs-storagecluster-cephfs": rpc error: code = Internal desc = rados: ret=-2, No such file or directory: "subvolume group 'csi' does not exist"



Expected results:

Here there is no expected result.
I can say, that I expect pvc to be bounded, but this case is more a description of what has been discover and discuss.
This is a behavior of OCS which need to be more describe in the OCS documentation.

Additional info:

We expect the following feedback from engineering : 

- what CRD should have been removed to be able to configure "FlexibleAutoscaling: true" without falling into the trap

- Does the expected workaround look to be safe and accurate, for now we don't see anything which lead to any issue on pvc side

- Do you think we might miss something obvious here from the supportability of the filesystem or the other service in the future ?