Bug 1915643

Summary: OCP4.6 to 4.7 upgrade failed due to manila csi driver operator sync error
Product: OpenShift Container Platform Reporter: Wei Duan <wduan>
Component: StorageAssignee: Mike Fedosin <mfedosin>
Storage sub component: OpenStack CSI Drivers QA Contact: Wei Duan <wduan>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: urgent CC: aos-bugs, pprinett, wking
Version: 4.7Keywords: UpcomingSprint, Upgrades
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:52:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
manila-operator-log none

Description Wei Duan 2021-01-13 06:05:01 UTC 
Created attachment 1746906 [details]
manila-operator-log

Description of problem:
Upgrade failed from OCP4.6 to 4.7.0-0.nightly-2021-01-12-203716 due to cluster operator storage degraded. 
$ occluster 
OpenStack
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-01-12-112514   True        True          85m     Unable to apply 4.7.0-0.nightly-2021-01-12-203716: the cluster operator storage is degraded

When checking the co status, we see manila sync error 
status:
  conditions:
  - lastTransitionTime: "2021-01-13T04:02:32Z"
    message: 'ManilaCSIDriverOperatorCRDegraded: ManilaControllerDegraded: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.'
    reason: ManilaCSIDriverOperatorCR_ManilaController_SyncError

From the manila operator log(with Debug devel)
I0113 04:27:32.413504       1 manila.go:88] Manila sync started
I0113 04:27:32.414033       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"manila-csi-driver-operator", UID:"eba82efa-e37c-4707-9813-e3e7d10d44ac", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'StorageClassUpdateFailed' Failed to update StorageClass.storage.k8s.io/csi-manila-ceph: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.
I0113 04:27:32.549082       1 manila.go:133] Syncing storage class for shareType type ceph
I0113 04:27:32.606454       1 request.go:581] Throttling request took 57.290385ms, request: GET:https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses/csi-manila-ceph
I0113 04:27:32.606518       1 request.go:581] Throttling request took 393.919879ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-manila-csi-driver/serviceaccounts/manila-csi-driver-controller-sa
I0113 04:27:32.618247       1 storage.go:51] StorageClass "csi-manila-ceph" changes: {"parameters":{"appendShareMetadata":"{\"openshiftClusterID\": \"\"}"}}
I0113 04:27:32.806450       1 request.go:581] Throttling request took 440.304705ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-manila-csi-driver/secrets/csi-manila-secrets
I0113 04:27:32.806492       1 request.go:581] Throttling request took 188.13889ms, request: PUT:https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses/csi-manila-ceph
I0113 04:27:32.814273       1 manila.go:124] Manila sync finished
I0113 04:27:32.814465       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"manila-csi-driver-operator", UID:"eba82efa-e37c-4707-9813-e3e7d10d44ac", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'StorageClassUpdateFailed' Failed to update StorageClass.storage.k8s.io/csi-manila-ceph: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.
E0113 04:27:32.814504       1 base_controller.go:250] "ManilaController" controller failed to sync "key", err: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.
I0113 04:27:33.006439       1 request.go:581] Throttling request took 592.800242ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-config/configmaps/cloud-provider-config


Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2021-01-12-203716

How reproducible:
2/2

Steps to Reproduce:
Upgrade OCP4.6 to OCP4.7 on OSP 

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
Comment 3 Wei Duan 2021-01-22 04:32:47 UTC 
Verified pass on upgrade from 4.6.9 to 4.7.0-0.nightly-2021-01-21-172657.
After upgrade, manila csi driver works fine.


$ oc get clusterversion version -o json | jq .status.history
[
  {
    "completionTime": "2021-01-22T02:57:46Z",
    "image": "registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-01-21-172657",
    "startedTime": "2021-01-22T01:44:26Z",
    "state": "Completed",
    "verified": false,
    "version": "4.7.0-0.nightly-2021-01-21-172657"
  },
  {
    "completionTime": "2021-01-20T06:55:36Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:43d5c84169a4b3ff307c29d7374f6d69a707de15e9fa90ad352b432f77c0cead",
    "startedTime": "2021-01-20T06:21:04Z",
    "state": "Completed",
    "verified": false,
    "version": "4.6.9"
  }
]
Comment 6 errata-xmlrpc 2021-02-24 15:52:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633
Comment 7 W. Trevor King 2021-03-31 04:22:13 UTC 
Wei added UpgradeBlocker back in January, but this got fixed before 4.7 GAed, so we never blocked any edges on it.  Replacing with Upgrades to remove this bug from our suspect queue [1].

[1]: https://github.com/openshift/enhancements/pull/475