Bug 1915643 - OCP4.6 to 4.7 upgrade failed due to manila csi driver operator sync error
Summary: OCP4.6 to 4.7 upgrade failed due to manila csi driver operator sync error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.7
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.7.0
Assignee: Mike Fedosin
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-13 06:05 UTC by Wei Duan
Modified: 2021-03-31 04:22 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:52:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
manila-operator-log (60.89 KB, text/plain)
2021-01-13 06:05 UTC, Wei Duan
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift csi-driver-manila-operator pull 77 0 None closed Bug 1915643: revert setting metadata in the storage classes 2021-01-25 08:03:16 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:52:56 UTC

Description Wei Duan 2021-01-13 06:05:01 UTC
Created attachment 1746906 [details]
manila-operator-log

Description of problem:
Upgrade failed from OCP4.6 to 4.7.0-0.nightly-2021-01-12-203716 due to cluster operator storage degraded. 
$ occluster 
OpenStack
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-01-12-112514   True        True          85m     Unable to apply 4.7.0-0.nightly-2021-01-12-203716: the cluster operator storage is degraded

When checking the co status, we see manila sync error 
status:
  conditions:
  - lastTransitionTime: "2021-01-13T04:02:32Z"
    message: 'ManilaCSIDriverOperatorCRDegraded: ManilaControllerDegraded: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.'
    reason: ManilaCSIDriverOperatorCR_ManilaController_SyncError

From the manila operator log(with Debug devel)
I0113 04:27:32.413504       1 manila.go:88] Manila sync started
I0113 04:27:32.414033       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"manila-csi-driver-operator", UID:"eba82efa-e37c-4707-9813-e3e7d10d44ac", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'StorageClassUpdateFailed' Failed to update StorageClass.storage.k8s.io/csi-manila-ceph: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.
I0113 04:27:32.549082       1 manila.go:133] Syncing storage class for shareType type ceph
I0113 04:27:32.606454       1 request.go:581] Throttling request took 57.290385ms, request: GET:https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses/csi-manila-ceph
I0113 04:27:32.606518       1 request.go:581] Throttling request took 393.919879ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-manila-csi-driver/serviceaccounts/manila-csi-driver-controller-sa
I0113 04:27:32.618247       1 storage.go:51] StorageClass "csi-manila-ceph" changes: {"parameters":{"appendShareMetadata":"{\"openshiftClusterID\": \"\"}"}}
I0113 04:27:32.806450       1 request.go:581] Throttling request took 440.304705ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-manila-csi-driver/secrets/csi-manila-secrets
I0113 04:27:32.806492       1 request.go:581] Throttling request took 188.13889ms, request: PUT:https://172.30.0.1:443/apis/storage.k8s.io/v1/storageclasses/csi-manila-ceph
I0113 04:27:32.814273       1 manila.go:124] Manila sync finished
I0113 04:27:32.814465       1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"manila-csi-driver-operator", UID:"eba82efa-e37c-4707-9813-e3e7d10d44ac", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'StorageClassUpdateFailed' Failed to update StorageClass.storage.k8s.io/csi-manila-ceph: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.
E0113 04:27:32.814504       1 base_controller.go:250] "ManilaController" controller failed to sync "key", err: StorageClass.storage.k8s.io "csi-manila-ceph" is invalid: parameters: Forbidden: updates to parameters are forbidden.
I0113 04:27:33.006439       1 request.go:581] Throttling request took 592.800242ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-config/configmaps/cloud-provider-config


Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2021-01-12-203716

How reproducible:
2/2

Steps to Reproduce:
Upgrade OCP4.6 to OCP4.7 on OSP 

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 3 Wei Duan 2021-01-22 04:32:47 UTC
Verified pass on upgrade from 4.6.9 to 4.7.0-0.nightly-2021-01-21-172657.
After upgrade, manila csi driver works fine.


$ oc get clusterversion version -o json | jq .status.history
[
  {
    "completionTime": "2021-01-22T02:57:46Z",
    "image": "registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-01-21-172657",
    "startedTime": "2021-01-22T01:44:26Z",
    "state": "Completed",
    "verified": false,
    "version": "4.7.0-0.nightly-2021-01-21-172657"
  },
  {
    "completionTime": "2021-01-20T06:55:36Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:43d5c84169a4b3ff307c29d7374f6d69a707de15e9fa90ad352b432f77c0cead",
    "startedTime": "2021-01-20T06:21:04Z",
    "state": "Completed",
    "verified": false,
    "version": "4.6.9"
  }
]

Comment 6 errata-xmlrpc 2021-02-24 15:52:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633

Comment 7 W. Trevor King 2021-03-31 04:22:13 UTC
Wei added UpgradeBlocker back in January, but this got fixed before 4.7 GAed, so we never blocked any edges on it.  Replacing with Upgrades to remove this bug from our suspect queue [1].

[1]: https://github.com/openshift/enhancements/pull/475


Note You need to log in before you can comment on or make changes to this bug.