Bug 1952891
| Summary: | Upgrade failed due to cinder csi driver not deployed | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wei Duan <wduan> |
| Component: | Storage | Assignee: | Matthew Booth <mbooth> |
| Storage sub component: | OpenStack CSI Drivers | QA Contact: | Jon Uriarte <juriarte> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | urgent | CC: | adduarte, aos-bugs, lwan, mbooth, mfedosin |
| Version: | 4.8 | Keywords: | Regression, Triaged |
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 23:03:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Wei Duan
2021-04-23 12:56:49 UTC
The cause of this issue is that long time ago there was a bug in gophercloud utils which caused incorrect yaml marshaling for empty fields I fixed that bug https://github.com/gophercloud/utils/pull/100 and the fix was available only in 4.3. It means that in 4.2 if cacert was not in the original clouds.yaml the utils produced `cacert: ""` in the system clouds.yaml, which technically not a problem. But when we introduced https://github.com/openshift/cloud-credential-operator/pull/314, it just checks that the key (cacert) exists in the file, omitting the fact it can be empty. To fix the issue we need to ignore empty cacert I'm going to post an alternative patch shortly. Doc text not required, as this bug was never released. Verified in 4.8.0-0.nightly-2021-06-14-145150 on top of OSP 16.1.6 (RHOS-16.1-RHEL-8-20210506.n.1), after the next upgrade chain:
4.2.0-0.nightly-2021-02-22-141219 -> 4.3.40 -> 4.4.33 -> 4.5.40 -> 4.6.35 -> 4.7.17 -> 4.8.0-0.nightly-2021-06-14-145150
The underlying OSP is without SSL enabled and the clouds.yaml doesn't contain the cacert param:
clouds:
...
shiftstack:
auth:
auth_url: http://10.0.0.104:5000
password: hidden
project_domain_name: Default
project_name: shiftstack
project_id: a2de4b65f83341d1942c201750fffdf6
user_domain_name: Default
username: shiftstack_user
identity_api_version: '3'
region_name: regionOne
$ openstack server list
+--------------------------------------+---------------------------+--------+-------------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------------------------+--------+-------------------------------------+-------+--------+
| e52f1172-dfb1-4328-bb47-db2ebf14ead8 | ostest-28klf-worker-lx9bv | ACTIVE | ostest-28klf-openshift=10.196.2.211 | rhcos | |
| b9e7e76e-e74e-4206-99d2-c94d9b2b4e3d | ostest-28klf-worker-2fvwv | ACTIVE | ostest-28klf-openshift=10.196.0.140 | rhcos | |
| 32f09721-a9e5-4e3a-8092-ba127f1e7609 | ostest-28klf-master-0 | ACTIVE | ostest-28klf-openshift=10.196.0.47 | rhcos | |
| 155849f7-27d8-45d1-99be-4d837252acdc | ostest-28klf-master-1 | ACTIVE | ostest-28klf-openshift=10.196.0.32 | rhcos | |
| ac37f988-b9cb-433e-be21-4f5328a66cd8 | ostest-28klf-master-2 | ACTIVE | ostest-28klf-openshift=10.196.1.107 | rhcos | |
+--------------------------------------+---------------------------+--------+-------------------------------------+-------+--------+
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.2.0-0.nightly-2021-02-22-141219 True False 60s Cluster version is 4.2.0-0.nightly-2021-02-22-141219
Upgrades:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.3.40 True False 3m49s Cluster version is 4.3.40
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.4.33 True False 3m33s Cluster version is 4.4.33
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.5.40 True False 8s Cluster version is 4.5.40
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.35 True False 3m53s Cluster version is 4.6.35
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.7.17 True False 3m44s Cluster version is 4.7.17
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.0-0.nightly-2021-06-14-145150 True False 12h Cluster version is 4.8.0-0.nightly-2021-06-14-145150
$ oc get pods -A | grep csi
openshift-cluster-csi-drivers manila-csi-driver-operator-5f55dbb4cb-772r4 1/1 Running 0 14h
openshift-cluster-csi-drivers openstack-cinder-csi-driver-controller-78f5ff7789-hrp5p 9/9 Running 0 14h
openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-bbxrc 2/2 Running 2 14h
openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-gtljz 2/2 Running 2 14h
openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-rwq56 2/2 Running 2 14h
openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-w5k92 2/2 Running 2 14h
openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-x86fc 2/2 Running 2 14h
openshift-cluster-csi-drivers openstack-cinder-csi-driver-operator-69579789f5-5p8g8 1/1 Running 0 14h
openshift-cluster-storage-operator csi-snapshot-controller-84f9687fc6-6rklp 1/1 Running 0 14h
openshift-cluster-storage-operator csi-snapshot-controller-84f9687fc6-jlg7n 1/1 Running 0 14h
openshift-cluster-storage-operator csi-snapshot-controller-operator-694786cd9-v6v9v 1/1 Running 0 14h
openshift-cluster-storage-operator csi-snapshot-webhook-6b7dbc67d4-cm9dw 1/1 Running 0 14h
openshift-cluster-storage-operator csi-snapshot-webhook-6b7dbc67d4-jjq7h 1/1 Running 0 14h
$ oc get secret -n openshift-cluster-csi-drivers openstack-cloud-credentials -o yaml
apiVersion: v1
data:
clouds.yaml: <content>
kind: Secret
...
$ echo "<content>" | base64 -d
clouds:
openstack:
auth:
application_credential_id: ""
application_credential_name: ""
application_credential_secret: ""
auth_url: http://10.0.0.104:5000
default_domain: ""
domain_id: ""
domain_name: ""
password: hidden
project_domain_id: ""
project_domain_name: Default
project_id: a2de4b65f83341d1942c201750fffdf6
project_name: shiftstack
token: ""
user_domain_id: ""
user_domain_name: Default
user_id: ""
username: shiftstack_user
auth_type: ""
cert: ""
cloud: ""
identity_api_version: "3"
key: ""
profile: ""
region_name: regionOne
regions: null
verify: true
volume_api_version: ""
The cluster is ok after the upgrade chain.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |