Bug 2050357

Summary:	4.9 nightly to 4.10 nightly failed due to Cloud Credential needs updating before upgrade
Product:	OpenShift Container Platform	Reporter:	Paige Rubendall <prubenda>
Component:	Cloud Credential Operator	Assignee:	Joel Diaz <jdiaz>
Status:	CLOSED NOTABUG	QA Contact:	wang lin <lwan>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.10
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-02-03 20:24:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Paige Rubendall 2022-02-03 18:24:01 UTC

Description of problem:
When upgrading from 4.9.0-0.nightly-arm64-2022-02-02-193301 - > 4.10.0-0.nightly-arm64-2022-02-02-151212 on a Disconnected AWS private cluster, the cloud credential operator is degraded because it says it needs to be updated before it can be upgraded 

deployment "openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready

Version-Release number of selected component (if applicable):4.10.0-0.nightly-arm64-2022-02-02-151212


How reproducible: Unknown


Steps to Reproduce:
1. Create aarch64_Disconnected IPI on AWS & Private cluster at version 4.9.0-0.nightly-arm64-2022-02-02-193301
2. Upgrade cluster using command

oc adm upgrade --to-image=ec2-18-219-165-61.us-east-2.compute.amazonaws.com:5000/ocp-arm64/release-arm64@sha256:c293b5f7ef89506e8450445cf46b4f8d59bcafc46c326a803fb6d2245cfc2cc6 --allow-explicit-upgrade --force

Actual results:
clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade

clusteroperator/network is progressing: Deployment "openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready

Expected results:
Cluster upgrades with no issues and all cluster operators are available, not progressing and not degraded

Additional info:
02-03 08:27:48.127  clusteroperators: 
02-03 08:27:48.128   NAME                                       VERSION                                    AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
02-03 08:27:48.128  authentication                             4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      95m     
02-03 08:27:48.128  baremetal                                  4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  cloud-controller-manager                   4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h54m   
02-03 08:27:48.128  cloud-credential                           4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  cluster-autoscaler                         4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  config-operator                            4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h53m   
02-03 08:27:48.128  console                                    4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      95m     
02-03 08:27:48.128  csi-snapshot-controller                    4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h53m   
02-03 08:27:48.128  dns                                        4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  etcd                                       4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h51m   
02-03 08:27:48.128  image-registry                             4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h43m   
02-03 08:27:48.128  ingress                                    4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h43m   
02-03 08:27:48.128  insights                                   4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h46m   
02-03 08:27:48.128  kube-apiserver                             4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h43m   
02-03 08:27:48.128  kube-controller-manager                    4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h51m   
02-03 08:27:48.128  kube-scheduler                             4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h51m   
02-03 08:27:48.128  kube-storage-version-migrator              4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      70m     
02-03 08:27:48.128  machine-api                                4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h42m   
02-03 08:27:48.128  machine-approver                           4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  machine-config                             4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h51m   
02-03 08:27:48.128  marketplace                                4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  monitoring                                 4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h42m   
02-03 08:27:48.128  network                                    4.10.0-0.nightly-arm64-2022-02-02-151212   True        True          False      7h53m   Deployment "openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready
02-03 08:27:48.128  node-tuning                                4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      68m     
02-03 08:27:48.128  openshift-apiserver                        4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h43m   
02-03 08:27:48.128  openshift-controller-manager               4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h46m   
02-03 08:27:48.128  openshift-samples                          4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      105m    
02-03 08:27:48.128  operator-lifecycle-manager                 4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  operator-lifecycle-manager-catalog         4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   
02-03 08:27:48.128  operator-lifecycle-manager-packageserver   4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      77m     
02-03 08:27:48.128  service-ca                                 4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h53m   
02-03 08:27:48.128  storage                                    4.10.0-0.nightly-arm64-2022-02-02-151212   True        False         False      7h52m   


02-03 08:30:20.638  ClusterID: dd8cf54d-1478-4b01-a8d2-3370a6018e6c
02-03 08:30:20.638  ClusterVersion: Stable at "4.10.0-0.nightly-arm64-2022-02-02-151212"
02-03 08:30:20.638  ClusterOperators:
02-03 08:30:20.638  	clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade.
02-03 08:30:20.638  	clusteroperator/network is progressing: Deployment "openshift-cloud-network-config-controller/cloud-network-config-controller" is waiting for other operators to become ready

Comment 1 Joel Diaz 2022-02-03 18:32:31 UTC

I'm guessing that this cluster is configured in a way where the cloud-credential-operator is set to Manual mode? You can check 'oc get cloudcredentials cluster -o yaml' output to see the .spec.credentialsMode. Providing a must-gather would also be helpful.

Comment 4 Joel Diaz 2022-02-03 19:13:38 UTC

Yes, this cluster is indeed in Manual mode:
spec:
  credentialsMode: Manual
  logLevel: Normal
  operatorLogLevel: Normal

So you need to do the annotations before upgrade to signal that you've performed the pre-upgrade credentials steps (assuming any were actually needed). https://docs.openshift.com/container-platform/4.9/authentication/managing_cloud_provider_credentials/cco-mode-manual.html#manually-maintained-credentials-upgrade_cco-mode-manual

Comment 5 Paige Rubendall 2022-02-03 20:24:00 UTC

Great, I'll open an automation bug for this. This was part of the upgrade ci we have in QE. Thanks for the pointer