Bug 1972028

Summary: Upgrade is failed when upgrade SNO cluster on gcp platform
Product: OpenShift Container Platform Reporter: Chao Yang <chaoyang>
Component: StorageAssignee: Fabio Bertinatto <fbertina>
Storage sub component: Storage QA Contact: Chao Yang <chaoyang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs, jsafrane
Version: 4.8   
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-18 17:34:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chao Yang 2021-06-15 05:47:09 UTC
Description of problem:
Upgrade is failed when upgrade SNO cluster on gcp platform, pod gcp-pd-csi-driver-controller is pending.

Version-Release number of selected component (if applicable):
oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-11-024306   True        False         3d15h   Cluster version is 4.8.0-0.nightly-2021-06-11-024306

How reproducible:
Always

Steps to Reproduce:
1.oc get co storage -o yaml
  - lastTransitionTime: "2021-06-11T14:17:03Z"
    message: 'GCPPDCSIDriverOperatorCRProgressing: GCPPDDriverControllerServiceControllerProgressing:
      Waiting for Deployment to deploy pods'
    reason: GCPPDCSIDriverOperatorCR_GCPPDDriverControllerServiceController_Deploying
    status: "True"
    type: Progressing

2.oc get pods -n openshift-cluster-csi-drivers
NAME                                           READY   STATUS    RESTARTS   AGE
gcp-pd-csi-driver-controller-cdcd79b8c-ck6l7   0/10    Pending   0          3d11h
gcp-pd-csi-driver-controller-d45f6f4cc-g7wt5   10/10   Running   13         3d12h
gcp-pd-csi-driver-node-zsjlp                   3/3     Running   3          3d11h
gcp-pd-csi-driver-operator-65697788c4-mrhxz    1/1     Running   57         3d11h

3.Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  56m (x423 over 6h52m)  default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
  Warning  FailedScheduling  2m4s (x60 over 52m)    default-scheduler  0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.


Actual results:
gcp-pd-csi-driver-controller is pending

Expected results:
Upgrade should be successful
Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 3 Chao Yang 2021-08-11 07:10:54 UTC
oc get co storage
NAME      VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
storage   4.9.0-0.nightly-2021-08-07-175228   True        False         False      41m     

oc get pods -n openshift-cluster-csi-drivers
NAME                                            READY   STATUS    RESTARTS   AGE
gcp-pd-csi-driver-controller-7b84668f5f-6nzjc   10/10   Running   17         42m
gcp-pd-csi-driver-node-qqgpc                    3/3     Running   3          42m
gcp-pd-csi-driver-operator-767bfd897b-gng4v     1/1     Running   1          45m

Comment 6 errata-xmlrpc 2021-10-18 17:34:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759