Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1878030

Summary: csi-snapshot-controller-operator did not be installed in proxy cluster due to operator container name changed
Product: OpenShift Container Platform Reporter: Wei Duan <wduan>
Component: StorageAssignee: Jan Safranek <jsafrane>
Storage sub component: Operators QA Contact: Wei Duan <wduan>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: aos-bugs, jsafrane, kewang, pdsilva, tsze, yapei
Version: 4.6Keywords: TestBlocker
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:40:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Wei Duan 2020-09-11 06:27:02 UTC 
Description of problem:
Installation with latest nightly failed due to csi-snapshot-controller-operator. There is no csi-snapshot-controller operator installed in cluster due to the operator container name changed. Which should be introduced by https://github.com/openshift/cluster-csi-snapshot-controller-operator/commit/2076baaf43ae40e12d13b54944be0e9ccaa9829a

Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-09-10-195619

How reproducible:
Always with latest nightly(09-10)

Steps to Reproduce:
1. Install cluster
2. check the co:
$ oc get co csi-snapshot-controller
NAME                      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
csi-snapshot-controller          

3. Check no csi-snapshot-controller related resource in openshift-cluster-storage-operator
[wduan@MINT config]$ oc -n openshift-cluster-storage-operator get all
NAME                                            READY   STATUS    RESTARTS   AGE
pod/cluster-storage-operator-85ffccc6c5-4t9rc   1/1     Running   0          3h24m
NAME                                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/csi-snapshot-controller-operator-metrics   ClusterIP   172.30.178.142   <none>        443/TCP   3h24m
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cluster-storage-operator   1/1     1            1           3h24m
NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/cluster-storage-operator-85ffccc6c5   1         1         1       3h24m

Actual results:
csi-snapshot-controller-operator did not be installed 

Expected results:
csi-snapshot-controller-operator should be installed 


Master Log:

Node Log (of failed PODs):

PV Dump:
  
PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
logs from CVO:
E0911 03:33:06.286094       1 task.go:81] error running apply for deployment "openshift-cluster-storage-operator/csi-snapshot-controller-operator" (246 of 602): requested injection for non-existent container: "operator"
Comment 1 pdsilva 2020-09-11 16:00:53 UTC 
I see this issue on ppc64le as well with cluster proxy: 

# oc get co csi-snapshot-controller
NAME                      VERSION                                     AVAILABLE   PROGRESSING   DEGRADED   SINCE
csi-snapshot-controller 

# oc version
Client Version: 4.6.0-0.nightly-ppc64le-2020-09-10-082636
Server Version: 4.6.0-0.nightly-ppc64le-2020-09-10-082636
Kubernetes Version: v1.19.0-rc.2+068702d
Comment 2 Abhinav Dahiya 2020-09-11 17:19:40 UTC 
*** Bug 1877972 has been marked as a duplicate of this bug. ***
Comment 3 Yadan Pei 2020-09-14 12:49:49 UTC 
During 4.5 -> 4.6 upgrade, we met the same issue

# oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      45m
cloud-credential                           4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h54m
cluster-autoscaler                         4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h36m
config-operator                            4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h50m
console                                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      46m
csi-snapshot-controller                    4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h32m
dns                                        4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h49m
etcd                                       4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h49m
image-registry                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h32m
ingress                                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
insights                                   4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h37m
kube-apiserver                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h49m
kube-controller-manager                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h49m
kube-scheduler                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h41m
kube-storage-version-migrator              4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h4m
machine-api                                4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h37m
machine-approver                           4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h42m
machine-config                             4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h49m
marketplace                                4.6.0-0.nightly-2020-09-12-230035   True        False         False      47m
monitoring                                 4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h37m
network                                    4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h51m
node-tuning                                4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
openshift-apiserver                        4.6.0-0.nightly-2020-09-12-230035   True        False         False      64m
openshift-controller-manager               4.6.0-0.nightly-2020-09-12-230035   True        False         False      47m
openshift-samples                          4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
operator-lifecycle-manager                 4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h50m
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h50m
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-09-12-230035   True        False         False      46m
service-ca                                 4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h51m
service-catalog-apiserver                  4.4.0-0.nightly-2020-09-12-143345   True        False         False      34s
service-catalog-controller-manager         4.4.0-0.nightly-2020-09-12-143345   True        False         False      7h22m
storage                                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
[root@preserved-qe-ui-rhel-1 ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-09-14-030924   True        True          68m     Unable to apply 4.6.0-0.nightly-2020-09-12-230035: the update could not be applied
[root@preserved-qe-ui-rhel-1 ~]# oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2020-09-14T02:17:05Z"
    generation: 4
    name: version
    resourceVersion: "342165"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: f9892040-3003-4ed5-b844-153817619379
  spec:
    channel: stable-4.4
    clusterID: 95702c52-8764-40f3-bb29-174b80d8484d
    desiredUpdate:
      force: true
      image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-12-230035
      version: ""
    upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
  status:
    availableUpdates: null
    conditions:
    - lastTransitionTime: "2020-09-14T02:47:09Z"
      message: Done applying 4.5.0-0.nightly-2020-09-14-030924
      status: "True"
      type: Available
    - lastTransitionTime: "2020-09-14T09:38:34Z"
      message: Could not update deployment "openshift-cluster-storage-operator/csi-snapshot-controller-operator"
        (246 of 602)
      reason: UpdatePayloadFailed
      status: "True"
      type: Failing
    - lastTransitionTime: "2020-09-14T09:03:01Z"
      message: 'Unable to apply 4.6.0-0.nightly-2020-09-12-230035: the update could
        not be applied'
      reason: UpdatePayloadFailed
      status: "True"
      type: Progressing
    - lastTransitionTime: "2020-09-14T06:18:19Z"
      status: "True"
      type: RetrievedUpdates
    - lastTransitionTime: "2020-09-14T02:48:15Z"
      message: |-
        Multiple cluster operators cannot be upgraded between minor versions:
        * Cluster operator marketplace cannot be upgraded between minor versions: DeprecatedAPIsInUse: The cluster has custom OperatorSource, which is deprecated in future versions. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated
        * Cluster operator service-catalog-apiserver cannot be upgraded between minor versions: _Managed: Upgradeable: The Service Catalog is deprecated, upgrades are not possible. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/applications/service_brokers/installing-service-catalog.html
        * Cluster operator service-catalog-controller-manager cannot be upgraded between minor versions: _Managed: Upgradeable: The Service Catalog is deprecated, upgrades are not possible. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/applications/service_brokers/installing-service-catalog.html
      reason: ClusterOperatorsNotUpgradeable
      status: "False"
      type: Upgradeable
    desired:
      image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-12-230035
      version: 4.6.0-0.nightly-2020-09-12-230035
    history:
    - completionTime: null
      image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-12-230035
      startedTime: "2020-09-14T09:03:01Z"
      state: Partial
      verified: false
      version: 4.6.0-0.nightly-2020-09-12-230035
    - completionTime: "2020-09-14T07:10:16Z"
      image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-09-14-030924
      startedTime: "2020-09-14T06:19:09Z"
      state: Completed
      verified: false
      version: 4.5.0-0.nightly-2020-09-14-030924
    - completionTime: "2020-09-14T02:47:09Z"
      image: registry.svc.ci.openshift.org/ocp/release@sha256:003f48deb2d4cc83318a2a007847bd518cad879cb5ca910d04cad225ecc02e27
      startedTime: "2020-09-14T02:17:13Z"
      state: Completed
      verified: false
      version: 4.4.0-0.nightly-2020-09-12-143345
    observedGeneration: 4
    versionHash: CAjvTPXmFcA=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Comment 6 pdsilva 2020-09-25 12:31:11 UTC 
Verified on Power as well with 4.6.0-0.nightly-ppc64le-2020-09-20-022016 nightly build. 
# oc get co csi-snapshot-controller
NAME                      VERSION                                     AVAILABLE   PROGRESSING   DEGRADED   SINCE
csi-snapshot-controller                    4.6.0-0.nightly-ppc64le-2020-09-20-022016   True        False         False      6h38m
Comment 9 errata-xmlrpc 2020-10-27 16:40:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196
Comment 10 W. Trevor King 2021-04-05 17:46:38 UTC 
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1].  If you feel like this bug still needs to be a suspect, please add keyword again.

[1]: https://github.com/openshift/enhancements/pull/475