Bug 1878030 - csi-snapshot-controller-operator did not be installed in proxy cluster due to operator container name changed
Summary: csi-snapshot-controller-operator did not be installed in proxy cluster due to...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.6
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.0
Assignee: Jan Safranek
QA Contact: Wei Duan
URL:
Whiteboard:
: 1877972 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-11 06:27 UTC by Wei Duan
Modified: 2020-09-25 12:31 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-csi-snapshot-controller-operator pull 48 None closed Bug 1878030: can not set env proxy for csi-snapshot-controller-operator container 2020-09-25 13:34:33 UTC

Description Wei Duan 2020-09-11 06:27:02 UTC
Description of problem:
Installation with latest nightly failed due to csi-snapshot-controller-operator. There is no csi-snapshot-controller operator installed in cluster due to the operator container name changed. Which should be introduced by https://github.com/openshift/cluster-csi-snapshot-controller-operator/commit/2076baaf43ae40e12d13b54944be0e9ccaa9829a

Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-09-10-195619

How reproducible:
Always with latest nightly(09-10)

Steps to Reproduce:
1. Install cluster
2. check the co:
$ oc get co csi-snapshot-controller
NAME                      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
csi-snapshot-controller          

3. Check no csi-snapshot-controller related resource in openshift-cluster-storage-operator
[wduan@MINT config]$ oc -n openshift-cluster-storage-operator get all
NAME                                            READY   STATUS    RESTARTS   AGE
pod/cluster-storage-operator-85ffccc6c5-4t9rc   1/1     Running   0          3h24m
NAME                                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/csi-snapshot-controller-operator-metrics   ClusterIP   172.30.178.142   <none>        443/TCP   3h24m
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cluster-storage-operator   1/1     1            1           3h24m
NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/cluster-storage-operator-85ffccc6c5   1         1         1       3h24m

Actual results:
csi-snapshot-controller-operator did not be installed 

Expected results:
csi-snapshot-controller-operator should be installed 


Master Log:

Node Log (of failed PODs):

PV Dump:
  
PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
logs from CVO:
E0911 03:33:06.286094       1 task.go:81] error running apply for deployment "openshift-cluster-storage-operator/csi-snapshot-controller-operator" (246 of 602): requested injection for non-existent container: "operator"

Comment 1 pdsilva 2020-09-11 16:00:53 UTC
I see this issue on ppc64le as well with cluster proxy: 

# oc get co csi-snapshot-controller
NAME                      VERSION                                     AVAILABLE   PROGRESSING   DEGRADED   SINCE
csi-snapshot-controller 

# oc version
Client Version: 4.6.0-0.nightly-ppc64le-2020-09-10-082636
Server Version: 4.6.0-0.nightly-ppc64le-2020-09-10-082636
Kubernetes Version: v1.19.0-rc.2+068702d

Comment 2 Abhinav Dahiya 2020-09-11 17:19:40 UTC
*** Bug 1877972 has been marked as a duplicate of this bug. ***

Comment 3 Yadan Pei 2020-09-14 12:49:49 UTC
During 4.5 -> 4.6 upgrade, we met the same issue

# oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      45m
cloud-credential                           4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h54m
cluster-autoscaler                         4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h36m
config-operator                            4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h50m
console                                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      46m
csi-snapshot-controller                    4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h32m
dns                                        4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h49m
etcd                                       4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h49m
image-registry                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h32m
ingress                                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
insights                                   4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h37m
kube-apiserver                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h49m
kube-controller-manager                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h49m
kube-scheduler                             4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h41m
kube-storage-version-migrator              4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h4m
machine-api                                4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h37m
machine-approver                           4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h42m
machine-config                             4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h49m
marketplace                                4.6.0-0.nightly-2020-09-12-230035   True        False         False      47m
monitoring                                 4.6.0-0.nightly-2020-09-12-230035   True        False         False      3h37m
network                                    4.5.0-0.nightly-2020-09-14-030924   True        False         False      7h51m
node-tuning                                4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
openshift-apiserver                        4.6.0-0.nightly-2020-09-12-230035   True        False         False      64m
openshift-controller-manager               4.6.0-0.nightly-2020-09-12-230035   True        False         False      47m
openshift-samples                          4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
operator-lifecycle-manager                 4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h50m
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h50m
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-09-12-230035   True        False         False      46m
service-ca                                 4.6.0-0.nightly-2020-09-12-230035   True        False         False      7h51m
service-catalog-apiserver                  4.4.0-0.nightly-2020-09-12-143345   True        False         False      34s
service-catalog-controller-manager         4.4.0-0.nightly-2020-09-12-143345   True        False         False      7h22m
storage                                    4.6.0-0.nightly-2020-09-12-230035   True        False         False      48m
[root@preserved-qe-ui-rhel-1 ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-09-14-030924   True        True          68m     Unable to apply 4.6.0-0.nightly-2020-09-12-230035: the update could not be applied
[root@preserved-qe-ui-rhel-1 ~]# oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2020-09-14T02:17:05Z"
    generation: 4
    name: version
    resourceVersion: "342165"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: f9892040-3003-4ed5-b844-153817619379
  spec:
    channel: stable-4.4
    clusterID: 95702c52-8764-40f3-bb29-174b80d8484d
    desiredUpdate:
      force: true
      image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-12-230035
      version: ""
    upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
  status:
    availableUpdates: null
    conditions:
    - lastTransitionTime: "2020-09-14T02:47:09Z"
      message: Done applying 4.5.0-0.nightly-2020-09-14-030924
      status: "True"
      type: Available
    - lastTransitionTime: "2020-09-14T09:38:34Z"
      message: Could not update deployment "openshift-cluster-storage-operator/csi-snapshot-controller-operator"
        (246 of 602)
      reason: UpdatePayloadFailed
      status: "True"
      type: Failing
    - lastTransitionTime: "2020-09-14T09:03:01Z"
      message: 'Unable to apply 4.6.0-0.nightly-2020-09-12-230035: the update could
        not be applied'
      reason: UpdatePayloadFailed
      status: "True"
      type: Progressing
    - lastTransitionTime: "2020-09-14T06:18:19Z"
      status: "True"
      type: RetrievedUpdates
    - lastTransitionTime: "2020-09-14T02:48:15Z"
      message: |-
        Multiple cluster operators cannot be upgraded between minor versions:
        * Cluster operator marketplace cannot be upgraded between minor versions: DeprecatedAPIsInUse: The cluster has custom OperatorSource, which is deprecated in future versions. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated
        * Cluster operator service-catalog-apiserver cannot be upgraded between minor versions: _Managed: Upgradeable: The Service Catalog is deprecated, upgrades are not possible. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/applications/service_brokers/installing-service-catalog.html
        * Cluster operator service-catalog-controller-manager cannot be upgraded between minor versions: _Managed: Upgradeable: The Service Catalog is deprecated, upgrades are not possible. Please visit this link for further details: https://docs.openshift.com/container-platform/4.4/applications/service_brokers/installing-service-catalog.html
      reason: ClusterOperatorsNotUpgradeable
      status: "False"
      type: Upgradeable
    desired:
      image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-12-230035
      version: 4.6.0-0.nightly-2020-09-12-230035
    history:
    - completionTime: null
      image: registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-09-12-230035
      startedTime: "2020-09-14T09:03:01Z"
      state: Partial
      verified: false
      version: 4.6.0-0.nightly-2020-09-12-230035
    - completionTime: "2020-09-14T07:10:16Z"
      image: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-09-14-030924
      startedTime: "2020-09-14T06:19:09Z"
      state: Completed
      verified: false
      version: 4.5.0-0.nightly-2020-09-14-030924
    - completionTime: "2020-09-14T02:47:09Z"
      image: registry.svc.ci.openshift.org/ocp/release@sha256:003f48deb2d4cc83318a2a007847bd518cad879cb5ca910d04cad225ecc02e27
      startedTime: "2020-09-14T02:17:13Z"
      state: Completed
      verified: false
      version: 4.4.0-0.nightly-2020-09-12-143345
    observedGeneration: 4
    versionHash: CAjvTPXmFcA=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 6 pdsilva 2020-09-25 12:31:11 UTC
Verified on Power as well with 4.6.0-0.nightly-ppc64le-2020-09-20-022016 nightly build. 
# oc get co csi-snapshot-controller
NAME                      VERSION                                     AVAILABLE   PROGRESSING   DEGRADED   SINCE
csi-snapshot-controller                    4.6.0-0.nightly-ppc64le-2020-09-20-022016   True        False         False      6h38m


Note You need to log in before you can comment on or make changes to this bug.