Description of problem: This issue has originally been reported in https://bugzilla.redhat.com/show_bug.cgi?id=1862991. However, fix was provided by Gal Zaidman and verified by Wenjing Zheng. I also deployed OCP4.6 couple of times since then and can confirm that I wasn't hitting this bug. However, on August 24 2020 I deployed two OCP4.6 clusters on different RHV environments and both deployemnts failed on this. Therefore I decided to open a new bug because the root cause might be different. Version-Release number of selected component (if applicable): oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 63m Unable to apply 4.6.0-0.nightly-2020-08-24-034934: the cluster operator image-registry has not yet successfully rolled out How reproducible: 100 % Steps to Reproduce: 1. Run openshift-installer on top of RHV (my install-config here: http://pastebin.test.redhat.com/895896) 2. Wait until the installation fails 3. Check the state of image-registry operator Actual results: oc status: http://pastebin.test.redhat.com/895931 oc get all -n openshift-image-registry: http://pastebin.test.redhat.com/895936 oc describe deployment.apps/image-registry: http://pastebin.test.redhat.com/895935 oc describe pod/image-registry-6464f46dc7-ckpb2: http://pastebin.test.redhat.com/895937 oc get events -n openshift-image-registry: http://pastebin.test.redhat.com/895940
# oc get all -n openshift-cluster-csi-drivers NAME READY STATUS RESTARTS AGE pod/ovirt-csi-driver-controller-74994665c7-zdccv 3/3 Running 0 3h32m pod/ovirt-csi-driver-node-2885v 3/3 Running 1 3h32m pod/ovirt-csi-driver-node-69vqk 3/3 Running 0 3h32m pod/ovirt-csi-driver-node-78hqn 3/3 Running 0 3h32m pod/ovirt-csi-driver-node-cm6hs 3/3 Running 0 3h20m pod/ovirt-csi-driver-node-gf9tt 3/3 Running 0 3h19m pod/ovirt-csi-driver-node-z6v6d 3/3 Running 0 3h32m pod/ovirt-csi-driver-operator-95477b8c7-gchlp 1/1 Running 0 3h51m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/ovirt-csi-driver-node 6 6 6 6 6 <none> 3h32m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ovirt-csi-driver-controller 1/1 1 1 3h32m deployment.apps/ovirt-csi-driver-operator 1/1 1 1 3h51m NAME DESIRED CURRENT READY AGE replicaset.apps/ovirt-csi-driver-controller-74994665c7 1 1 1 3h32m replicaset.apps/ovirt-csi-driver-operator-95477b8c7 1 1 1 3h51m # oc logs pod/ovirt-csi-driver-controller-74994665c7-zdccv -c csi-driver -n openshift-cluster-csi-drivers I0824 10:17:39.462782 1 ovirt-csi-driver.go:42] Driver vendor csi.ovirt.org 0.1.1 I0824 10:17:41.826417 1 driver.go:39] Setting the rpc server I0824 10:17:50.208116 1 controller.go:36] Creating disk pvc-9bf197c7-1564-4a66-9136-0f67c3d20f3c
# oc logs pod/ovirt-csi-driver-controller-74994665c7-zdccv -c csi-attacher -n openshift-cluster-csi-drivers | head -n 8 I0824 10:17:59.345293 1 main.go:91] Version: v4.6.0-202008210209.p0-0-g4c59d76-dirty I0824 10:17:59.347404 1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock I0824 10:17:59.351075 1 common.go:111] Probing CSI driver for readiness I0824 10:17:59.432277 1 main.go:136] CSI driver name: "csi.ovirt.org" W0824 10:17:59.432301 1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified. I0824 10:17:59.433193 1 main.go:162] CSI driver supports ControllerPublishUnpublish, using real CSI handler I0824 10:17:59.433609 1 controller.go:121] Starting CSI attacher E0824 10:17:59.446020 1 reflector.go:156] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:ovirt-csi-driver-controller-sa" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope <then the whole log is full of error like these> # oc logs pod/ovirt-csi-driver-controller-74994665c7-zdccv -c csi-attacher -n openshift-cluster-csi-drivers | tail -n 5 E0824 13:59:14.139761 1 reflector.go:156] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:ovirt-csi-driver-controller-sa" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope E0824 13:59:15.143140 1 reflector.go:156] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:ovirt-csi-driver-controller-sa" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope E0824 13:59:16.146062 1 reflector.go:156] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:ovirt-csi-driver-controller-sa" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope E0824 13:59:17.148060 1 reflector.go:156] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:ovirt-csi-driver-controller-sa" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope E0824 13:59:18.150623 1 reflector.go:156] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:ovirt-csi-driver-controller-sa" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope
*** This bug has been marked as a duplicate of bug 1871051 ***