Bug 2046254

Summary: Topolvm-controller is failing to pull image
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Shay Rozen <srozen>
Component: lvm-operatorAssignee: N Balachandran <nibalach>
Status: CLOSED ERRATA QA Contact: Shay Rozen <srozen>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.10CC: branto, ikave, jolmomar, lgangava, madam, mmuench, muagarwa, nberry, ocs-bugs, odf-bz-bot, rperiyas, sapillai
Target Milestone: ---Keywords: Automation, TestBlocker
Target Release: ODF 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.10.0-132 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-13 18:51:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shay Rozen 2022-01-26 13:26:39 UTC 
Description of problem (please be detailed as possible and provide log
snippests):
When trying to create lvm-cluster Topolvm-controller is failing to pull image.
On pod describe you can see multiple errors:
  Warning  Failed          6m56s                  kubelet            Failed to pull image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095": rpc error: code = Unknown desc = reading manifest sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095 in registry.redhat.io/odf4/odf-topolvm-rhel8: unknown: Not Found
  Warning  Failed          6m55s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3": rpc error: code = Unknown desc = reading manifest sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3 in registry.redhat.io/openshift4/ose-csi-external-provisioner: StatusCode: 404, Not found
  Warning  Failed          6m55s                  kubelet            Error: ErrImagePull
  Normal   Pulling         6m55s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Normal   Pulling         6m53s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m53s                  kubelet            Error: ErrImagePull
  Warning  Failed          6m53s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18": rpc error: code = Unknown desc = reading manifest sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18 in registry.redhat.io/openshift4/ose-csi-external-resizer: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442": rpc error: code = Unknown desc = reading manifest sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442 in registry.redhat.io/openshift4/ose-csi-livenessprobe: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ErrImagePull
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3"
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   Pulling         6m39s (x2 over 6m57s)  kubelet            Pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"
  Normal   BackOff         111s (x18 over 6m51s)  kubelet            Back-off pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"




Version of all relevant components (if applicable):
lvm 4.10 
ocp 4.10

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Lvm cluster is not created

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
No

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Install ocp4.10
2. install ocs catalogesource.
3. Create namespace+operator group+subscription for odf-lvm-operator
4. Create lvm-cluster

Actual results:
Multiple error regarding image pull in Topolvm-controller
  Warning  Failed          6m56s                  kubelet            Error: ErrImagePull
  Normal   Pulling         6m56s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3"
  Warning  Failed          6m56s                  kubelet            Failed to pull image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095": rpc error: code = Unknown desc = reading manifest sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095 in registry.redhat.io/odf4/odf-topolvm-rhel8: unknown: Not Found
  Warning  Failed          6m55s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3": rpc error: code = Unknown desc = reading manifest sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3 in registry.redhat.io/openshift4/ose-csi-external-provisioner: StatusCode: 404, Not found
  Warning  Failed          6m55s                  kubelet            Error: ErrImagePull
  Normal   Pulling         6m55s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Normal   Pulling         6m53s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m53s                  kubelet            Error: ErrImagePull
  Warning  Failed          6m53s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18": rpc error: code = Unknown desc = reading manifest sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18 in registry.redhat.io/openshift4/ose-csi-external-resizer: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442": rpc error: code = Unknown desc = reading manifest sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442 in registry.redhat.io/openshift4/ose-csi-livenessprobe: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ErrImagePull
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3"
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   Pulling         6m39s (x2 over 6m57s)  kubelet            Pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"
  Normal   BackOff         111s (x18 over 6m51s)  kubelet            Back-off pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"


Expected results:
Images should be pulled and topolvm-conroller should be running 

Additional info:
Comment 8 Shay Rozen 2022-02-01 15:30:55 UTC 
fixed in 4.10.130
Comment 13 errata-xmlrpc 2022-04-13 18:51:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.10.0 enhancement, security & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1372