2046254 – Topolvm-controller is failing to pull image

Bug 2046254 - Topolvm-controller is failing to pull image

Summary: Topolvm-controller is failing to pull image

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenShift Data Foundation
Classification:	Red Hat Storage
Component:	lvm-operator
Sub Component:
Version:	4.10
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	ODF 4.10.0
Assignee:	N Balachandran
QA Contact:	Shay Rozen
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-01-26 13:26 UTC by Shay Rozen
Modified:	2023-08-09 16:46 UTC (History)
CC List:	12 users (show)
Fixed In Version:	4.10.0-132
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-04-13 18:51:56 UTC
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	red-hat-storage lvm-operator pull 90	0	None	open	Bug 2046254: fix: sets env vars in the manager deployment	2022-01-27 15:12:36 UTC
Red Hat Product Errata	RHSA-2022:1372	0	None	None	None	2022-04-13 18:52:08 UTC

Description Shay Rozen 2022-01-26 13:26:39 UTC

Description of problem (please be detailed as possible and provide log
snippests):
When trying to create lvm-cluster Topolvm-controller is failing to pull image.
On pod describe you can see multiple errors:
  Warning  Failed          6m56s                  kubelet            Failed to pull image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095": rpc error: code = Unknown desc = reading manifest sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095 in registry.redhat.io/odf4/odf-topolvm-rhel8: unknown: Not Found
  Warning  Failed          6m55s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3": rpc error: code = Unknown desc = reading manifest sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3 in registry.redhat.io/openshift4/ose-csi-external-provisioner: StatusCode: 404, Not found
  Warning  Failed          6m55s                  kubelet            Error: ErrImagePull
  Normal   Pulling         6m55s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Normal   Pulling         6m53s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m53s                  kubelet            Error: ErrImagePull
  Warning  Failed          6m53s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18": rpc error: code = Unknown desc = reading manifest sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18 in registry.redhat.io/openshift4/ose-csi-external-resizer: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442": rpc error: code = Unknown desc = reading manifest sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442 in registry.redhat.io/openshift4/ose-csi-livenessprobe: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ErrImagePull
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3"
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   Pulling         6m39s (x2 over 6m57s)  kubelet            Pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"
  Normal   BackOff         111s (x18 over 6m51s)  kubelet            Back-off pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"




Version of all relevant components (if applicable):
lvm 4.10 
ocp 4.10

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Lvm cluster is not created

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
No

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Install ocp4.10
2. install ocs catalogesource.
3. Create namespace+operator group+subscription for odf-lvm-operator
4. Create lvm-cluster

Actual results:
Multiple error regarding image pull in Topolvm-controller
  Warning  Failed          6m56s                  kubelet            Error: ErrImagePull
  Normal   Pulling         6m56s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3"
  Warning  Failed          6m56s                  kubelet            Failed to pull image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095": rpc error: code = Unknown desc = reading manifest sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095 in registry.redhat.io/odf4/odf-topolvm-rhel8: unknown: Not Found
  Warning  Failed          6m55s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3": rpc error: code = Unknown desc = reading manifest sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3 in registry.redhat.io/openshift4/ose-csi-external-provisioner: StatusCode: 404, Not found
  Warning  Failed          6m55s                  kubelet            Error: ErrImagePull
  Normal   Pulling         6m55s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Normal   Pulling         6m53s                  kubelet            Pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m53s                  kubelet            Error: ErrImagePull
  Warning  Failed          6m53s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18": rpc error: code = Unknown desc = reading manifest sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18 in registry.redhat.io/openshift4/ose-csi-external-resizer: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442": rpc error: code = Unknown desc = reading manifest sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442 in registry.redhat.io/openshift4/ose-csi-livenessprobe: StatusCode: 404, Not found
  Warning  Failed          6m51s                  kubelet            Error: ErrImagePull
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:5c84de9ed6d4b202d326559341d61c63500bda32633af7dbd34ee40410ef5442"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:771101e1d1e84b553b25f457608af9cb17c03d54ee265b0bbb639c4169cf51b3"
  Normal   BackOff         6m51s                  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:437dbe55a80e2d4b81098881e796c166b2c48b41b3ed52ac7e9cbaca05dd1c18"
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Warning  Failed          6m51s                  kubelet            Error: ImagePullBackOff
  Normal   Pulling         6m39s (x2 over 6m57s)  kubelet            Pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"
  Normal   BackOff         111s (x18 over 6m51s)  kubelet            Back-off pulling image "registry.redhat.io/odf4/odf-topolvm-rhel8@sha256:a9907461ff68e9a5a818a36e6c83ace7f3711601fa75cb2c42a5bc9ec17f0095"


Expected results:
Images should be pulled and topolvm-conroller should be running 

Additional info:

Comment 8 Shay Rozen 2022-02-01 15:30:55 UTC

fixed in 4.10.130

Comment 13 errata-xmlrpc 2022-04-13 18:51:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.10.0 enhancement, security & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1372

Note You need to log in before you can comment on or make changes to this bug.