Bug 2015239

Summary: MetalLB Operator: Unable to install the operator because of inability to pull required images
Product: OpenShift Container Platform Reporter: Arti Sood <asood>
Component: ReleaseAssignee: Luke Meyer <lmeyer>
Status: CLOSED CURRENTRELEASE QA Contact: Arti Sood <asood>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.9CC: aos-bugs, bridget.rios, gmargali, jdelft, jokerman, lmeyer, mifiedle
Target Milestone: ---Keywords: Reopened
Target Release: 4.9.z   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-12 09:50:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Arti Sood 2021-10-18 17:37:36 UTC
Description of problem:
Set up a cluster on BM (upi-on-baremetal/versioned-installer-openstack)

Debugging from ART team.

Comet shows the two metallb images shipped to openshift4/metallb-rhel8-operator and openshift4/metallb-rhel8 but in ocp-build-data we have them configured with the names ose-metallb-operator and ose-metallb which is what ended up in the bundle pullspecs.

the two need to match. 


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Log into the web console and attempt to install the operator in metallb-system project.


Actual results:
oc describe pod metallb-operator-controller-manager-7c846b4cc5-qmtdk -n metallb-system

It fails with error:-
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       14m                   default-scheduler  Successfully assigned metallb-system/metallb-operator-controller-manager-7c846b4cc5-qmtdk to asood-10183-bm-j6b58-compute-0
  Normal   AddedInterface  14m                   multus             Add eth0 [10.128.4.9/23] from ovn-kubernetes
  Normal   Pulling         13m (x4 over 14m)     kubelet            Pulling image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"
  Warning  Failed          13m (x4 over 14m)     kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d": rpc error: code = Unknown desc = reading manifest sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d in registry.redhat.io/openshift4/ose-metallb-operator: unknown: Not Found
  Warning  Failed          13m (x4 over 14m)     kubelet            Error: ErrImagePull
  Warning  Failed          13m (x6 over 14m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff         4m44s (x43 over 14m)  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"


Expected results:
The components should be installed successfully.

Additional info:

Comment 1 Luke Meyer 2021-10-18 17:53:23 UTC
Fixed when ose-metallb-operator-bundle-container-v4.9.0.202109302016.p0.git.9f0a9fc.assembly.stream-4 or later is released.

https://github.com/openshift/ocp-build-data/commit/55a50b2d9d4b175933ce512cf25ec907b1afd1ae fixes future builds.

Comment 7 errata-xmlrpc 2021-10-20 11:32:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.0 OLM Operators metadata update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3895

Comment 9 bridget.rios 2021-11-11 22:39:39 UTC
I'm experiencing the same issue. I've upgraded my cluster to the latest 4.5.9 and am still experiencing an image pull issue. 

Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       12m                   default-scheduler  Successfully assigned metallb-system/metallb-operator-controller-manager-79dc574bb5-8lqpr to worker1.bri.cp.fyre.ibm.com
  Normal   AddedInterface  12m                   multus             Add eth0 [10.254.20.15/22] from openshift-sdn
  Normal   Pulling         11m (x4 over 12m)     kubelet            Pulling image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"
  Warning  Failed          11m (x4 over 12m)     kubelet            Failed to pull image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d": rpc error: code = Unknown desc = reading manifest sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d in registry.redhat.io/openshift4/ose-metallb-operator: unknown: Not Found
  Warning  Failed          11m (x4 over 12m)     kubelet            Error: ErrImagePull
  Warning  Failed          10m (x6 over 12m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff         2m40s (x40 over 12m)  kubelet            Back-off pulling image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"

Comment 10 Joep van Delft 2021-11-12 09:50:00 UTC
This is fixed now, as ose-metallb{,-operator} images have been built. The release getting out around the 23rd.

Closing this bug as CURRENTRELEASE, as it has been shipped before. If a customer notification is necessary, please clone this bug. Thanks!

Comment 11 Guy Margalit 2021-11-12 19:33:19 UTC
Joep can you explain why this is still an issue for Bridget with OCP 4.9.5? I didn't understand your answer.

Comment 13 Joep van Delft 2021-11-15 11:08:16 UTC
Images have been built. The built images have not been shipped yet. That's why 4.9.5 does not have the fix.

This bug has been closed and shipped in the Customer Portal before. We cannot fix/ship same bug twice, reopening a shipped bug should not be possible, but is. Please open another bug as a clone of this one if a customer notification in the Portal is warranted.

Comment 14 Guy Margalit 2021-11-15 11:21:57 UTC
(In reply to Joep van Delft from comment #13)
> Images have been built. The built images have not been shipped yet. That's
> why 4.9.5 does not have the fix.

Thank you Joep.

I have couple of questions in Bridget's name:
 1. When is the fix expected to be shipped?
 2. Is that bound to an openshift release and will be included in 4.9.6?
 3. Is there a workaround for 4.9.5?

Thanks in advance.

Comment 15 bridget.rios 2021-12-01 22:13:41 UTC
*** Bug 2028288 has been marked as a duplicate of this bug. ***