Bug 2028288 - MetalLB Image Pull Backoff Issue. Metal LB won't install successfully. [NEEDINFO]
Summary: MetalLB Image Pull Backoff Issue. Metal LB won't install successfully.
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Release
Version: 4.9
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.9.z
Assignee: Luke Meyer
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-01 22:11 UTC by bridget.rios
Modified: 2021-12-14 22:21 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-14 22:21:30 UTC
Target Upstream Version:
Embargoed:
bridget.rios: needinfo? (lmeyer)


Attachments (Terms of Use)

Description bridget.rios 2021-12-01 22:11:41 UTC
Description of problem:
This may be a duplicate of 2015239 but I'm getting an Image pull back off issue. I was told this issue would be fixed with the release of 4.9.8, but the problem is still there. 

Version-Release number of selected component (if applicable):
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.8     True        False         21h     Cluster version is 4.9.8

How reproducible:
Always

Steps to Reproduce:
1.
# oc get packagemanifests -n openshift-marketplace metallb-operator
NAME               CATALOG                       AGE
metallb-operator   Openshift Container Storage   16h

2.Create metallb namespace:
# cat << EOF | oc apply -f -
apiVersion: metallb.io/v1beta1
kind: Namespace
metadata:
  name: metallb-system
EOF

3.Create an Operator group custom resource in the namespace
# cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: metallb-operator
  namespace: metallb-system
spec:
  targetNamespaces:
  - metallb-system
EOF

4.Confirm the Operator group is installed in the namespace
# oc get operatorgroup -n metallb-system
NAME               AGE
metallb-operator   15h

5.Subscribe to the MetalLB Operator
# OC_VERSION=$(oc version -o yaml | grep openshiftVersion | \
    grep -o '[0-9]*[.][0-9]*' | head -1)

#cat << EOF| oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: metallb-operator-sub
  namespace: metallb-system
spec:
  channel: "${OC_VERSION}"
  name: metallb-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

6.Confirm the install plan is in the namespace
#oc get installplan -n metallb-system
NAME            CSV                                   APPROVAL    APPROVED
install-vct9x   metallb-operator.4.9.0-202109302016   Automatic   true

7.Verify that the Operator is installed
#oc get clusterserviceversion -n metallb-system \
>   -o custom-columns=Name:.metadata.name,Phase:.status.phase
Name                                  Phase
metallb-operator.4.9.0-202109302016   Failed

Actual results:
Name                                  Phase
metallb-operator.4.9.0-202109302016   Failed

oc get pods -n metallb-system
NAME                                                 READY   STATUS             RESTARTS   AGE
metallb-operator-controller-manager-b5f644cf-l4g9k   0/1     ImagePullBackOff   0          15h

Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  Failed   3h50m (x140 over 15h)   kubelet  Failed to pull image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d": rpc error: code = Unknown desc = reading manifest sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d in registry.redhat.io/openshift4/ose-metallb-operator: unknown: Not Found
  Normal   Pulling  14m (x182 over 15h)     kubelet  Pulling image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"
  Normal   BackOff  4m57s (x4052 over 15h)  kubelet  Back-off pulling image "registry.redhat.io/openshift4/ose-metallb-operator@sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"

Expected results:
Name                                  Phase
metallb-operator.4.9.0-nnnnnnnnnnnn   Succeeded

Additional info:
Used these two prior bugs 
Bug 2021461
Bug 2015239

Comment 1 bridget.rios 2021-12-01 22:13:41 UTC

*** This bug has been marked as a duplicate of bug 2015239 ***

Comment 3 Luke Meyer 2021-12-02 19:16:11 UTC
(In reply to bridget.rios from comment #0)
> Description of problem:
> This may be a duplicate of 2015239 but I'm getting an Image pull back off
> issue. I was told this issue would be fixed with the release of 4.9.8, but
> the problem is still there. 

I can see that you have updated the cluster to 4.9.8, but OLM operator updates are performed separately, and it looks like that hasn't happened.

> 7.Verify that the Operator is installed
> #oc get clusterserviceversion -n metallb-system \
> >   -o custom-columns=Name:.metadata.name,Phase:.status.phase
> Name                                  Phase
> metallb-operator.4.9.0-202109302016   Failed
> 
> Actual results:
> Name                                  Phase
> metallb-operator.4.9.0-202109302016   Failed
 
This is the one that shipped (broken) at GA. Along with 4.9.8 we shipped a new one; if you update, it should be metallb-operator.4.9.0-202111151318 (expect to be updated further next week FYI).

> oc get pods -n metallb-system
> NAME                                                 READY   STATUS         
> RESTARTS   AGE
> metallb-operator-controller-manager-b5f644cf-l4g9k   0/1    
> ImagePullBackOff   0          15h
> 
> Events:
>   Type     Reason   Age                     From     Message
>   ----     ------   ----                    ----     -------
>   Warning  Failed   3h50m (x140 over 15h)   kubelet  Failed to pull image
> "registry.redhat.io/openshift4/ose-metallb-operator@sha256:
> 2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d": rpc
> error: code = Unknown desc = reading manifest
> sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d in
> registry.redhat.io/openshift4/ose-metallb-operator: unknown: Not Found
>   Normal   Pulling  14m (x182 over 15h)     kubelet  Pulling image
> "registry.redhat.io/openshift4/ose-metallb-operator@sha256:
> 2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"
>   Normal   BackOff  4m57s (x4052 over 15h)  kubelet  Back-off pulling image
> "registry.redhat.io/openshift4/ose-metallb-operator@sha256:
> 2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"

metallb-operator.4.9.0-202111151318 refers to:
registry.redhat.io/openshift4/metallb-rhel8-operator@sha256:a4496f67da4c4bd049a07c3787a3327c7f4d47beb27d4f9a1f533cebd021a448
registry.redhat.io/openshift4/metallb-rhel8@sha256:4f3f63bb4333e03bfb7d4f5b280ea029a820706ac617602281d237318eabb11e

... which are the correct locations for the operator and operand.

I suspect if you update the operator, you will be successful. Can you confirm?

Comment 7 bridget.rios 2021-12-06 21:15:45 UTC
(In reply to Luke Meyer from comment #3)
> (In reply to bridget.rios from comment #0)
> > Description of problem:
> > This may be a duplicate of 2015239 but I'm getting an Image pull back off
> > issue. I was told this issue would be fixed with the release of 4.9.8, but
> > the problem is still there. 
> 
> I can see that you have updated the cluster to 4.9.8, but OLM operator
> updates are performed separately, and it looks like that hasn't happened.
> 
> > 7.Verify that the Operator is installed
> > #oc get clusterserviceversion -n metallb-system \
> > >   -o custom-columns=Name:.metadata.name,Phase:.status.phase
> > Name                                  Phase
> > metallb-operator.4.9.0-202109302016   Failed
> > 
> > Actual results:
> > Name                                  Phase
> > metallb-operator.4.9.0-202109302016   Failed
>  
> This is the one that shipped (broken) at GA. Along with 4.9.8 we shipped a
> new one; if you update, it should be metallb-operator.4.9.0-202111151318
> (expect to be updated further next week FYI).
> 
> > oc get pods -n metallb-system
> > NAME                                                 READY   STATUS         
> > RESTARTS   AGE
> > metallb-operator-controller-manager-b5f644cf-l4g9k   0/1    
> > ImagePullBackOff   0          15h
> > 
> > Events:
> >   Type     Reason   Age                     From     Message
> >   ----     ------   ----                    ----     -------
> >   Warning  Failed   3h50m (x140 over 15h)   kubelet  Failed to pull image
> > "registry.redhat.io/openshift4/ose-metallb-operator@sha256:
> > 2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d": rpc
> > error: code = Unknown desc = reading manifest
> > sha256:2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d in
> > registry.redhat.io/openshift4/ose-metallb-operator: unknown: Not Found
> >   Normal   Pulling  14m (x182 over 15h)     kubelet  Pulling image
> > "registry.redhat.io/openshift4/ose-metallb-operator@sha256:
> > 2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"
> >   Normal   BackOff  4m57s (x4052 over 15h)  kubelet  Back-off pulling image
> > "registry.redhat.io/openshift4/ose-metallb-operator@sha256:
> > 2389be5104b2c25fe67f9942550076806437740c6c1976ce675f49ddba053b0d"
> 
> metallb-operator.4.9.0-202111151318 refers to:
> registry.redhat.io/openshift4/metallb-rhel8-operator@sha256:
> a4496f67da4c4bd049a07c3787a3327c7f4d47beb27d4f9a1f533cebd021a448
> registry.redhat.io/openshift4/metallb-rhel8@sha256:
> 4f3f63bb4333e03bfb7d4f5b280ea029a820706ac617602281d237318eabb11e
> 
> ... which are the correct locations for the operator and operand.
> 
> I suspect if you update the operator, you will be successful. Can you
> confirm?

Are there steps to upgrade OLM? I tried but I'm afraid I was unsuccessful.

Comment 8 bridget.rios 2021-12-06 21:16:58 UTC
@lmeyer 
Are there steps to Upgrade OLM? I tried but was unfortunately unsuccessful.

Comment 9 bridget.rios 2021-12-14 22:21:30 UTC
Issue was fixed! Thank you!
[root.cp.fyre.ibm.com ~]# oc get installplan -n metallb-system
NAME            CSV                                   APPROVAL    APPROVED
install-lznjx   metallb-operator.4.9.0-202111291435   Automatic   true

[root.cp.fyre.ibm.com ~]# oc get clusterserviceversion -n metallb-system -o custom-columns=Name:.metadata.name,Phase:.status.phase 
Name                                  Phase
metallb-operator.4.9.0-202111291435   Succeeded

[root.cp.fyre.ibm.com ~]# oc get deployment -n metallb-system controller
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
controller   1/1     1            1           20s


Note You need to log in before you can comment on or make changes to this bug.