Bug 1821783

Summary: openshift-marketplace pods using images not present in release payload.
Product: OpenShift Container Platform Reporter: Luis Sanchez <sanchezl>
Component: OLMAssignee: Evan Cordell <ecordell>
OLM sub component: OperatorHub QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: wking
Version: 4.4   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:48:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luis Sanchez 2020-04-07 15:35:44 UTC
openshift-marketplace/certified-operators and openshift-marketplace/redhat-operators pods not using images from release payload after 4.1->4.2->4.3->4.4 upgrade.


buildlog.txt snippet:

[It] [Top Level] [Feature:Platform] Managed cluster should ensure pods use downstream images from our release image with proper ImagePullPolicy [Suite:openshift/conformance/parallel]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/operators/images.go:21
Apr  7 06:34:53.134: INFO: Running 'oc --namespace= --kubeconfig= adm release info --pullspecs -o json --registry-config /tmp/image-pull-secret636389395'
Apr  7 06:34:54.536: INFO: Release Info image=registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-04-04-025830
Apr  7 06:34:54.536: INFO: Release Info number of tags 109

Apr  7 06:37:23.038: INFO: Pods found with invalid container images not present in release payload: openshift-marketplace/certified-operators-797cc58648-47qbj/certified-operators image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c517ffebffb8f5ffd3cc04f2491270b59b61079ca889be664945e166f90f420
openshift-marketplace/community-operators-849dd49d8d-tl5t7/community-operators image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c517ffebffb8f5ffd3cc04f2491270b59b61079ca889be664945e166f90f420
openshift-marketplace/redhat-operators-ffb9b86c6-dr74f/redhat-operators image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c517ffebffb8f5ffd3cc04f2491270b59b61079ca889be664945e166f90f420
Apr  7 06:37:23.038: INFO: Pods found with invalid container image pull policy not equal to IfNotPresent: 
Apr  7 06:37:23.038: INFO: Pods with invalid dowstream images: 
Apr  7 06:37:23.039: FAIL: Pods found with invalid container images not present in release payload: openshift-marketplace/certified-operators-797cc58648-47qbj/certified-operators image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c517ffebffb8f5ffd3cc04f2491270b59b61079ca889be664945e166f90f420
openshift-marketplace/community-operators-849dd49d8d-tl5t7/community-operators image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c517ffebffb8f5ffd3cc04f2491270b59b61079ca889be664945e166f90f420
openshift-marketplace/redhat-operators-ffb9b86c6-dr74f/redhat-operators image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c517ffebffb8f5ffd3cc04f2491270b59b61079ca889be664945e166f90f420


CI Search:

https://search.svc.ci.openshift.org/?search=Pods+found+with+invalid+container+images+not+present+in+release+payload.*+openshift-marketplace.certified-operators&maxAge=48h&context=2&type=bug%2Bjunit

Comment 1 Jian Zhang 2020-04-10 06:24:33 UTC
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-04-09-220855   True        False         168m    Cluster version is 4.4.0-0.nightly-2020-04-09-220855

mac:~ jianzhang$ oc get pods
NAME                                    READY   STATUS    RESTARTS   AGE
certified-operators-7c78cf8594-flg7d    1/1     Running   0          83s
community-operators-5645f84b57-kwsf8    1/1     Running   0          85s
marketplace-operator-59ffc649dd-2h2vk   1/1     Running   0          84m
quay-applications-d65844df4-lq8zw       1/1     Running   0          84m
redhat-marketplace-5c598d8c9-r2gcf      1/1     Running   0          85s
redhat-operators-748f96cbd8-dqbvv       1/1     Running   0          84s
mac:~ jianzhang$ oc exec certified-operators-7c78cf8594-flg7d   -- cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)
mac:~ jianzhang$ oc exec redhat-operators-748f96cbd8-dqbvv    -- cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)
mac:~ jianzhang$ oc exec community-operators-5645f84b57-kwsf8 -- cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)
mac:~ jianzhang$ oc exec marketplace-operator-59ffc649dd-2h2vk -- cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)
mac:~ jianzhang$ oc exec redhat-marketplace-5c598d8c9-r2gcf -- cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.8 (Maipo)

As we can see, they did use the downstream image on 4.4.0-0.nightly-2020-04-09-220855 cluster, verify it.

Comment 4 errata-xmlrpc 2020-05-04 11:48:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581

Comment 5 W. Trevor King 2020-05-08 04:52:56 UTC
I'm still seeing these in CI.  Opened bug 1833207 to track, since this one is already CLOSED ERRATA.