Bug 1907386
| Summary: | Release update service operator for OCP 4.6 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Lalatendu Mohanty <lmohanty> |
| Component: | OpenShift Update Service | Assignee: | Lalatendu Mohanty <lmohanty> |
| OpenShift Update Service sub component: | operator | QA Contact: | liujia <jiajliu> |
| Status: | CLOSED ERRATA | Docs Contact: | Kathryn Alexander <kalexand> |
| Severity: | high | ||
| Priority: | unspecified | CC: | ableisch, cstark, yanyang |
| Version: | 4.6 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-02 05:39:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Lalatendu Mohanty
2020-12-14 12:05:10 UTC
Hi Lalatendu Mohanty I just found an inconsistent digest in operator build, could u double confirm if it's a build issue and is a re-build needed for osus operator? About cincinnati-operator-container-v4.6.0-1(https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1397370), the image info is as following. ``` 'index': {'digests': {'application/vnd.docker.distribution.manifest.list.v2+json': 'sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8'}, 'floating_tags': ['latest', 'v4.6.0'], 'pull': ['registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator@sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8', 'registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0-1'], 'tags': ['v4.6.0-1'], 'unique_tags': ['rhaos-4.6-rhel-8-containers-candidate-59532-20201201130149']}, ``` Pull above image with tag(v4.6.0-1) succeed, but pull it with digest(sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8) will fail. Another interesting finding is that, from bundle image build(https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1421388), the referred operator image is registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator@sha256:59e6e4cf633c2a4143693962d21564d6d5fec2b9078a66500d3d2f8476e6daca, and it can be pulled successfully with the digest. So the digest info in osus operator build(sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8) seems not correct. # podman images|grep update-service-operator registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator sha256:59e6e4cf633c2a4143693962d21564d6d5fec2b9078a66500d3d2f8476e6daca dce4714049a8 2 weeks ago 140 MB registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator v4.6.0-1 dce4714049a8 2 weeks ago 140 MB You are right, I should have used "registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator@sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8" as this is the manifest based digest and registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator@sha256:59e6e4cf633c2a4143693962d21564d6d5fec2b9078a66500d3d2f8476e6daca was x86_64 specific digest The issue still existed. OSUS operator image: v4.6.0-3 Bundle image: v1.0-4 The operator image pullspecs with digest in bundle manifest is @sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8(no image available). # podman inspect registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0-3|jq .[].Digest "sha256:9f778b57cec0947e34c94b2c6ad024179c823c50f9a25393ec437728ce2f892a" # podman inspect registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0-2|jq .[].Digest "sha256:54e9f0a63418d734588b4bef887615bad34314551131e65460f300a423430e57" # podman inspect registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0-1|jq .[].Digest "sha256:59e6e4cf633c2a4143693962d21564d6d5fec2b9078a66500d3d2f8476e6daca" The tag v4.6.0-3 does match the expected operator digest @sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8,so no correct operator image could be found and mirrored to local registry. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 7h59m Successfully assigned mytest/updateservice-operator-77c6cd74f9-zvcbh to jliu468-ksntb-w-b-1.c.openshift-qe.internal Normal AddedInterface 7h59m multus Add eth0 [10.128.2.17/23] Warning Failed 7h58m kubelet, jliu468-ksntb-w-b-1.c.openshift-qe.internal Failed to pull image "registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8": rpc error: code = Unknown desc = (Mirrors also failed: [jliu468.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openshift-update-service/openshift-update-service-rhel8-operator@sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8: Error reading manifest sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8 in jliu468.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openshift-update-service/openshift-update-service-rhel8-operator: manifest unknown: manifest unknown]): registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8: error pinging docker registry registry.redhat.io: Get "https://registry.redhat.io/v2/": dial tcp 104.86.16.95:443: i/o timeout Warning Failed 7h58m kubelet, jliu468-ksntb-w-b-1.c.openshift-qe.internal Error: ErrImagePull Normal BackOff 7h58m kubelet, jliu468-ksntb-w-b-1.c.openshift-qe.internal Back-off pulling image "registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8" Warning Failed 7h58m kubelet, jliu468-ksntb-w-b-1.c.openshift-qe.internal Error: ImagePullBackOff Normal Pulling 7h58m (x2 over 7h59m) kubelet, jliu468-ksntb-w-b-1.c.openshift-qe.internal Pulling image "registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:fd2d4ed73703ea4c06329d90104b4dd858ea34b7f513601c6d93f3fa9ed140e8" I have done some more changes to fix the issue and the build cincinnati-operator-metadata-container-v1.0-9 is attached to the errata [1]. Please check if the issue is fixed. [1] https://errata.devel.redhat.com/advisory/67040 The install of osus operator still failed with the bundle image. OSUS operator image: v4.6.0-4 Bundle image: v1.0-10 # ./oc get po -n openshift-update-service NAME READY STATUS RESTARTS AGE updateservice-operator-7bcc5d7fc6-zsvg6 0/1 ImagePullBackOff 0 4m38s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 8h default-scheduler Successfully assigned openshift-update-service/updateservice-operator-7bcc5d7fc6-zsvg6 to jliu-a46-m9xlz-w-b-1.c.openshift-qe.internal Normal AddedInterface 8h multus Add eth0 [10.128.2.14/23] Normal Pulling 7h59m (x4 over 8h) kubelet Pulling image "registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0" Warning Failed 7h59m (x4 over 8h) kubelet Failed to pull image "registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0": rpc error: code = Unknown desc = error pinging docker registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": dial tcp: lookup registry-proxy.engineering.redhat.com on 169.254.169.254:53: no such host Warning Failed 7h59m (x4 over 8h) kubelet Error: ErrImagePull Normal BackOff 7h58m (x6 over 8h) kubelet Back-off pulling image "registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0" Warning Failed 7h58m (x6 over 8h) kubelet Error: ImagePullBackOff ICSP was created and applied successfully. # ./oc get imagecontentsourcepolicies.operator.openshift.io image-policy-osus -ojson|jq .spec { "repositoryDigestMirrors": [ { "mirrors": [ "jliu-a46.mirror-registry.qe.gcp.devcluster.openshift.com:5000" ], "source": "registry-proxy.engineering.redhat.com" } ] } From pod's log, the image still failed to be pulled, which means the icsp does not take effect. I guess it should because we use image tag but not digest in csv since only digest make sense for icsp. # ./oc describe deployment updateservice-operator |grep Image Image: registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0 So try to replace the tag with digest in deployment and osus operator was deployed successfully. # ./oc describe deployment updateservice-operator |grep Image Image: registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator@sha256:9d4ff2c8a5c7b7c037567505337a9405da276e9b54ddb3fb971f88262751d848 # ./oc get po -n openshift-update-service NAME READY STATUS RESTARTS AGE updateservice-operator-84d88c6477-86m4j 1/1 Running 0 13m Above issues can be reproduced as following steps:
1. Install disconnected cluster with a local registry and disable the default catalog source
# ./oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
2. Build index image with bundle image(cincinnati-operator-metadata-container-v1.0-10) with opm tool
# skopeo copy docker://registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-cincinnati-operator-metadata-container:v1.0-10 docker://${DIS_REGISTRY}/rh-osbs/cincinnati-cincinnati-operator-metadata-container:v1.0-10 --src-tls-verify=false
# ./bin/opm index add --bundles ${DIS_REGISTRY}/rh-osbs/cincinnati-cincinnati-operator-metadata-container:v1.0-10 --tag ${DIS_REGISTRY}/rh-osbs/osus-index:1.1 --container-tool podman
3. Mirror operator(cincinnati-operator-container-v4.6.0-4) and index image to local registry
# podman push ${DIS_REGISTRY}/rh-osbs/osus-index:1.1
# skopeo copy docker://registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0-4 docker://${DIS_REGISTRY}/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0 --src-tls-verify=false
4. Create ImageContentSourcePolicy for registry-proxy
# cat icsp_up.yaml
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: image-policy-osus
spec:
repositoryDigestMirrors:
- mirrors:
- jliu-a46.mirror-registry.qe.gcp.devcluster.openshift.com:5000
source: registry-proxy.engineering.redhat.com
# ./oc create -f icsp_up.yaml
5. Create CatalogSource with above index image
# cat cs.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: osus-cs
namespace: openshift-marketplace
spec:
sourceType: grpc
image: jliu-a46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/rh-osbs/osus-index:1.1
displayName: osus operator
publisher: osus qe
# ./oc create -f cs.yaml
6. Login into the web console, and go to the Operators → OperatorHub page:
Choose the osus operator and Install with default variables.
Then you check the operator installation failed, and check more details as comment9.
Still ran into the issue using cincinnati-operator-metadata-container-v1.0-11 and cincinnati-operator-container-v4.6.0-4. # oc get all NAME READY STATUS RESTARTS AGE pod/updateservice-operator-7c49795d9b-fls2t 0/1 ImagePullBackOff 0 3m5s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/updateservice-operator-metrics ClusterIP 172.30.2.227 <none> 8443/TCP 3m6s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/updateservice-operator 0/1 1 0 3m5s NAME DESIRED CURRENT READY AGE replicaset.apps/updateservice-operator-7c49795d9b 1 1 0 3m5s # oc get event 110s Normal Pulling pod/updateservice-operator-7c49795d9b-fls2t Pulling image "registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0" 110s Warning Failed pod/updateservice-operator-7c49795d9b-fls2t Failed to pull image "registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0": rpc error: code = Unknown desc = error pinging docker registry registry-proxy.engineering.redhat.com: Get "https://registry-proxy.engineering.redhat.com/v2/": dial tcp: lookup registry-proxy.engineering.redhat.com on 169.254.169.254:53: no such host 110s Warning Failed pod/updateservice-operator-7c49795d9b-fls2t Error: ErrImagePull 97s Normal BackOff pod/updateservice-operator-7c49795d9b-fls2t Back-off pulling image "registry-proxy.engineering.redhat.com/rh-osbs/openshift-update-service-openshift-update-service-operator:v4.6.0" 2m1s Warning Failed pod/updateservice-operator-7c49795d9b-fls2t Error: ImagePullBackOff OSUS operator image: v4.6.0-6(https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1506385) Bundle image: v1.0-20 The issue in comment9 has been fixed(with digest instead of tag), but we ran back into the previous issue in comment7. # ./oc get po -n openshift-update-service NAME READY STATUS RESTARTS AGE updateservice-operator-5457769dd4-t5cds 0/1 ImagePullBackOff 0 15m # ./oc describe po updateservice-operator-5457769dd4-t5cds -n openshift-update-service Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 8h default-scheduler Successfully assigned openshift-update-service/updateservice-operator-5457769dd4-t5cds to jliu-a46-2sq5j-w-b-1.c.openshift-qe.internal Normal AddedInterface 8h multus Add eth0 [10.128.2.8/23] Normal Pulling 8h (x4 over 8h) kubelet Pulling image "registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4" Warning Failed 8h (x4 over 8h) kubelet Failed to pull image "registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4": rpc error: code = Unknown desc = (Mirrors also failed: [jliu-a46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openshift-update-service/openshift-update-service-rhel8-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4: Error reading manifest sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4 in jliu-a46.mirror-registry.qe.gcp.devcluster.openshift.com:5000/openshift-update-service/openshift-update-service-rhel8-operator: manifest unknown: manifest unknown]): registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4: error pinging docker registry registry.redhat.io: Get "https://registry.redhat.io/v2/": dial tcp 23.222.141.28:443: i/o timeout Warning Failed 8h (x4 over 8h) kubelet Error: ErrImagePull Warning Failed 8h (x7 over 8h) kubelet Error: ImagePullBackOff Normal BackOff 7h59m (x38 over 8h) kubelet Back-off pulling image "registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4" The tag(v4.6.0-6/v4.6.0) of operator image does match the expected operator digest @sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4,so no correct operator image could be fetched. # podman inspect registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-openshift-update-service-operator:v4.6.0-6|jq .[].Digest "sha256:f2ae92c14fc1cac8da180c019a22d38700bc5eb4224b2d9c41bb322f19fcdf10" # podman inspect registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-openshift-update-service-operator:v4.6.0|jq .[].Digest "sha256:f2ae92c14fc1cac8da180c019a22d38700bc5eb4224b2d9c41bb322f19fcdf10" # podman pull registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-openshift-update-service-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4 Trying to pull docker://registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-openshift-update-service-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4...Failed (0x179ec00,0xc4203f8a20) Error: error pulling image "registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-openshift-update-service-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4": Invalid image name "registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-openshift-update-service-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4", unknown transport "registry-proxy.engineering.redhat.com/rh-osbs/cincinnati-openshift-update-service-operator@sha256" I guess the problem should be the `image` reference does not match with correct target operator digest. But i'm not quite sure how we build the bundle image to update the csv in it. # ./oc get csv -n openshift-update-service -oyaml|grep -A2 image: image: registry.redhat.io/openshift-update-service/openshift-update-service-rhel8-operator@sha256:b4e59e1987401198c44be8683211cedc76d05dc7ee7923b35c9e8aede0e8c3d4 imagePullPolicy: Always name: updateservice-operator Retest with the latest bundle and operator image after resolve the image digest issue. The operator can be deployed successfully. # ./oc get po NAME READY STATUS RESTARTS AGE updateservice-operator-888d8784c-hnchb 1/1 Running 0 53s Moving to on_qa with "cincinnati-operator-metadata-container-v1.0-20" According to comment13, verify the bug. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHEA: OSUS enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2196 |