Bug 2215083

Summary: ImagePullBackOff for cluster-network-addons-operator and hco-operator
Product: Container Native Virtualization (CNV) Reporter: Alberto Gonzalez <alberto.gonzalez>
Component: InstallationAssignee: Simone Tiraboschi <stirabos>
Status: NEW --- QA Contact: Natalie Gavrielov <ngavrilo>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.10.9CC: dholler, stirabos
Target Milestone: ---Flags: stirabos: needinfo? (alberto.gonzalez)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alberto Gonzalez 2023-06-14 17:10:47 UTC 
Description of problem:

Images for version 4.10.9 are rebased and now the digest is not the correct. Pulling the images for cluster-network-addons-operator and hco-operator fails


Version-Release number of selected component (if applicable):
4.10.9


How reproducible:
Install CNV 4.10.9


Actual results:

Back-off pulling image "registry.redhat.io/container-native-virtualization/cluster-network-addons-operator@sha256:74433260a14cc51099f875e6b2de763cb9a61661596bbd899d902be93c90a524"

Back-off pulling image "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:d6672501dc0ae1661a89db70b2f400b4fb5025c5724f937c2fd9c90d0d9c8cc0"


Expected results:

Pods running properly


Additional info:
[lab-user@provision ~]$ oc describe PackageManifest kubevirt-hyperconverged|grep -e addons-operator -e hyperconverged-cluster-operator
        Container Image:                                  registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:d6672501dc0ae1661a89db70b2f400b4fb5025c5724f937c2fd9c90d0d9c8cc0
        Repository:                                       https://github.com/kubevirt/hyperconverged-cluster-operator
        registry.redhat.io/container-native-virtualization/cluster-network-addons-operator@sha256:74433260a14cc51099f875e6b2de763cb9a61661596bbd899d902be93c90a524
        registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:d6672501dc0ae1661a89db70b2f400b4fb5025c5724f937c2fd9c90d0d9c8cc0


As you can see diesg images are different from yesterday:
https://catalog.redhat.com/software/containers/container-native-virtualization/hyperconverged-cluster-operator/5cac6a51d70cc57c44aaa9fe?tag=v4.10.9-4&push_date=1686670676000

https://catalog.redhat.com/software/containers/container-native-virtualization/cluster-network-addons-operator/5c8a92285a13464733ed08af?tag=v4.10.9-4&push_date=1686670676000
Comment 1 Simone Tiraboschi 2023-08-10 14:23:27 UTC 
Today everything works as expected for me: those 2 images are correctly pullable.

Is this still relevant?


stirabos@t14s:~$ skopeo inspect docker://registry.redhat.io/container-native-virtualization/cluster-network-addons-operator@sha256:74433260a14cc51099f875e6b2de763cb9a61661596bbd899d902be93c90a524
{
    "Name": "registry.redhat.io/container-native-virtualization/cluster-network-addons-operator",
    "Digest": "sha256:74433260a14cc51099f875e6b2de763cb9a61661596bbd899d902be93c90a524",
    "RepoTags": [
        "v4.8.2",
        "v4.8.5-4",
        "v2.6.0-17",
        "v2.4.7",
        "v2.5.1-1",
        "v4.8.2-1",
        "v2.4.7-4",
        "v2.4.7-5",
        "v2.4.3",
        "v2.4.2",
        "v4.8.6-6-source",
        "v2.6.9-6",
        "v2.5.7",
        "v4.8.4",
        "v2.4.1",
        "v4.8.3",
        "v4.8.0",
        "v4.10.0-49",
        "v2.4.0",
        "v4.8.1",
        "v2.4.5",
        "v4.8.6-6",
        "v4.8.6",
        "v2.1.0",
        "v2.4.6-1",
        "v2.4.2-1",
        "v2.4.6",
        "v4.9.3-3",
        "v2.5.1",
        "v2.1.0-13",
        "v2.4.4",
        "v2.4.0-38",
        "v4.10.0",
        "v2.6.6",
        "v2.5.6",
        "v4.8.3-8",
        "v4.9.5-5-source",
        "v2.6.7-3",
        "v2.4.9",
        "v2.5.3",
        "v4.9.5",
        "v4.10.0-49-source",
        "v2.6.10-5-source",
        "v4.10.1-12-source",
        "v4.10.2-3",
        "v2.3.0-28",
        "v4.10.1",
        "2.4",
        "v2.5.5",
        "v4.8.1-2",
        "v2.2.0-8",
        "v4.9.4-7-source",
        "2.6",
        "v2.4.4-1",
        "v2.5.2",
        "v2.6.10",
        "v4.9.3-3-source",
        "v2.6.0",
        "v2.5.8",
        "v2.2.0",
        "v2.6.8-1",
        "v4.8.4-3",
        "v2.6.10-5",
        "v2.5.6-3",
        "v2.6.5",
        "v4.9.0-27",
        "v4.9.1",
        "v4.10.2",
        "v2.5.4-4",
        "v2.6.5-2",
        "v4.9.1-9-source",
        "v2.6.3",
        "v2.6.3-1",
        "v2.0.0-16",
        "v2.4.9-1",
        "v4.10.4",
        "v4.9.5-5",
        "v4.8.5",
        "v4.9.4-7",
        "v2.6.1-2",
        "v2.5.0",
        "v2.6.1",
        "v2.5.2-4",
        "v4.9.0",
        "v4.9.4",
        "v4.9.2",
        "v4.10.2-3-source",
        "v4.10.3",
        "v2.5.4",
        "v2.0.0",
        "v4.9.2-1",
        "v2.6.9",
        "v2.5.0-17",
        "v4.10.1-12",
        "v2.3.0",
        "v4.10.3-4",
        "v4.8.4-3-source",
        "v4.8.0-28",
        "v2.5.8-2",
        "v4.10.4-2",
        "v2.5.7-1",
        "v2.6.9-6-source",
        "v4.10.4-2-source",
        "v4.9.1-9",
        "v4.9.3",
        "v2.6.2",
        "v2.6.8",
        "v4.8.5-4-source",
        "v2.6.2-1",
        "v2.4.1-2",
        "v2.4.3-2",
        "v4.10.3-4-source",
        "v2.6.7",
        "v4.9.2-1-source",
        "v2.5.5-2",
        "v2.6.6-4",
        "v2.4.5-1",
        "v2.5.3-3",
        "v4.10.5",
        "v4.10.5-1",
        "v4.10.5-1-source",
        "v4.11.0-26-source",
        "v4.9.6-5-source",
        "v4.9.7-4-source",
        "v4.10.6-3-source",
        "v4.8.7",
        "v4.11.1-8",
        "v4.10.6",
        "v4.9.6",
        "v4.11.1-8-source",
        "v4.10.7-3",
        "v4.8.7-5-source",
        "v4.11.0-26",
        "v4.9.7-4",
        "v4.9",
        "v4.9.7",
        "v4.11.1",
        "v4.11.0",
        "v4.10.7",
        "v4.10.7-3-source",
        "v4.9.6-5",
        "v4.10.6-3",
        "v4.8.7-5",
        "v4.11.3-3-source",
        "v4.10.8",
        "v4.11.2-1-source",
        "v4.10.8-8",
        "v4.11.3-3",
        "v4.12.0",
        "v4.11.3",
        "v4.11.2",
        "v4.11.2-1",
        "v4.12.2-2",
        "v4.12.2-2-source",
        "v4.12.0-24",
        "v4.12.2",
        "v4.12.1",
        "v4.12.1-4",
        "v4.12.0-24-source",
        "v4.12.1-4-source",
        "v4.10.8-8-source",
        "v4.11.4-7",
        "v4.11.4",
        "v4.12.3",
        "v4.12.3-2",
        "v4.11.4-7-source",
        "v4.12.3-2-source",
        "v4.10.9",
        "v4.10.9-4",
        "v4.10.9-4-source",
        "v4.12.4-6",
        "v4.12.4",
        "v4.12.4-6-source",
        "v4.10.10-4",
        "v4.10",
        "v4.10.10",
        "v4.10.10-4-source",
        "v4.11.5-4",
        "v4.11.5",
        "v4.11",
        "v4.11.5-4-source",
        "v4.12.5-7",
        "v4.12",
        "v4.12.5",
        "v4.12.5-7-source"
    ],
    "Created": "2023-04-25T22:06:49.036615675Z",
    "DockerVersion": "",
    "Labels": {
        "architecture": "x86_64",
        "build-date": "2023-04-25T22:01:42",
        "com.redhat.component": "cluster-network-addons-operator-container",
        "com.redhat.license_terms": "https://www.redhat.com/agreements",
        "description": "Cluster Network Addons Operator",
        "distribution-scope": "public",
        "io.buildah.version": "1.27.3",
        "io.k8s.description": "Cluster Network Addons Operator",
        "io.k8s.display-name": "cluster-network-addons-operator",
        "io.openshift.expose-services": "",
        "io.openshift.tags": "cnv,operator,network",
        "maintainer": "phoracek",
        "name": "container-native-virtualization/cluster-network-addons-operator",
        "release": "4",
        "summary": "Cluster Network Addons Operator",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/container-native-virtualization/cluster-network-addons-operator/images/v4.10.9-4",
        "vcs-ref": "b7de991426d2f71e4276ab92f6aaa0b1babcfb8c",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "v4.10.9"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:7d8746ab4ad8e51637d5a71d17e876a7432c1bb34ba3db89985a0cb9500cd64b",
        "sha256:301090ddad26866fd2cf99e9290b9e652cc2f04b1ea10df969202f21dd8986c2"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "Digest": "sha256:7d8746ab4ad8e51637d5a71d17e876a7432c1bb34ba3db89985a0cb9500cd64b",
            "Size": 82259072,
            "Annotations": null
        },
        {
            "MIMEType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "Digest": "sha256:301090ddad26866fd2cf99e9290b9e652cc2f04b1ea10df969202f21dd8986c2",
            "Size": 39309638,
            "Annotations": null
        }
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "container=oci",
        "ENTRYPOINT=/entrypoint",
        "OPERATOR=/cluster-network-addons-operator",
        "MANIFEST_TEMPLATOR=/manifest-templator",
        "CSV_TEMPLATE=/cluster-network-addons-operator.VERSION.clusterserviceversion.yaml.in",
        "USER_UID=1001",
        "USER_NAME=cluster-network-addons-operator",
        "SOURCE_DIR=/remote-source/app"
    ]
}
stirabos@t14s:~$ skopeo inspect docker://registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator@sha256:d6672501dc0ae1661a89db70b2f400b4fb5025c5724f937c2fd9c90d0d9c8cc0
{
    "Name": "registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator",
    "Digest": "sha256:d6672501dc0ae1661a89db70b2f400b4fb5025c5724f937c2fd9c90d0d9c8cc0",
    "RepoTags": [
        "v2.5.3-3",
        "v4.8.1-3",
        "v2.6.9",
        "v4.8.5-6",
        "v4.8.6",
        "v2.4.4",
        "v2.6.9-4",
        "v2.5.0-53",
        "v4.9.2-6",
        "v2.4.4-1",
        "v2.5.1-2",
        "v2.6.5",
        "v2.4.9",
        "v2.4.9-2",
        "v2.5.1",
        "v4.10.2-2-source",
        "v4.10.3",
        "v4.8.2-2",
        "v2.6.10",
        "v2.6.1-7",
        "v4.9.3-4",
        "v2.5.5-3",
        "v4.8.1",
        "v2.4.2-1",
        "v2.4.8",
        "v4.9.4-5",
        "v2.0.0-32",
        "v4.10.4-2-source",
        "v4.10.1",
        "v2.2.0",
        "v4.10.2",
        "v4.10.0-133",
        "v2.5.4",
        "v2.0.0",
        "v2.5.2-3",
        "v2.4.5",
        "v2.4.0",
        "v2.6.7-1",
        "v2.5.0",
        "v4.10.2-2",
        "v4.9.2",
        "v2.6.5-1",
        "v2.6.2",
        "v2.1.0",
        "v2.6.6-4",
        "v2.5.5",
        "v2.4.1",
        "v2.5.7",
        "v2.4.0-70",
        "v2.6.3-3",
        "v4.8.3-5",
        "v2.6.8",
        "v2.6.10-4-source",
        "v4.9.1",
        "v4.8.5",
        "v4.9.5-6-source",
        "v2.5.6-5",
        "v2.3.0-61",
        "v4.10.3-4",
        "v2.5.2",
        "v2.4.6-2",
        "v2.3.0",
        "v4.8.6-3",
        "v2.6.0-77",
        "v4.9.5-6",
        "v2.4.2",
        "v4.8.4-4",
        "v2.5.4-6",
        "v4.8.2",
        "v2.6.8-1",
        "v2.4.7",
        "v4.8.5-6-source",
        "v4.9.1-6-source",
        "v4.8.3",
        "v4.9.2-6-source",
        "v4.8.0-62",
        "v2.4.8-3",
        "v2.6.6",
        "v2.4.3-2",
        "v4.8.0",
        "v4.8.4-4-source",
        "v4.10.3-4-source",
        "v2.5.8-2",
        "v4.10.0",
        "v2.6.9-4-source",
        "2.6",
        "v4.10.4-2",
        "v4.9.4-5-source",
        "v4.10.0-133-source",
        "v4.9.5",
        "v4.9.0",
        "v4.8.6-3-source",
        "v4.10.1-19-source",
        "v4.9.3",
        "v2.1.0-22",
        "v2.5.7-2",
        "v2.6.3",
        "v2.6.1",
        "v4.8.4",
        "v2.6.7",
        "v2.4.1-5",
        "v4.9.3-4-source",
        "v2.6.2-2",
        "v2.6.0",
        "v2.4.7-6",
        "v2.5.8",
        "v4.10.4",
        "v4.9.4",
        "v4.10.1-19",
        "v2.2.0-12",
        "v4.9.0-59",
        "v2.4.3",
        "v2.4.5-1",
        "v2.5.3",
        "v4.9.1-6",
        "v2.5.6",
        "v2.6.10-4",
        "2.4",
        "v2.4.6",
        "v4.10.5",
        "v4.10.5-1",
        "v4.10.5-1-source",
        "v4.8.7",
        "v4.9.6",
        "v4.9",
        "v4.9.6-3",
        "v4.9.7",
        "v4.9.7-4",
        "v4.10.7-2",
        "v4.8.7-4",
        "v4.11.1-10",
        "v4.11.1",
        "v4.11.0",
        "v4.10.7",
        "v4.10.6",
        "v4.11.0-96",
        "v4.9.6-3-source",
        "v4.9.7-4-source",
        "v4.10.6-4-source",
        "v4.11.1-10-source",
        "v4.10.6-4",
        "v4.10.7-2-source",
        "v4.8.7-4-source",
        "v4.11.0-96-source",
        "v4.12.2-2-source",
        "v4.12.0-96",
        "v4.10.8",
        "v4.12.1-5",
        "v4.12.1-5-source",
        "v4.11.2-2",
        "v4.12.0-96-source",
        "v4.11.2-2-source",
        "v4.10.8-2-source",
        "v4.11.2",
        "v4.10.8-2",
        "v4.12.1",
        "v4.11.3-2-source",
        "v4.11.3",
        "v4.11.3-2",
        "v4.12.2-2",
        "v4.12.2",
        "v4.12.0",
        "v4.11.4",
        "v4.11.4-7",
        "v4.12.3",
        "v4.12.3-4",
        "v4.12.3-4-source",
        "v4.11.4-7-source",
        "v4.10.9",
        "v4.10.9-4",
        "v4.10.9-4-source",
        "v4.12.4",
        "v4.12.4-7",
        "v4.12.4-7-source",
        "v4.10",
        "v4.10.10-3",
        "v4.10.10",
        "v4.10.10-3-source",
        "v4.11.5",
        "v4.11.5-4",
        "v4.11",
        "v4.11.5-4-source",
        "v4.12.5-7",
        "v4.12.5",
        "v4.12",
        "v4.12.5-7-source"
    ],
    "Created": "2023-04-26T07:48:07.2219238Z",
    "DockerVersion": "",
    "Labels": {
        "architecture": "x86_64",
        "build-date": "2023-04-26T07:42:53",
        "com.redhat.component": "hyperconverged-cluster-operator-container",
        "com.redhat.license_terms": "https://www.redhat.com/agreements",
        "description": "OpenShift Virtualization HyperConverged Cluster Operator",
        "distribution-scope": "public",
        "io.buildah.version": "1.27.3",
        "io.k8s.description": "OpenShift Virtualization HyperConverged Cluster Operator",
        "io.k8s.display-name": "hyperconverged-cluster-operator",
        "io.openshift.expose-services": "",
        "io.openshift.tags": "virtualization,kubevirt,operator",
        "maintainer": "Simone Tiraboschi \u003cstirabos\u003e",
        "name": "container-native-virtualization/hyperconverged-cluster-operator",
        "release": "4",
        "summary": "OpenShift Virtualization HyperConverged Cluster Operator",
        "upstream-vcs-ref": "dd1ccbbda5524959fa2bbf5e79f24b9841b79880",
        "upstream-vcs-type": "git",
        "upstream-version": "v1.6.0-30-gdd1ccbbd",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/container-native-virtualization/hyperconverged-cluster-operator/images/v4.10.9-4",
        "vcs-ref": "c0acb9bf1f6c40b1dbbe9a00b35d8ecc1f361268",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "v4.10.9"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:7d8746ab4ad8e51637d5a71d17e876a7432c1bb34ba3db89985a0cb9500cd64b",
        "sha256:159936233e356f70eb334eacd47f63f6abf7d21fd0b7d1582400b0e7a8ff250c"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "Digest": "sha256:7d8746ab4ad8e51637d5a71d17e876a7432c1bb34ba3db89985a0cb9500cd64b",
            "Size": 82259072,
            "Annotations": null
        },
        {
            "MIMEType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "Digest": "sha256:159936233e356f70eb334eacd47f63f6abf7d21fd0b7d1582400b0e7a8ff250c",
            "Size": 53649933,
            "Annotations": null
        }
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "container=oci",
        "USER_UID=1001",
        "USER_NAME=hyperconverged-cluster-operator",
        "KUBEVIRT_CLIENT_GO_SCHEME_REGISTRATION_VERSION=v1"
    ]
}