Description of problem (please be detailed as possible and provide log snippests): Livnes and readiness probes are failing on redhat-operators pod hence can't pull any image from the manifest: openshift-marketplace 10m Normal Killing pod/redhat-operators-j8qw5 Stopping container registry-server openshift-marketplace 9m55s Normal Scheduled pod/redhat-operators-hnmps Successfully assigned openshift-marketplace/redhat-operators-hnmps to compute-2 by control-plane-2 openshift-marketplace 9m4s Normal Pulling pod/redhat-operators-hnmps Pulling image "quay.io/rhceph-dev/ocs-registry:4.11.0-115" openshift-marketplace 9m54s Normal AddedInterface pod/redhat-operators-hnmps Add eth0 [10.129.2.214/23] from openshift-sdn openshift-marketplace 9m2s Normal Created pod/redhat-operators-hnmps Created container registry-server openshift-marketplace 9m2s Normal Started pod/redhat-operators-hnmps Started container registry-server openshift-marketplace 9m39s Normal Pulled pod/redhat-operators-hnmps Successfully pulled image "quay.io/rhceph-dev/ocs-registry:4.11.0-115" in 14.4209307s openshift-marketplace 8m44s Warning Unhealthy pod/redhat-operators-hnmps Liveness probe failed: timeout: failed to connect service ":50051" within 1s openshift-marketplace 8m44s Warning Unhealthy pod/redhat-operators-hnmps Readiness probe failed: timeout: failed to connect service ":50051" within 1s openshift-marketplace 9m4s Warning Unhealthy pod/redhat-operators-hnmps Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of fba769d81c3091270ed847a5ad65ef538ded93240689bade0ca821ffb3315238 is running failed: open /proc/3634056/stat: no such file or directory: container process not found openshift-marketplace 9m4s Normal Killing pod/redhat-operators-hnmps Container registry-server failed liveness probe, will be restarted Maybe this is the reason: openshift-marketplace 11m Warning Unhealthy pod/redhat-operators-hnmps Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of fba769d81c3091270ed847a5ad65ef538ded93240689bade0ca821ffb3315238 is running failed: open /proc/3634056/stat: no such file or directory: container process not found Version of all relevant components (if applicable): odf-4.11.0-115 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Can't install odf +lvm Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Yes, on 4.11.0-113 both clusters were pulling image. Steps to Reproduce: 1. Install catalogsource 4.11.0-115 2. Try to deploy LVM or ODF-OPERATOR Actual results: Images are not pulled with openshift-marketplace 6m25s Warning Failed pod/5953a0ca15d876bcdec70414f9fa1504c4175b964da1afbc7baaae21f3n4qgn Failed to pull image "quay.io/rhceph-dev/odf4-odf-operator-bundle@sha256:4b7a966329d13af8b1ed5813972c9167c36febc4c05caff1597d9679481f3618": rpc error: code = Unknown desc = reading manifest sha256:4b7a966329d13af8b1ed5813972c9167c36febc4c05caff1597d9679481f3618 in quay.io/rhceph-dev/odf4-odf-operator-bundle: manifest unknown: manifest unknown openshift-marketplace 6m41s Warning Failed pod/8c2d59d1001957885e96e61744a4544a24d7c7debf28491e586b83af74x9k2b Failed to pull image "quay.io/rhceph-dev/odf4-mcg-operator-bundle@sha256:56c13849ffeb234f12ea650c3c20505a935808a03cbb8c0050e981ddf26d4554": rpc error: code = Unknown desc = reading manifest sha256:56c13849ffeb234f12ea650c3c20505a935808a03cbb8c0050e981ddf26d4554 in quay.io/rhceph-dev/odf4-mcg-operator-bundle: manifest unknown: manifest unknown Expected results: Image should be pulled succesfully. Additional info:
It is a problem with some compose bundle that was shared between two builds which re-run the mirroring pipeline. Next build should be ok