Description of the problem: In Telco far edge use case, a cluster might have limited management bandwidth. Upgrading such a cluster is expected to be time limited (capped by a maintenance window), and will involve container image pre-caching on the node. Image pre-caching solution for nodes with limited management bandwidth relies on the assumption, that workloads can use locally stored images without contacting a registry. This is possible if container image pull policy is set to "IfNotPresent". The performance-addon-rhel8-operator image pull policy is set to "Always"
Verification: Version: ocp: 4.10.0-rc.1 pao: registry-proxy.engineering.redhat.com/rh-osbs/openshift4-performance-addon-rhel8-operator@sha256:607b829f1ac58e2851d6188ccc2acdccb5f4e9ef4d092a2851fff05f20d74017 corresponding to v4.10.0-32 Steps: -Install pao using PPC & inspect into the pod's replicaset: value: performance-addon-operator.v4.10.0 image: registry.redhat.io/openshift4/performance-addon-rhel8-operator@sha256:607b829f1ac58e2851d6188ccc2acdccb5f4e9ef4d092a2851fff05f20d74017 imagePullPolicy: IfNotPresent [1] -Reboot the node on which the pod currently running. -Check the pod description/events: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned openshift-performance-addon-operator/performance-operator-6bfbf5dd54-6bhml to ocp410-master-2.demo.lab.mniranja Normal AddedInterface 15m multus Add eth0 [10.132.0.59/23] from openshift-sdn Normal Pulling 15m kubelet Pulling image "registry.redhat.io/openshift4/performance-addon-rhel8-operator@sha256:607b829f1ac58e2851d6188ccc2acdccb5f4e9ef4d092a2851fff05f20d74017" Normal Pulled 14m kubelet Successfully pulled image "registry.redhat.io/openshift4/performance-addon-rhel8-operator@sha256:607b829f1ac58e2851d6188ccc2acdccb5f4e9ef4d092a2851fff05f20d74017" in 32.747670506s Normal Created 14m kubelet Created container performance-operator Normal Started 14m kubelet Started container performance-operator Warning NodeNotReady 3m47s node-controller Node is not ready [2] Normal AddedInterface 2m27s multus Add eth0 [10.132.0.19/23] from openshift-sdn Normal Pulled 2m27s kubelet Container image "registry.redhat.io/openshift4/performance-addon-rhel8-operator@sha256:607b829f1ac58e2851d6188ccc2acdccb5f4e9ef4d092a2851fff05f20d74017" already present on machine [3] Normal Created 2m26s kubelet Created container performance-operator Normal Started 2m25s kubelet Started container performance-operator [1] the pull policy is now IfNotPresent. [2] the node is rebooting [3] after the machine is up again it trie to rerun the pod & pull the image, since the image already exists on that node, it uses it & does no additional pulling. As a negative check, you may also delete the pull secret before restarting the node. deleting the pull secret: oc delete secrets/pull-secret -n openshift-config Verified successfully.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.10 low-latency extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:0640