Description of problem: When using ImageStream name in DeploymentConfig as image value, the actual image in the Pod gets resolved to docker.io instead of internal registry. This results in ImagePull Error Version-Release number of selected component (if applicable): OCP 4.8 (reproduced on 4.8.5 and 4.8.7) How reproducible: always Steps to Reproduce: 1. Deploy an ImageStream pointing to en existing image (e.g. https://gist.github.com/vpavlin/c251bf6f4ad6aad7fb4eb9adcb98d9fc) 2. Deploy a DeploymentConfig pointing to the IS from #1 (e.g. https://gist.github.com/vpavlin/ef3c8e83ca1cfbe474921b7dc4cee069) 3. Resulting pod fails with ImagePull error Actual results: Pod tries to pull the image docker.io/library/$IMAGESTREAM_NAME Expected results: Pod pulls the image referenced by ImageStream from internal registry Additional info: This works fine in 4.7
As context, this is about using the behavior documented here to have a deployment+pod updated on admission, to replace the image field value with the value of a resolved imagestreamtag: https://docs.openshift.com/container-platform/4.8/openshift_images/using-imagestreams-with-kube-resources.html
*** Bug 2011215 has been marked as a duplicate of this bug. ***
The gist that supposed to have DeploymentConfig has only Pod. Do you have an example for DeploymentConfig?
We ran into this after an upgrade from 4.6 -> 4.7. It seems the imagestream is getting resolved to an image URL within the template, not at pod creation time. For example we have an app that uses a StatefulSet, whose template referenced "some-image-stream:latest". After the 4.7 upgrade pods are getting created with an image of "some-image-stream:latest", rather than whatever some-image-stream:latest currently refers to at pod creation time as it had under 4.6. The resolution is *not* repeated if the StatefulSet is reapplied. During StatefulSet creation "some-image-stream:latest" is resolved to a reference. But when reapplying the original definition the referenced image URL is replaced with "some-image-stream:latest". And since the lookup is no longer being done at pod creation time this leads to image pull errors (possibly, unless the name happens to exist in Docker Hub). In some cases `oc set triggers statefulset/example-statefulset --from-image=ubi:latest --containers example-statefulset` may get the job done. We used this approach for CronJobs that broke due to this change in behavior. In one case we had an operator-managed StatefulSet that would remove any custom annotations and hence the trigger. Fortunately we could configure the operator with an alternate image URL, such as image-registry.openshift-image-registry.svc:5000/cfk-test/some-image-stream:latest, so we were ultimately able to get all apps up and running after the upgrade. If users are broadly expected to replace image-lookup with annotation-based triggers, then image-lookup should be properly deprecated. Otherwise the previous behavior should be restored as having resolution work only at object creation time runs against expectations. Sample StatefulSet: apiVersion: apps/v1 kind: StatefulSet metadata: name: example-statefulset namespace: ref-test spec: serviceName: example-statefulset replicas: 3 selector: matchLabels: app: example-statefulset template: metadata: labels: app: example-statefulset spec: terminationGracePeriodSeconds: 10 containers: - name: example-statefulset image: 'ubi:latest' command: - /bin/sleep - '300' Import an image stream and set image-lookup: PS C:\Users\chet\Desktop\4.7> oc import-image registry.access.redhat.com/ubi8/ubi --confirm --scheduled --reference-policy=local imagestream.image.openshift.io/ubi imported Name: ubi Namespace: ref-test Created: Less than a second ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2021-10-13T18:46:13Z Image Repository: default-route-openshift-image-registry.apps.cluster.example.com/ref-test/ubi Image Lookup: local=false Unique Images: 1 Tags: 1 latest updates automatically from registry registry.access.redhat.com/ubi8/ubi prefer registry pullthrough when referencing this tag * registry.access.redhat.com/ubi8/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 Less than a second ago Image Name: ubi:latest Docker Image: registry.access.redhat.com/ubi8/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 Name: sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 Created: Less than a second ago Annotations: image.openshift.io/dockerLayersOrder=ascending Image Size: 83.38MB in 2 layers Layers: 83.37MB sha256:262268b65bd5f33784d6a61514964887bc18bc00c60c588bc62bfae7edca46f1 1.795kB sha256:06038631a24a25348b51d1bfc7d0a0ee555552a8998f8328f9b657d02dd4c64c Image Created: 4 weeks ago Author: <none> Arch: amd64 Command: /bin/bash Working Dir: <none> User: <none> Exposes Ports: <none> Docker Labels: architecture=x86_64 build-date=2021-09-14T16:37:54.438551 com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com com.redhat.component=ubi8-container com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly. distribution-scope=public io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly. io.k8s.display-name=Red Hat Universal Base Image 8 io.openshift.expose-services= io.openshift.tags=base rhel8 maintainer=Red Hat, Inc. name=ubi8 release=211 summary=Provides the latest release of Red Hat Universal Base Image 8. url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.4-211 vcs-ref=ed5adf70c28eb951940c72f4173fa32c4bca2165 vcs-type=git vendor=Red Hat, Inc. version=8.4 Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci PS C:\Users\chet\Desktop\4.7> oc set image-lookup ubi imagestream.image.openshift.io/ubi image lookup updated Create the initial statefulset and watch it scale up: PS C:\Users\chet\Desktop\4.7> oc apply -f example-statefulset.yaml statefulset.apps/example-statefulset created PS C:\Users\chet\Desktop\4.7> oc get po -w NAME READY STATUS RESTARTS AGE example-statefulset-0 1/1 Running 0 4s example-statefulset-1 0/1 ContainerCreating 0 1s example-statefulset-1 0/1 ContainerCreating 0 2s example-statefulset-1 1/1 Running 0 3s example-statefulset-2 0/1 Pending 0 0s example-statefulset-2 0/1 Pending 0 0s example-statefulset-2 0/1 Pending 0 0s example-statefulset-2 0/1 ContainerCreating 0 0s example-statefulset-2 0/1 ContainerCreating 0 2s example-statefulset-2 1/1 Running 0 3s PS C:\Users\chet\Desktop\4.7> oc get po NAME READY STATUS RESTARTS AGE example-statefulset-0 1/1 Running 0 15s example-statefulset-1 1/1 Running 0 12s example-statefulset-2 1/1 Running 0 9s Verifying that OCP had replaced 'ubi:latest' on the StatefulSet's template: PS C:\Users\chet\Desktop\4.7> oc get statefulset example-statefulset -o go-template='{{range $c := .spec.template.spec.containers}}{{$c.image}}{{\"\n\"}}{{end}}' image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 And that it reflects on the pods: PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}} - {{$c.image}}{{\"\n\"}}{{end}}{{end}}' example-statefulset-0: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-1: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-2: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 Now reapply the sample YAML and watch the rollout: PS C:\Users\chet\Desktop\4.7> oc apply -f example-statefulset.yaml statefulset.apps/example-statefulset configured PS C:\Users\chet\Desktop\4.7> oc get po -w NAME READY STATUS RESTARTS AGE example-statefulset-0 1/1 Running 0 38s example-statefulset-1 1/1 Running 0 35s example-statefulset-2 1/1 Terminating 0 32s example-statefulset-2 0/1 Terminating 0 41s example-statefulset-2 0/1 Terminating 0 53s example-statefulset-2 0/1 Terminating 0 53s example-statefulset-2 0/1 Pending 0 0s example-statefulset-2 0/1 Pending 0 0s example-statefulset-2 0/1 Pending 0 0s example-statefulset-2 0/1 ContainerCreating 0 0s example-statefulset-2 0/1 ContainerCreating 0 2s example-statefulset-2 0/1 ErrImagePull 0 5s example-statefulset-2 0/1 ImagePullBackOff 0 19s We'd applied the same YAML twice, so breakage isn't expected. Checking the image in the pod template, it's no longer being dereferenced: PS C:\Users\chet\Desktop\4.7> oc get statefulset example-statefulset -o go-template='{{range $c := .spec.template.spec.containers}}{{$c.image}}{{\"\n\"}}{{end}}' ubi:latest And the new pod shows that it's no longer being derefenced at pod creation time: PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}} - {{$c.image}}{{\"\n\"}}{{end}}{{end}}' example-statefulset-0: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-1: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-2: - ubi:latest As mentioned, setting a trigger is a workaround in cases where the object can be annotated: PS C:\Users\chet\Desktop\4.7> oc set triggers statefulset/example-statefulset --from-image=ubi:latest --containers example-statefulset statefulset.apps/example-statefulset triggers updated Checking the results: PS C:\Users\chet\Desktop\4.7> oc get statefulset example-statefulset -o go-template='{{range $c := .spec.template.spec.containers}}{{$c.image}}{{\"\n\"}}{{end}}' image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 The template is now updated as expected. Let's check pods: PS C:\Users\chet\Desktop\4.7> oc get po -w NAME READY STATUS RESTARTS AGE example-statefulset-0 1/1 Running 2 13m example-statefulset-1 1/1 Running 2 13m example-statefulset-2 0/1 ImagePullBackOff 0 12m No change. I'm not sure whether my expectations are reasonable, but the second `oc apply` did cause an immediate rollout. I'd have expected the template change to have caused one, but don't have the experience with StatefulSets to know if there's another explanation. PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}} - {{$c.image}}{{\"\n\"}}{{end}}{{end}}' example-statefulset-0: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-1: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-2: - ubi:latest Sometimes you just have to delete the pod. PS C:\Users\chet\Desktop\4.7> oc delete po example-statefulset-2 pod "example-statefulset-2" deleted And now we're good with the workaround: PS C:\Users\chet\Desktop\4.7> oc get po -w NAME READY STATUS RESTARTS AGE example-statefulset-0 1/1 Running 2 14m example-statefulset-1 1/1 Running 2 14m example-statefulset-2 1/1 Running 0 5s PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}} - {{$c.image}}{{\"\n\"}}{{end}}{{end}}' example-statefulset-0: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-1: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1 example-statefulset-2: - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
We're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the UpgradeBlocker flag has been added to this bug. It will be removed if the assessment indicates that this should not block upgrade edges. The expectation is that the assignee answers these questions. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time What is the impact? Is it serious enough to warrant blocking edges? example: Up to 2 minute disruption in edge routing example: Up to 90seconds of API downtime example: etcd loses quorum and you have to restore from backup How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? example: Issue resolves itself after five minutes example: Admin uses oc to fix things example: Admin must SSH to hosts, restore from backups, or other non standard admin activities Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? example: No, it’s always been like this we just never noticed example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1
Who is impacted? Customers who use spec.lookupPolicy.local=true on image streams or the annotation alpha.image.policy.openshift.io/resolve-names, and who received the fix for BZ 1925180, i.e. who use 4.6.45+, 4.7.28+, 4.8.4+, or 4.9.0+. What is the impact? The image stream names are no longer resolved into image references in pods that are created by deployment configs, in some cases in statefulsets, and possibly in other objects. How involved is remediation? Direct image references should be used instead of image stream names, i.e. instead of IMAGESTREAMNAME:TAG the reference image-registry.openshift-image-registry.svc:5000/NAMESPACE/IMAGESTREAMNAME:TAG should be used. Is this a regression? Yes, older versions don't have this problem.
Customers facing this issue can use the workaround for this issue which has been published as a Red Hat Knowledge Base article https://access.redhat.com/solutions/6455191
After the impact analysis we do not think this is a upgrade blocker and decided not to block the edges for this bug. Also we have an easy workaround #17 for this issue.
I can reproduce the issue in 4.8.5 Steps to reproduce: 1. Create is using following content: ``` kind: ImageStream apiVersion: image.openshift.io/v1 metadata: name: vpavlin-test spec: lookupPolicy: local: true tags: - name: latest annotations: null from: kind: DockerImage name: 'quay.io/centos/centos:latest' importPolicy: {} referencePolicy: type: Source ``` 2. Create pods using about is ``` kind: Pod apiVersion: v1 metadata: name: "test" namespace: "vpavlin-test" labels: ocp551: ocp551 annotations: "descheduler.alpha.kubernetes.io/evict": "" spec: containers: - name: pause image: bug-test:latest ``` 3. Check the pod [root@localhost ~]# oc get pods NAME READY STATUS RESTARTS AGE test 0/1 ImagePullBackOff 0 3m8s ``` state: waiting: message: | rpc error: code = Unknown desc = Error reading manifest latest in docker.io/library/bug-test: errors: denied: requested access to the resource is denied unauthorized: authentication required reason: ErrImagePull ```
Verified it, when using image stream are now point to real image reference now in dc or statefulset Version: 4.10.0-0.ci.test-2021-11-03-015419-ci-ln-qh8d50b-latest Steps: 1. Create a deployment config as Comments 23, imagestream point to real image: quay.io/centos/centos 2. Create stateful set a. Import an image stream and set image-lookup $oc import-image registry.access.redhat.com/ubi8/ubi --confirm --scheduled --reference-policy=local $oc set image-lookup ubi b. Create the initial statefulset ``` apiVersion: apps/v1 kind: StatefulSet metadata: name: example-statefulset namespace: ref-test spec: serviceName: example-statefulset replicas: 3 selector: matchLabels: app: example-statefulset template: metadata: labels: app: example-statefulset spec: terminationGracePeriodSeconds: 10 containers: - name: example-statefulset image: 'ubi:latest' command: - /bin/sleep - '300' ``` c. Check the pod: [root@localhost ~]# oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{"\n"}} - {{$c.image}}{{"\n"}}{{end}}{{end}}' example-statefulset-0: - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0 example-statefulset-1: - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0 example-statefulset-2: - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0 d. Reapply the sample YAML,after restart, pods are running [root@localhost ~]# oc apply -f example-statefulset.yaml statefulset.apps/example-statefulset configured [root@localhost ~]# oc get pods NAME READY STATUS RESTARTS AGE example-statefulset-0 1/1 Running 2 (39s ago) 10m example-statefulset-1 1/1 Running 2 (31s ago) 10m example-statefulset-2 1/1 Running 2 (22s ago) 10m [root@localhost ~]# oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{"\n"}} - {{$c.image}}{{"\n"}}{{end}}{{end}}' example-statefulset-0: - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0 example-statefulset-1: - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0 example-statefulset-2: - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0 e. setting a trigger, pods are still running [root@localhost ~]# oc set triggers statefulset/example-statefulset --from-image=ubi:latest --containers example-statefulset statefulset.apps/example-statefulset triggers updated [root@localhost ~]# oc get pods NAME READY STATUS RESTARTS AGE example-statefulset-0 1/1 Running 4 (2m9s ago) 23m example-statefulset-1 1/1 Running 4 (2m1s ago) 22m example-statefulset-2 1/1 Running 4 (106s ago) 22m
There is no "easy workaround". It's not reasonable to expect people to make changes to the deployment configs of dozens of projects because of a bug that has been introduced into OpenShift. When can we expect a fix to be released please?
Walk the 'Blocks' chain to find out about backports. This 4.10 bug blocks the 4.9.z bug 2018442, which blocks the 4.8.z bug 2020644, which blocks the 4.7.z bug 2022188. So subscribe to bug 2022188 to hear about 4.7 progress and get a message when it ships.
Further hint (which I just discovered): clicking "Show advanced fields" will display both "Blocks" and "Target Release". "Version", which is shown by default, is the version in which the bug was found.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days