As context, this is about using the behavior documented here to have a deployment+pod updated on admission, to replace the image field value with the value of a resolved imagestreamtag:

https://docs.openshift.com/container-platform/4.8/openshift_images/using-imagestreams-with-kube-resources.html

*** Bug 2011215 has been marked as a duplicate of this bug. ***

The gist that supposed to have DeploymentConfig has only Pod. Do you have an example for DeploymentConfig?

We ran into this after an upgrade from 4.6 -> 4.7. It seems the imagestream is getting resolved to an image URL within the template, not at pod creation time.

For example we have an app that uses a StatefulSet, whose template referenced "some-image-stream:latest". After the 4.7 upgrade pods are getting created with an image of "some-image-stream:latest", rather than whatever some-image-stream:latest currently refers to at pod creation time as it had under 4.6.

The resolution is *not* repeated if the StatefulSet is reapplied. During StatefulSet creation "some-image-stream:latest" is resolved to a reference. But when reapplying the original definition the referenced image URL is replaced with "some-image-stream:latest". And since the lookup is no longer being done at pod creation time this leads to image pull errors (possibly, unless the name happens to exist in Docker Hub).

In some cases `oc set triggers statefulset/example-statefulset --from-image=ubi:latest --containers example-statefulset` may get the job done. We used this approach for CronJobs that broke due to this change in behavior. In one case we had an operator-managed StatefulSet that would remove any custom annotations and hence the trigger. Fortunately we could configure the operator with an alternate image URL, such as image-registry.openshift-image-registry.svc:5000/cfk-test/some-image-stream:latest, so we were ultimately able to get all apps up and running after the upgrade.

If users are broadly expected to replace image-lookup with annotation-based triggers, then image-lookup should be properly deprecated. Otherwise the previous behavior should be restored as having resolution work only at object creation time runs against expectations.

Sample StatefulSet:

  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: example-statefulset
    namespace: ref-test
  spec:
    serviceName: example-statefulset
    replicas: 3
    selector:
      matchLabels:
        app: example-statefulset
    template:
      metadata:
        labels:
          app: example-statefulset
      spec:
        terminationGracePeriodSeconds: 10
        containers:
          - name: example-statefulset
            image: 'ubi:latest'
            command:
              - /bin/sleep
              - '300'

Import an image stream and set image-lookup:

  PS C:\Users\chet\Desktop\4.7> oc import-image registry.access.redhat.com/ubi8/ubi --confirm --scheduled --reference-policy=local
  imagestream.image.openshift.io/ubi imported
  
  Name:                   ubi
  Namespace:              ref-test
  Created:                Less than a second ago
  Labels:                 <none>
  Annotations:            openshift.io/image.dockerRepositoryCheck=2021-10-13T18:46:13Z
  Image Repository:       default-route-openshift-image-registry.apps.cluster.example.com/ref-test/ubi
  Image Lookup:           local=false
  Unique Images:          1
  Tags:                   1
  
  latest
    updates automatically from registry registry.access.redhat.com/ubi8/ubi
      prefer registry pullthrough when referencing this tag
  
    * registry.access.redhat.com/ubi8/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
        Less than a second ago
  
  Image Name:     ubi:latest
  Docker Image:   registry.access.redhat.com/ubi8/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  Name:           sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  Created:        Less than a second ago
  Annotations:    image.openshift.io/dockerLayersOrder=ascending
  Image Size:     83.38MB in 2 layers
  Layers:         83.37MB sha256:262268b65bd5f33784d6a61514964887bc18bc00c60c588bc62bfae7edca46f1
                  1.795kB sha256:06038631a24a25348b51d1bfc7d0a0ee555552a8998f8328f9b657d02dd4c64c
  Image Created:  4 weeks ago
  Author:         <none>
  Arch:           amd64
  Command:        /bin/bash
  Working Dir:    <none>
  User:           <none>
  Exposes Ports:  <none>
  Docker Labels:  architecture=x86_64
                  build-date=2021-09-14T16:37:54.438551
                  com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com
                  com.redhat.component=ubi8-container
                  com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI
                  description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.
                  distribution-scope=public
                  io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.
                  io.k8s.display-name=Red Hat Universal Base Image 8
                  io.openshift.expose-services=
                  io.openshift.tags=base rhel8
                  maintainer=Red Hat, Inc.
                  name=ubi8
                  release=211
                  summary=Provides the latest release of Red Hat Universal Base Image 8.
                  url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.4-211
                  vcs-ref=ed5adf70c28eb951940c72f4173fa32c4bca2165
                  vcs-type=git
                  vendor=Red Hat, Inc.
                  version=8.4
  Environment:    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
                  container=oci
  
  
  PS C:\Users\chet\Desktop\4.7> oc set image-lookup ubi
  imagestream.image.openshift.io/ubi image lookup updated

Create the initial statefulset and watch it scale up:

  PS C:\Users\chet\Desktop\4.7> oc apply -f example-statefulset.yaml
  statefulset.apps/example-statefulset created
  PS C:\Users\chet\Desktop\4.7> oc get po -w
  NAME                    READY   STATUS              RESTARTS   AGE
  example-statefulset-0   1/1     Running             0          4s
  example-statefulset-1   0/1     ContainerCreating   0          1s
  example-statefulset-1   0/1     ContainerCreating   0          2s
  example-statefulset-1   1/1     Running             0          3s
  example-statefulset-2   0/1     Pending             0          0s
  example-statefulset-2   0/1     Pending             0          0s
  example-statefulset-2   0/1     Pending             0          0s
  example-statefulset-2   0/1     ContainerCreating   0          0s
  example-statefulset-2   0/1     ContainerCreating   0          2s
  example-statefulset-2   1/1     Running             0          3s
  PS C:\Users\chet\Desktop\4.7> oc get po
  NAME                    READY   STATUS    RESTARTS   AGE
  example-statefulset-0   1/1     Running   0          15s
  example-statefulset-1   1/1     Running   0          12s
  example-statefulset-2   1/1     Running   0          9s

Verifying that OCP had replaced 'ubi:latest' on the StatefulSet's template:

  PS C:\Users\chet\Desktop\4.7> oc get statefulset example-statefulset -o go-template='{{range $c := .spec.template.spec.containers}}{{$c.image}}{{\"\n\"}}{{end}}'
  image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1

And that it reflects on the pods:

  PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}}  - {{$c.image}}{{\"\n\"}}{{end}}{{end}}'
  example-statefulset-0:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-1:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-2:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1

Now reapply the sample YAML and watch the rollout:

  PS C:\Users\chet\Desktop\4.7> oc apply -f example-statefulset.yaml
  statefulset.apps/example-statefulset configured
  
  PS C:\Users\chet\Desktop\4.7> oc get po -w
  NAME                    READY   STATUS        RESTARTS   AGE
  example-statefulset-0   1/1     Running       0          38s
  example-statefulset-1   1/1     Running       0          35s
  example-statefulset-2   1/1     Terminating   0          32s
  example-statefulset-2   0/1     Terminating   0          41s
  example-statefulset-2   0/1     Terminating   0          53s
  example-statefulset-2   0/1     Terminating   0          53s
  example-statefulset-2   0/1     Pending       0          0s
  example-statefulset-2   0/1     Pending       0          0s
  example-statefulset-2   0/1     Pending       0          0s
  example-statefulset-2   0/1     ContainerCreating   0          0s
  example-statefulset-2   0/1     ContainerCreating   0          2s
  example-statefulset-2   0/1     ErrImagePull        0          5s
  example-statefulset-2   0/1     ImagePullBackOff    0          19s

We'd applied the same YAML twice, so breakage isn't expected. Checking the image in the pod template, it's no longer being dereferenced:

  PS C:\Users\chet\Desktop\4.7> oc get statefulset example-statefulset -o go-template='{{range $c := .spec.template.spec.containers}}{{$c.image}}{{\"\n\"}}{{end}}'
  ubi:latest

And the new pod shows that it's no longer being derefenced at pod creation time:

  PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}}  - {{$c.image}}{{\"\n\"}}{{end}}{{end}}'
  example-statefulset-0:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-1:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-2:
    - ubi:latest

As mentioned, setting a trigger is a workaround in cases where the object can be annotated:

  PS C:\Users\chet\Desktop\4.7> oc set triggers statefulset/example-statefulset --from-image=ubi:latest --containers  example-statefulset
  statefulset.apps/example-statefulset triggers updated

Checking the results:

  PS C:\Users\chet\Desktop\4.7> oc get statefulset example-statefulset -o go-template='{{range $c := .spec.template.spec.containers}}{{$c.image}}{{\"\n\"}}{{end}}'
  image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1

The template is now updated as expected. Let's check pods:

  PS C:\Users\chet\Desktop\4.7> oc get po -w
  NAME                    READY   STATUS             RESTARTS   AGE
  example-statefulset-0   1/1     Running            2          13m
  example-statefulset-1   1/1     Running            2          13m
  example-statefulset-2   0/1     ImagePullBackOff   0          12m

No change. I'm not sure whether my expectations are reasonable, but the second `oc apply` did cause an immediate rollout. I'd have expected the template change to have caused one, but don't have the experience with StatefulSets to know if there's another explanation.

  PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}}  - {{$c.image}}{{\"\n\"}}{{end}}{{end}}'
  example-statefulset-0:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-1:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-2:
    - ubi:latest

Sometimes you just have to delete the pod.

  PS C:\Users\chet\Desktop\4.7> oc delete po example-statefulset-2
  pod "example-statefulset-2" deleted

And now we're good with the workaround:

  PS C:\Users\chet\Desktop\4.7> oc get po -w
  NAME                    READY   STATUS    RESTARTS   AGE
  example-statefulset-0   1/1     Running   2          14m
  example-statefulset-1   1/1     Running   2          14m
  example-statefulset-2   1/1     Running   0          5s
  PS C:\Users\chet\Desktop\4.7> oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{\"\n\"}}  - {{$c.image}}{{\"\n\"}}{{end}}{{end}}'
  example-statefulset-0:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-1:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1
  example-statefulset-2:
    - image-registry.openshift-image-registry.svc:5000/ref-test/ubi@sha256:5e334d76fc059f7b44ee8fc2da6a2e8b240582d0214364c8c88596d20b33d7f1

We're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the UpgradeBlocker flag has been added to this bug. It will be removed if the assessment indicates that this should not block upgrade edges. The expectation is that the assignee answers these questions.

Who is impacted?  If we have to block upgrade edges based on this issue, which edges would need blocking?
  example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet
  example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time
What is the impact?  Is it serious enough to warrant blocking edges?
  example: Up to 2 minute disruption in edge routing
  example: Up to 90seconds of API downtime
  example: etcd loses quorum and you have to restore from backup
How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)?
  example: Issue resolves itself after five minutes
  example: Admin uses oc to fix things
  example: Admin must SSH to hosts, restore from backups, or other non standard admin activities
Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)?
  example: No, it’s always been like this we just never noticed
  example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1

Who is impacted?

Customers who use spec.lookupPolicy.local=true on image streams or the annotation alpha.image.policy.openshift.io/resolve-names, and who received the fix for BZ 1925180, i.e. who use 4.6.45+, 4.7.28+, 4.8.4+, or 4.9.0+.

What is the impact?

The image stream names are no longer resolved into image references in pods that are created by deployment configs, in some cases in statefulsets, and possibly in other objects.

How involved is remediation?

Direct image references should be used instead of image stream names, i.e. instead of IMAGESTREAMNAME:TAG the reference image-registry.openshift-image-registry.svc:5000/NAMESPACE/IMAGESTREAMNAME:TAG should be used.

Is this a regression?

Yes, older versions don't have this problem.

Customers facing this issue can use the workaround for this issue which has been published as a Red Hat Knowledge Base article https://access.redhat.com/solutions/6455191

After the impact analysis we do not think this is a upgrade blocker and decided not to block the edges for this bug. Also we have an easy workaround #17 for this issue.

I can reproduce the issue in 4.8.5

Steps to reproduce:
1. Create is using following content:
```
kind: ImageStream
apiVersion: image.openshift.io/v1
metadata:
  name: vpavlin-test
spec:
  lookupPolicy:
    local: true
  tags:
    - name: latest
      annotations: null
      from:
        kind: DockerImage
        name: 'quay.io/centos/centos:latest'
      importPolicy: {}
      referencePolicy:
        type: Source
```
2. Create pods using about is
```
kind: Pod
apiVersion: v1
metadata:
  name: "test"
  namespace: "vpavlin-test"
  labels:
    ocp551: ocp551
  annotations:
    "descheduler.alpha.kubernetes.io/evict": ""
spec:
  containers:
  - name: pause
    image: bug-test:latest
```
3. Check the pod
[root@localhost ~]# oc get pods
NAME   READY   STATUS             RESTARTS   AGE
test   0/1     ImagePullBackOff   0          3m8s
```
    state:
      waiting:
        message: |
          rpc error: code = Unknown desc = Error reading manifest latest in docker.io/library/bug-test: errors:
          denied: requested access to the resource is denied
          unauthorized: authentication required
        reason: ErrImagePull
```

Verified it, when using image stream  are now point to real image reference now in dc or statefulset
Version:
4.10.0-0.ci.test-2021-11-03-015419-ci-ln-qh8d50b-latest

Steps:
1. Create a deployment config as Comments 23, imagestream point to real image: quay.io/centos/centos  

2. Create stateful set
   a. Import an image stream and set image-lookup
      $oc import-image registry.access.redhat.com/ubi8/ubi --confirm --scheduled --reference-policy=local
      $oc set image-lookup ubi
   b. Create the initial statefulset
    ```
      apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: example-statefulset
    namespace: ref-test
  spec:
    serviceName: example-statefulset
    replicas: 3
    selector:
      matchLabels:
        app: example-statefulset
    template:
      metadata:
        labels:
          app: example-statefulset
      spec:
        terminationGracePeriodSeconds: 10
        containers:
          - name: example-statefulset
            image: 'ubi:latest'
            command:
              - /bin/sleep
              - '300'
     ```
    c. Check the pod: 
[root@localhost ~]#  oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{"\n"}}  - {{$c.image}}{{"\n"}}{{end}}{{end}}'
example-statefulset-0:
  - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0
example-statefulset-1:
  - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0
example-statefulset-2:
  - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0
   
    d. Reapply the sample YAML,after restart, pods are running
     [root@localhost ~]#  oc apply -f example-statefulset.yaml
     statefulset.apps/example-statefulset configured
     [root@localhost ~]# oc get pods
NAME                    READY   STATUS    RESTARTS      AGE
example-statefulset-0   1/1     Running   2 (39s ago)   10m
example-statefulset-1   1/1     Running   2 (31s ago)   10m
example-statefulset-2   1/1     Running   2 (22s ago)   10m
[root@localhost ~]#  oc get po -o go-template='{{range $p := .items}}{{$p.metadata.name}}:{{range $c := $p.spec.containers}}{{"\n"}}  - {{$c.image}}{{"\n"}}{{end}}{{end}}'
example-statefulset-0:
  - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0
example-statefulset-1:
  - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0
example-statefulset-2:
  - image-registry.openshift-image-registry.svc:5000/wewang/ubi@sha256:53d3a0837cf04ba4c687ddb39b487dc86efdbc14bd528778a7fa522637b1a3a0

   e.  setting a trigger, pods are still running
   [root@localhost ~]# oc set triggers statefulset/example-statefulset --from-image=ubi:latest --containers  example-statefulset
statefulset.apps/example-statefulset triggers updated
[root@localhost ~]# oc get pods
NAME                    READY   STATUS    RESTARTS       AGE
example-statefulset-0   1/1     Running   4 (2m9s ago)   23m
example-statefulset-1   1/1     Running   4 (2m1s ago)   22m
example-statefulset-2   1/1     Running   4 (106s ago)   22m

There is no "easy workaround". It's not reasonable to expect people to make changes to the deployment configs of dozens of projects because of a bug that has been introduced into OpenShift. When can we expect a fix to be released please?

Walk the 'Blocks' chain to find out about backports.  This 4.10 bug blocks the 4.9.z bug 2018442, which blocks the 4.8.z bug 2020644, which blocks the 4.7.z bug 2022188.  So subscribe to bug 2022188 to hear about 4.7 progress and get a message when it ships.

Further hint (which I just discovered): clicking "Show advanced fields" will display both "Blocks" and "Target Release".

"Version", which is shown by default, is the version in which the bug was found.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days