Bug 2075346

+++ This bug was initially created as a clone of Bug #2060532 +++

In bug 2030776 I updated openshift/release to build and deploy a CI image of LSO. I can see from the CI in https://github.com/openshift/local-storage-operator/pull/327 that this image does get built and deployed:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_local-storage-operator/327/pull-ci-openshift-local-storage-operator-master-e2e-operator/1498661877167689728/artifacts/e2e-operator/gather-extra/artifacts/pods.json

                "name": "local-storage-operator-778b5c876f-hck6h",
                "namespace": "openshift-local-storage",
                "ownerReferences": [
                    {
                        "apiVersion": "apps/v1",
                        "blockOwnerDeletion": true,
                        "controller": true,
                        "kind": "ReplicaSet",
                        "name": "local-storage-operator-778b5c876f",
                        "uid": "f399c537-b8c7-4b4f-9528-3dacbfbfde53"
                    }
                ],
                "resourceVersion": "29968",
                "uid": "118bac8a-a9a0-4cd4-8e9f-aea58ac7891e"
            },
            "spec": {
                "containers": [
                    {
                        "args": [
                            "--leader-elect"
                        ],
                        "command": [
                            "local-storage-operator"
                        ],
                        "env": [
                            {
                                "name": "WATCH_NAMESPACE",
                                "valueFrom": {
                                    "fieldRef": {
                                        "apiVersion": "v1",
                                        "fieldPath": "metadata.namespace"
                                    }
                                }
                            },
                            {
                                "name": "DISKMAKER_IMAGE",
                                "value": "registry.build01.ci.openshift.org/ci-op-0x7p6lit/pipeline@sha256:d2707d124a6a714a9a7973c199b8229ae3d07988f9970c8b9910918b3f078fa2"
                            },
                            {
                                "name": "KUBE_RBAC_PROXY_IMAGE",
                                "value": "quay.io/openshift/origin-kube-rbac-proxy:latest"
                            },
                            {
                                "name": "PRIORITY_CLASS_NAME",
                                "value": "openshift-user-critical"
                            },
                            {
                                "name": "OPERATOR_CONDITION_NAME",
                                "value": "local-storage-operator.v4.11.0"
                            }
                        ],
                        "image": "registry.build01.ci.openshift.org/ci-op-0x7p6lit/pipeline@sha256:a0d36c7382267e0b2789066af0fa4de792bbcf64a09f1374a22f62e19889a364",
                        "imagePullPolicy": "IfNotPresent",
                        "name": "local-storage-operator",


However, there is something else going on when running hack/test-e2e.sh as part of e2e-operator. Wei pointed out some events like this, showing the default image (and in the default namespace) still:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_local-storage-operator/327/pull-ci-openshift-local-storage-operator-master-e2e-operator/1498661877167689728/artifacts/e2e-operator/gather-extra/artifacts/events.json

        {
            "apiVersion": "v1",
            "count": 1,
            "eventTime": null,
            "firstTimestamp": "2022-03-01T15:05:29Z",
            "involvedObject": {
                "apiVersion": "v1",
                "fieldPath": "spec.containers{local-storage-operator}",
                "kind": "Pod",
                "name": "local-storage-operator-577fddbf77-69792",
                "namespace": "default",
                "resourceVersion": "31181",
                "uid": "2b52eecb-a9e0-46a8-8c8e-15e8dfade6e2"
            },
            "kind": "Event",
            "lastTimestamp": "2022-03-01T15:05:29Z",
            "message": "Pulling image \"quay.io/openshift/origin-local-storage-operator\"",
            "metadata": {
                "creationTimestamp": "2022-03-01T15:05:29Z",
                "name": "local-storage-operator-577fddbf77-69792.16d84a17c51ae7ed",
                "namespace": "default",
                "resourceVersion": "31204",
                "uid": "b894832a-05b0-4a95-89ac-1e2bef4a61b6"
            },
            "reason": "Pulling",
            "reportingComponent": "",
            "reportingInstance": "",
            "source": {
                "component": "kubelet",
                "host": "ip-10-0-203-120.us-west-2.compute.internal"
            },
            "type": "Normal"
        },


And from the test logs I see that test-e2e.sh is actually using a separate manifest.yaml that still deploys the default image to the default namespace:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_local-storage-operator/327/pull-ci-openshift-local-storage-operator-master-e2e-operator/1498661877167689728/artifacts/e2e-operator/test/artifacts/manifest.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: local-storage-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: local-storage-operator
  template:
    metadata:
      annotations:
        target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
      labels:
        name: local-storage-operator
    spec:
      serviceAccountName: local-storage-operator
      containers:
        - name: local-storage-operator
          image: quay.io/openshift/origin-local-storage-operator
          ports:
          - containerPort: 60000
            name: metrics
          command:
          - local-storage-operator
          args:
          - --leader-elect
          imagePullPolicy: IfNotPresent
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: DISKMAKER_IMAGE
              value: quay.io/openshift/origin-local-storage-diskmaker
            - name: KUBE_RBAC_PROXY_IMAGE
              value: quay.io/openshift/origin-kube-rbac-proxy:latest


The substitutions here update the manifest that gets deployed by bundle.Dockerfile:

https://github.com/openshift/release/blob/c52cc269b7e7657725266bf60ca17fe974caa267/ci-operator/config/openshift/local-storage-operator/openshift-local-storage-operator-master.yaml#L48-L51

But it does not affect the images in test-e2e.sh which apparently builds a separate manifest to test:

https://github.com/openshift/local-storage-operator/blob/a9d477a312fa3a7411366f8d025dec14b0aee8ae/hack/test-e2e.sh#L36-L40

We need to fix this so that the CI image is what actually gets tested by test-e2e.sh.