Bug 1949294

Moving this to the OLM component as this has to do with Subscriptions -> operator deployments. This mechanism is required in order for users to communicate the location of rhcos live isos and essential for disconnected flows.

Adding @asegurap to evaluate if this is a blocker or not. My assessment: yes. but I would not be surprised if there was additional criteria I'm not aware of.

It looks like this is a bug in the way OLM overrides existing environment variables vs adding new environment variables.

A workaround is to always list the overrides before the new env vars in the subscription.config.  For example, if your CSV has env vars [A,B,C] and you want to override B and add [X,Y,Z], use [B,X,Y,Z] in the subscription config.

Upstream PR: https://github.com/operator-framework/operator-lifecycle-manager/pull/2098

Downstream PR: https://github.com/openshift/operator-framework-olm/pull/53

> A workaround is to always list the overrides before the new env vars in the subscription.config.  For example, if your CSV has env vars [A,B,C] and you want to override B and add [X,Y,Z], use [B,X,Y,Z] in the subscription config.

I did not see this as a workaround that works. Here's what I did - it was very simple. I just want to override a single env var in my operator deployment:

1. Create a subscription to the Red Hat Kiali Operator:

cat <<EOM | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: kiali-ossm
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: kiali-ossm
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  config:
    env:
    - name: "ANSIBLE_CONFIG"
      value: "/opt/ansible/ansible-profiler.cfg"
EOM

2. Look at the Operator Deployment and Pod and notice the ANSIBLE_CONFIG env var value remains the default of "/etc/ansible/ansible.cfg" - it did not pick up my override:

oc get -n openshift-operators deployment kiali-operator -o yaml

oc get -n openshift-operators pods -l app=kiali-operator -o yaml

Note: I can edit the CSV and successfully change the ANSIBLE_CONFIG (or any other env var I want) - that is my workaround to this problem.

Cluster version is 4.8.0-0.nightly-2021-04-15-202330
[root@preserve-olm-env jian]# oc exec catalog-operator-58b8498745-ksz8t -- olm --version
OLM version: 0.17.0
git commit: 8c384e081188facc5f00b992f72e0b77f44037a5

1, Create an OperatorGroup in the default project.
[root@preserve-olm-env jian]# oc get og -n default
NAME            AGE
default-kqlhs   20m

2, Subscribe etcd operator with the below ENVs:
[root@preserve-olm-env jian]# cat sub-etcd.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd
  namespace: default
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
  config:
    env:
    - name: ISO_IMAGE_TYPE
      value: "minimal-iso"
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.2","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.2-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-140229","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
    - name: OPERATOR_CONDITION_NAME
      value: etcdoperator.v0.9.5
    - name: MY_POD_NAMESPACE
      value: default

[root@preserve-olm-env jian]# oc create -f sub-etcd.yaml 
subscription.operators.coreos.com/etcd created
[root@preserve-olm-env jian]# oc get sub
NAME   PACKAGE   SOURCE                CHANNEL
etcd   etcd      community-operators   singlenamespace-alpha
[root@preserve-olm-env jian]# oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-crjzc   etcdoperator.v0.9.4   Automatic   true
[root@preserve-olm-env jian]# oc get csv
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded
[root@preserve-olm-env jian]# oc get pods
NAME                             READY   STATUS    RESTARTS   AGE
etcd-operator-85c5ccdd65-24cvp   3/3     Running   0          9s

3, Check the ENVs in the etcd operator pods.
[root@preserve-olm-env jian]# oc get pods etcd-operator-85c5ccdd65-24cvp -o yaml
apiVersion: v1
kind: Pod
...
        env:
        - name: MY_POD_NAMESPACE
          value: default
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ISO_IMAGE_TYPE
          value: minimal-iso
        - name: OPENSHIFT_VERSIONS
          value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.2","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.2-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-140229","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
        - name: OPERATOR_CONDITION_NAME
          value: etcdoperator.v0.9.4

1) the new MY_POD_NAMESPACE overrides the container's env var.
2) OPENSHIFT_VERSIONS variable has 4.8 entries.
3) OPERATOR_CONDITION_NAME overrides the one defined in the subscription 
[root@preserve-olm-env jian]# oc get operatorcondition
NAME                  AGE
etcdoperator.v0.9.4   4m39s

LGTM, verify it.

Also confirmed in my instance:

oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-20-070522   True        False         99m     Cluster version is 4.8.0-0.nightly-2021-04-20-070522

oc get subscriptions.operators.coreos.com assisted-service-operator -o yaml 
spec:
  channel: alpha
  config:
    env:
    - name: ISO_IMAGE_TYPE
      value: minimal-iso
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.5","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.5-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-20-101404","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'


oc get pod assisted-service-846887c65-p4ttn -o yaml
    - name: SELF_VERSION
      value: quay.io/ocpmetal/assisted-service:latest
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4
.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.5","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.5-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-20-101404","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
    - name: ISO_IMAGE_TYPE


updated OPENSHIFT_VERSIONS env now propagates to the operand pod.

Confirmed fixed from my side as well.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438