Bug 1949294 - [assisted operator] OPENSHIFT_VERSIONS in assisted operator subscription does not propagate
Summary: [assisted operator] OPENSHIFT_VERSIONS in assisted operator subscription does...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.8
Hardware: All
OS: Linux
high
high
Target Milestone: ---
: 4.8.0
Assignee: Joe Lanford
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-13 21:42 UTC by Chad Crum
Modified: 2022-02-07 13:40 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:00:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift operator-framework-olm pull 53 0 None open Bug 1949294: fix environment variable merging from subscription.config 2021-04-14 14:53:42 UTC
Github operator-framework operator-lifecycle-manager pull 2098 0 None closed fix environment variable merging from subscription.config 2021-04-14 14:53:46 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:00:38 UTC

Description Chad Crum 2021-04-13 21:42:20 UTC
Description of problem:
OPENSHIFT_VERSIONS env var set in the operator subscription does not propagate to the assisted operator or operand unless the variable is the first variable defined in the subscription.

Version-Release number of selected component (if applicable):
ipv4 / connected / libvirt
      imageID: quay.io/ocpmetal/assisted-service-operator-bundle@sha256:0d4c3662ea493c34997a00c07e7aff916e707c8dcfcda3a065dcb4c7071f9818
      imageID: quay.io/ocpmetal/assisted-service-index@sha256:f723b8a99c8f898260e233e0c5f2bb110c7d82356f46cd3616adae7b5c93f20b
4.8.0-0.nightly-2021-04-13-091654

How reproducible:
100%

Steps to Reproduce:
1. Deploy ocp 4.8 cluster
2. Create catalog source for assisted service operator via recent index image + operator bundle
3. Create subscription for assisted service operator using the following:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: assisted-service-operator
  namespace: assisted-installer
spec:
  channel: "alpha"
  installPlanApproval: Automatic
  name: assisted-service-operator
  source: assisted-installer-index
  sourceNamespace: openshift-marketplace
  startingCSV: assisted-service-operator.v0.0.1
  config:
    env:
    - name: ISO_IMAGE_TYPE
      value: "minimal-iso"
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.2","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.2-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-140229","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
4. Deploy  assisted service operand with assisted service config cr

Actual results:
OPENSHIFT_VERSIONS variable in assisted operator and operand pods (and corresponding deployments) will only have the 4.6 and 4.7 entries - the 4.8 entry will be missing even though it should be there.

Expected results:
The assisted operator and operand both properly get the subscription OPENSHIFT_VERSION variable applied

Additional info:
If the subscription cr has the OPENSHIFT_VERSIONS variable set first in the list (as below), it will properly propagate to the operator/operand. In fact if editing the subscription and making the change, the operator and operand pods will restart and get the correct value. This does not happen if OPENSHIFT_VERSIONS is not set as the first entry.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: assisted-service-operator
  namespace: assisted-installer
spec:
  channel: "alpha"
  installPlanApproval: Automatic
  name: assisted-service-operator
  source: assisted-installer-index
  sourceNamespace: openshift-marketplace
  startingCSV: assisted-service-operator.v0.0.1
  config:
    env:
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.2","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.2-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-140229","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
    - name: ISO_IMAGE_TYPE
      value: "minimal-iso"

Comment 1 David Zager 2021-04-13 22:45:38 UTC
Moving this to the OLM component as this has to do with Subscriptions -> operator deployments. This mechanism is required in order for users to communicate the location of rhcos live isos and essential for disconnected flows.

Adding @asegurap@redhat.com to evaluate if this is a blocker or not. My assessment: yes. but I would not be surprised if there was additional criteria I'm not aware of.

Comment 2 Joe Lanford 2021-04-14 04:03:07 UTC
It looks like this is a bug in the way OLM overrides existing environment variables vs adding new environment variables.

A workaround is to always list the overrides before the new env vars in the subscription.config.  For example, if your CSV has env vars [A,B,C] and you want to override B and add [X,Y,Z], use [B,X,Y,Z] in the subscription config.

Comment 4 Joe Lanford 2021-04-14 14:54:13 UTC
Downstream PR: https://github.com/openshift/operator-framework-olm/pull/53

Comment 6 jmazzite 2021-04-15 11:47:13 UTC
> A workaround is to always list the overrides before the new env vars in the subscription.config.  For example, if your CSV has env vars [A,B,C] and you want to override B and add [X,Y,Z], use [B,X,Y,Z] in the subscription config.

I did not see this as a workaround that works. Here's what I did - it was very simple. I just want to override a single env var in my operator deployment:

1. Create a subscription to the Red Hat Kiali Operator:

cat <<EOM | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: kiali-ossm
  namespace: openshift-operators
spec:
  channel: stable
  installPlanApproval: Automatic
  name: kiali-ossm
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  config:
    env:
    - name: "ANSIBLE_CONFIG"
      value: "/opt/ansible/ansible-profiler.cfg"
EOM

2. Look at the Operator Deployment and Pod and notice the ANSIBLE_CONFIG env var value remains the default of "/etc/ansible/ansible.cfg" - it did not pick up my override:

oc get -n openshift-operators deployment kiali-operator -o yaml

oc get -n openshift-operators pods -l app=kiali-operator -o yaml

Note: I can edit the CSV and successfully change the ANSIBLE_CONFIG (or any other env var I want) - that is my workaround to this problem.

Comment 7 Jian Zhang 2021-04-16 02:59:16 UTC
Cluster version is 4.8.0-0.nightly-2021-04-15-202330
[root@preserve-olm-env jian]# oc exec catalog-operator-58b8498745-ksz8t -- olm --version
OLM version: 0.17.0
git commit: 8c384e081188facc5f00b992f72e0b77f44037a5

1, Create an OperatorGroup in the default project.
[root@preserve-olm-env jian]# oc get og -n default
NAME            AGE
default-kqlhs   20m

2, Subscribe etcd operator with the below ENVs:
[root@preserve-olm-env jian]# cat sub-etcd.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd
  namespace: default
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
  config:
    env:
    - name: ISO_IMAGE_TYPE
      value: "minimal-iso"
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.2","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.2-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-140229","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
    - name: OPERATOR_CONDITION_NAME
      value: etcdoperator.v0.9.5
    - name: MY_POD_NAMESPACE
      value: default

[root@preserve-olm-env jian]# oc create -f sub-etcd.yaml 
subscription.operators.coreos.com/etcd created
[root@preserve-olm-env jian]# oc get sub
NAME   PACKAGE   SOURCE                CHANNEL
etcd   etcd      community-operators   singlenamespace-alpha
[root@preserve-olm-env jian]# oc get ip
NAME            CSV                   APPROVAL    APPROVED
install-crjzc   etcdoperator.v0.9.4   Automatic   true
[root@preserve-olm-env jian]# oc get csv
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded
[root@preserve-olm-env jian]# oc get pods
NAME                             READY   STATUS    RESTARTS   AGE
etcd-operator-85c5ccdd65-24cvp   3/3     Running   0          9s

3, Check the ENVs in the etcd operator pods.
[root@preserve-olm-env jian]# oc get pods etcd-operator-85c5ccdd65-24cvp -o yaml
apiVersion: v1
kind: Pod
...
        env:
        - name: MY_POD_NAMESPACE
          value: default
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ISO_IMAGE_TYPE
          value: minimal-iso
        - name: OPENSHIFT_VERSIONS
          value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.2","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.2-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-140229","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
        - name: OPERATOR_CONDITION_NAME
          value: etcdoperator.v0.9.4

1) the new MY_POD_NAMESPACE overrides the container's env var.
2) OPENSHIFT_VERSIONS variable has 4.8 entries.
3) OPERATOR_CONDITION_NAME overrides the one defined in the subscription 
[root@preserve-olm-env jian]# oc get operatorcondition
NAME                  AGE
etcdoperator.v0.9.4   4m39s

LGTM, verify it.

Comment 8 Chad Crum 2021-04-20 22:39:18 UTC
Also confirmed in my instance:

oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-20-070522   True        False         99m     Cluster version is 4.8.0-0.nightly-2021-04-20-070522

oc get subscriptions.operators.coreos.com assisted-service-operator -o yaml 
spec:
  channel: alpha
  config:
    env:
    - name: ISO_IMAGE_TYPE
      value: minimal-iso
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.5","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.5-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-20-101404","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'


oc get pod assisted-service-846887c65-p4ttn -o yaml
    - name: SELF_VERSION
      value: quay.io/ocpmetal/assisted-service:latest
    - name: OPENSHIFT_VERSIONS
      value: '{"4.6":{"display_name":"4.6.16","release_image":"quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4
.6/4.6.8/rhcos-4.6.8-x86_64-live.x86_64.iso","rhcos_version":"46.82.202012051820-0","support_level":"production"},"4.7":{"display_name":"4.7.5","release_image":"quay.io/openshift-release-dev/ocp-release:4.7.5-x86_64","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"},"4.8":{"display_name":"4.8","release_image":"registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-20-101404","rhcos_image":"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-live.x86_64.iso","rhcos_version":"47.83.202102090044-0","support_level":"production"}}'
    - name: ISO_IMAGE_TYPE


updated OPENSHIFT_VERSIONS env now propagates to the operand pod.

Confirmed fixed from my side as well.

Comment 11 errata-xmlrpc 2021-07-27 23:00:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.