Bug 1751443 - Failed to config operator env
Summary: Failed to config operator env
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Kevin Rizza
QA Contact: yhui
URL:
Whiteboard:
: 1805256 (view as bug list)
Depends On:
Blocks: 1805258
TreeView+ depends on / blocked
 
Reported: 2019-09-12 02:50 UTC by Jian Zhang
Modified: 2020-07-13 17:11 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1805258 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:11:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
catalog source (253 bytes, patch)
2020-01-10 10:23 UTC, Jian Zhang
no flags Details | Diff
subscription (356 bytes, patch)
2020-01-10 10:23 UTC, Jian Zhang
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:11:52 UTC

Description Jian Zhang 2019-09-12 02:50:17 UTC
Description of problem:
Failed to config the operator env.

Version-Release number of selected component (if applicable):
Cluster version is 4.2.0-0.nightly-2019-09-11-032939
mac:~ jianzhang$ oc exec catalog-operator-68f64f5465-lz4rq -- olm --version
OLM version: 0.11.0
git commit: 95572bce2416891c270c90a4437dc4a121a43e72


How reproducible:
always

Steps to Reproduce:
1. Create the test CatalogSource in openshift-marketplace project.
mac:~ jianzhang$ cat cs-env.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: etcd-env-operator
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/jiazha/etcd-operator:env2
  displayName: ETCD Env Operators
  publisher: jian

mac:~ jianzhang$ oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     19h
community-operators   Community Operators   grpc   Red Hat     19h
etcd-env-operator     ETCD Env Operators    grpc   jian        29m
redhat-operators      Red Hat Operators     grpc   Red Hat     19h

2, Create a OperatorGroup in project default.
mac:~ jianzhang$ cat og.yaml 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: jian-operators
  namespace: default
spec:
  targetNamespaces:
  - default

mac:~ jianzhang$ oc get og -n default
NAME      AGE
test-og   29m


3, Create a subscription with env configure. Like below:
mac:~ jianzhang$ cat sub-env2.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: etcd-env-
  namespace: default
spec:
  source: etcd-env-operator
  sourceNamespace: openshift-marketplace
  name: etcd
  startingCSV: etcdoperator.v0.9.2
  channel: alpha
  config:
    env:
    - name: ARGS1
      value: -v=4
    - name: EMPTY_ENV


4, Check the etcd-operator env vars.
mac:~ jianzhang$ oc get sub
NAME             PACKAGE   SOURCE              CHANNEL
etcd-env-gm2cw   etcd      etcd-env-operator   alpha
mac:~ jianzhang$ oc get csv
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.2   etcd      0.9.2     etcdoperator.v0.9.0   Succeeded

Actual results:
No correspond env value set in the operator pod. 
mac:~ jianzhang$ oc get deployment etcd-operator -o yaml -n default |grep -i "ARGS1" -A 3
        - $(ARGS1)
        - $(ARGS2)
        env:
        - name: MY_POD_NAMESPACE
mac:~ jianzhang$ oc get deployment etcd-operator -o yaml -n default |grep -i "EMPTY" -A 3
mac:~ jianzhang$ 


Expected results:
The empty env variable should be injected into the pod. And, the value of the "ARGS1" should be `-v=4`.

Additional info:

Comment 1 Evan Cordell 2019-09-12 14:34:09 UTC
I followed the same steps as you described:

1. Create the test CatalogSource in openshift-marketplace project.

2. Create a OperatorGroup in project default.

3. Create a subscription with env configured

4. Check the etcd-operator env vars:

oc -n default get deployments etcd-operator -o json | jq '.spec.template.spec.containers[0].env'
[
  {
    "name": "MY_POD_NAMESPACE",
    "valueFrom": {
      "fieldRef": {
        "apiVersion": "v1",
        "fieldPath": "metadata.namespace"
      }
    }
  },
  {
    "name": "MY_POD_NAME",
    "valueFrom": {
      "fieldRef": {
        "apiVersion": "v1",
        "fieldPath": "metadata.name"
      }
    }
  },
  {
    "name": "ARGS1",
    "value": "-v=4"
  },
  {
    "name": "EMPTY_ENV"
  }
]

Comment 3 Evan Cordell 2019-09-17 13:19:26 UTC
Hi Jian,

If you wait a little bit, does it resolve?

Comment 4 Dan Geoffroy 2019-09-18 19:09:25 UTC
Moving this to 4.3.  Will continue to try and reproduce but even if this is valid, we dont feel its a release blocker.  Will evaluate early in the 4.3 cycle and consider as a 4.2.z backport if valid.

Comment 5 Jian Zhang 2019-09-19 05:28:35 UTC
Hi, Evan

> If you wait a little bit, does it resolve?

No, I wait for about 20 minutes, but no env vars injected.

Comment 9 Jian Zhang 2019-11-13 02:38:15 UTC
Nick,

Yes, it happens 100% for me. All of the steps in comment 7.

Comment 16 Bowen Song 2019-12-05 17:00:18 UTC
Reassigning to target 4.4 as this continues to investigate. This is not a release blocker since we tried multiple times with the correct YAML files and produced the expected result - environment variables are present in the deployments. We expect the error may happen with invalid YAML files but then it is no longer a bug.

Comment 17 Jian Zhang 2020-01-10 10:23:02 UTC
Created attachment 1651222 [details]
catalog source

Comment 18 Jian Zhang 2020-01-10 10:23:55 UTC
Created attachment 1651223 [details]
subscription

Comment 20 Ben Luddy 2020-02-17 18:15:48 UTC
The attached subscription doesn't seem to reference the correct catalog source:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: etcd-env-
  namespace: default
spec:
  source: community-operators
  sourceNamespace: openshift-marketplace
  name: etcd
  startingCSV: etcdoperator.v0.9.4
  channel: singlenamespace-alpha
  config:
    env:
    - name: ARGS1
      value: "-v=4"
    - name: EMPTY_ENV
---
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: etcd-env-operator
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/jiazha/etcd-operator:env2
  displayName: ETCD Env Operators
  publisher: jian

That wasn't true of the subscription mentioned in the initial correspondence, though:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: etcd-env-
  namespace: default
spec:
  source: etcd-env-operator
  sourceNamespace: openshift-marketplace
  name: etcd
  startingCSV: etcdoperator.v0.9.2
  channel: alpha
  config:
    env:
    - name: ARGS1
      value: -v=4
    - name: EMPTY_ENV

Was that manifest file accidentally modified at some point, or is the attachment incorrect?

Comment 21 Nick Hale 2020-02-20 15:13:23 UTC
*** Bug 1805256 has been marked as a duplicate of this bug. ***

Comment 22 Ben Luddy 2020-02-21 17:56:35 UTC
Jian, could you also confirm that there was no pre-existing Deployment from a previous Subscription for this operator? If so, it sounds like this could be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1751903#c11.

Comment 23 Jian Zhang 2020-02-22 02:54:14 UTC
Hi, Ben

I'm sorry for the late to reply. For comment 20, I tried it again, I think that is an attachment mistake, sorry for the confusion.
Still failed to config the operator env. As follows:
mac:~ jianzhang$ oc -n default get deployment etcd-operator -o json|jq '.spec.template.spec.containers[0].env'
[
  {
    "name": "MY_POD_NAMESPACE",
    "valueFrom": {
      "fieldRef": {
        "apiVersion": "v1",
        "fieldPath": "metadata.namespace"
      }
    }
  },
  {
    "name": "MY_POD_NAME",
    "valueFrom": {
      "fieldRef": {
        "apiVersion": "v1",
        "fieldPath": "metadata.name"
      }
    }
  }
]

mac:~ jianzhang$ cat sub-env2.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: etcd-env-
  namespace: default
spec:
  source: etcd-env-operator
  sourceNamespace: openshift-marketplace
  name: etcd
  startingCSV: etcdoperator.v0.9.2
  channel: alpha
  config:
    env:
    - name: ARGS1
      value: "-v=4"
    - name: EMPTY_ENV

mac:~ jianzhang$ cat cs-env.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: etcd-env-operator
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/jiazha/etcd-operator:env2 (You can see here about the manifest stores in this image: https://github.com/jianzhangbjz/operator-registry/blob/etcd-env/manifests/etcd/0.9.2/etcdoperator.v0.9.2.clusterserviceversion.yaml#L189-L190)
  displayName: ETCD Env Operators
  publisher: jian


mac:~ jianzhang$ oc version
Client Version: 4.4.0-0.nightly-2020-02-17-211020
Server Version: 4.4.0-0.nightly-2020-02-16-205302
Kubernetes Version: v1.17.1

mac:~ jianzhang$ oc get sub -n default
NAME             PACKAGE   SOURCE              CHANNEL
etcd-env-r25lg   etcd      etcd-env-operator   alpha
mac:~ jianzhang$ oc get csv -n default
NAME                  DISPLAY   VERSION   REPLACES   PHASE
etcdoperator.v0.9.2   etcd      0.9.2                Succeeded
mac:~ jianzhang$ oc get pods -n default
NAME                            READY   STATUS    RESTARTS   AGE
etcd-operator-d8794959d-6nxtc   3/3     Running   0          8m24s
mac:~ jianzhang$ oc get sub -n default etcd-env-r25lg -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  creationTimestamp: "2020-02-22T02:41:17Z"
  generateName: etcd-env-
  generation: 1
  name: etcd-env-r25lg
  namespace: default
  resourceVersion: "57582"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/default/subscriptions/etcd-env-r25lg
  uid: 5a274569-68be-4fc9-b436-c5a4cf5bf96a
spec:
  channel: alpha
  name: etcd
  source: etcd-env-operator
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.2
...

mac:~ jianzhang$ oc get catalogsource -n openshift-marketplace etcd-env-operator -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  creationTimestamp: "2020-02-22T02:31:33Z"
  generation: 1
  name: etcd-env-operator
  namespace: openshift-marketplace
  resourceVersion: "56917"
  selfLink: /apis/operators.coreos.com/v1alpha1/namespaces/openshift-marketplace/catalogsources/etcd-env-operator
  uid: 75518d8a-8a24-4d63-b471-4685e2d44316
spec:
  displayName: ETCD Env Operators
  image: quay.io/jiazha/etcd-operator:env2
  publisher: jian
  sourceType: grpc
status:
  connectionState:
    address: etcd-env-operator.openshift-marketplace.svc:50051
    lastConnect: "2020-02-22T02:39:16Z"
    lastObservedState: READY
  registryService:
    createdAt: "2020-02-22T02:31:33Z"
    port: "50051"
    protocol: grpc
    serviceName: etcd-env-operator
    serviceNamespace: openshift-marketplace

Comment 24 Jian Zhang 2020-02-22 03:08:04 UTC
> Jian, could you also confirm that there was no pre-existing Deployment from a previous Subscription for this operator?

Yes, I can confirm that. This is a clean cluster, so there is no it.
mac:~ jianzhang$ oc get sub -A
NAMESPACE   NAME             PACKAGE   SOURCE              CHANNEL
default     etcd-env-r25lg   etcd      etcd-env-operator   alpha
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-16-205302   True        False         136m    Cluster version is 4.4.0-0.nightly-2020-02-16-205302

And, the latest version of the operator in this CatalogSource image is etcdoperator.v0.9.2. So no upgrading problem in here.
https://github.com/jianzhangbjz/operator-registry/tree/etcd-env/manifests/etcd/0.9.2

Comment 25 Kevin Rizza 2020-02-25 15:27:29 UTC
Hi Jian,

I tried this exact workflow with an OpenShift 4.4 cluster and I'm also not able to reproduce :)

Can you try to reproduce this again with the latest 4.4 and if you are able to reproduce it also send over the catalog operator and olm operator logs?

Comment 26 Evan Cordell 2020-02-28 13:33:39 UTC
Could you try to reproduce with a build that includes this PR? https://github.com/operator-framework/operator-lifecycle-manager/pull/1333

Comment 30 yhui 2020-03-10 11:57:48 UTC
Cluster version is 4.5.0-0.nightly-2020-03-06-190457.
[hui@localhost 1751443]$ oc exec catalog-operator-6d54448f87-z2d7c -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.14.2
git commit: 8985872bfd5888c8d7d49e4dfc9be162a87691e7

The bug has been verified on this cluster.

The actual results are as follows.
[hui@localhost 1751443]$ oc get deployment etcd-operator -o yaml -n default |grep -i "ARGS1" -A 3
        - $(ARGS1)
        - $(ARGS2)
        env:
        - name: MY_POD_NAMESPACE
--
        - name: ARGS1
          value: -v=4
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2
--
        - name: ARGS1
          value: -v=4
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2
--
        - name: ARGS1
          value: -v=4
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2

[hui@localhost 1751443]$ oc get deployment etcd-operator -o yaml -n default |grep -i "EMPTY" -A 3
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2
        imagePullPolicy: IfNotPresent
        name: etcd-operator
--
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2
        imagePullPolicy: IfNotPresent
        name: etcd-backup-operator
--
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2
        imagePullPolicy: IfNotPresent
        name: etcd-restore-operator

Comment 31 yhui 2020-03-13 15:28:14 UTC
Description of problem:
Tested the bug again and found it is not resolved completely. The operator env variable is not configured as the value set in the subscription.

Version-Release number of selected component (if applicable):
Cluster version is 4.5.0-0.nightly-2020-03-10-234440.
[hui@localhost test-run]$ oc exec catalog-operator-5868c7d9cd-xckcd -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.14.2
git commit: 8985872bfd5888c8d7d49e4dfc9be162a87691e7

How reproducible:
always

Steps to Reproduce:
1. Create the test CatalogSource in openshift-marketplace project.
$ cat cs-env.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: etcd-env-operator
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/jiazha/etcd-operator:env2
  displayName: ETCD Env Operators
  publisher: jian

$ oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
etcd-env-operator     ETCD Env Operators    grpc   jian        29m


2. Create a OperatorGroup in project default.
$ cat og.yaml 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: jian-operators
  namespace: default
spec:
  targetNamespaces:
  - default

$ oc get og -n default
NAME      AGE
test-og   29m



3. Create a Subscription resource without setting the `env` field.
$ cat sub-env.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 generateName: etcd-env-
 namespace: default
spec:
 source: etcd-env-operator
 sourceNamespace: openshift-marketplace
 name: etcd
 startingCSV: etcdoperator.v0.9.2
 channel: alpha

$ oc get deployment etcd-operator -o yaml -n default|grep "\- command" -A 5
      - command:
        - etcd-operator
        - --create-crd=false
        - $(ARGS1)
        - $(ARGS2)



4. Edit this Subscription object, add the env variable. Like below:
$ cat sub-env2.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: etcd-env-
  namespace: default
spec:
  source: etcd-env-operator
  sourceNamespace: openshift-marketplace
  name: etcd
  startingCSV: etcdoperator.v0.9.2
  channel: alpha
  config:
    env:
    - name: ARGS1
      value: -v=4
    - name: EMPTY_ENV



5. Check the etcd-operator env vars.
$ oc get sub
NAME             PACKAGE   SOURCE              CHANNEL
etcd-env-gm2cw   etcd      etcd-env-operator   alpha
$ oc get csv
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.2   etcd      0.9.2     etcdoperator.v0.9.0   Succeeded


Actual results:
The env value $(ARGS1) in the deployment is not replaced by the value (-v=4) set in the subscription. 
$ oc get deployment etcd-operator -o yaml -n default |grep "\- command" -A 5
      - command:
        - etcd-operator
        - --create-crd=false
        - $(ARGS1)
        - $(ARGS2)
        env:
        - name: MY_POD_NAMESPACE


Expected results:
The env variable "ARGS1" should be replaced by `-v=4`.
The expected result of the command should be:
$ oc get deployment etcd-operator -o yaml -n default |grep "\- command" -A 5
      - command:
        - etcd-operator
        - --create-crd=false
        - -v=4
        env:
        - name: MY_POD_NAMESPACE

Comment 32 Evan Cordell 2020-03-13 16:24:30 UTC
I walked through the exact same steps on  4.4.0-0.nightly-2020-03-13-073111

Edited Subscription: 

```
spec:
  channel: singlenamespace-alpha
  config:
    env:
      - name: ARGS1
        value: '-v=4'
      - name: EMPTY_ENV
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
```


Looked at the deployment:

```
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: ARGS1
              value: '-v=4'
            - name: EMPTY_ENV
```

I also watched the events for the CSV and saw it transition back and re-install because it noticed the subscription change.

Could you try to reproduce this again, but this time please share the full contents of all related object (subscription, csv, deployment) at each step? I cannot reproduce and I would like to make sure we're not missing something important.

Comment 35 yhui 2020-03-18 06:40:14 UTC
(In reply to Evan Cordell from comment #32)
> I walked through the exact same steps on  4.4.0-0.nightly-2020-03-13-073111
> 
> Edited Subscription: 
> 
> ```
> spec:
>   channel: singlenamespace-alpha
>   config:
>     env:
>       - name: ARGS1
>         value: '-v=4'
>       - name: EMPTY_ENV
>   installPlanApproval: Automatic
>   name: etcd
>   source: community-operators
>   sourceNamespace: openshift-marketplace
>   startingCSV: etcdoperator.v0.9.4
> ```
> 
> 
> Looked at the deployment:
> 
> ```
>           env:
>             - name: MY_POD_NAMESPACE
>               valueFrom:
>                 fieldRef:
>                   apiVersion: v1
>                   fieldPath: metadata.namespace
>             - name: MY_POD_NAME
>               valueFrom:
>                 fieldRef:
>                   apiVersion: v1
>                   fieldPath: metadata.name
>             - name: ARGS1
>               value: '-v=4'
>             - name: EMPTY_ENV
> ```

Yes, the newly added env part is injected to the deployment. This is what we expected. But the env variable $(ARGS1) in the deployment command: part is not replaced by its value -v=4. 

  Look at the deployment:
  ```
          command:
            - etcd-operator
            - --create-crd=false

            - $(ARGS1)                           # This env variable should be replaced by its value -v=4 ?

            - $(ARGS2)
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: MY_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: ARGS1                        # This part is what we expected.
              value: '-v=4'
            - name: EMPTY_ENV
  ```

> 
> I also watched the events for the CSV and saw it transition back and
> re-install because it noticed the subscription change.
> 
> Could you try to reproduce this again, but this time please share the full
> contents of all related object (subscription, csv, deployment) at each step?
> I cannot reproduce and I would like to make sure we're not missing something
> important.

Comment 36 yhui 2020-03-18 07:20:56 UTC
Description of problem:
Tested the bug again and found it is not resolved completely. 
The operator env variable $(ARGS1) is not configured as the value set in the subscription.

Version-Release number of selected component (if applicable):
Cluster version is 4.5.0-0.nightly-2020-03-17-225152.
[hui@localhost test-run]$ oc exec catalog-operator-69b7cd5db9-v9gbm -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.14.2
git commit: 3455a009647abeb4f1791b3539a9a660411b8895

How reproducible:
always

Steps to Reproduce:
1. Create the test CatalogSource in openshift-marketplace project.
$ cat cs-env.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: etcd-env-operator
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/jiazha/etcd-operator:env2       #The image is built by ourselves and it includes the env variable $(ARGS1) in spec.containers.command. 
  # https://github.com/jianzhangbjz/operator-registry/blob/etcd-env/manifests/etcd/0.9.2/etcdoperator.v0.9.2.clusterserviceversion.yaml#L189-L190
  displayName: ETCD Env Operators
  publisher: jian

$ oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
etcd-env-operator     ETCD Env Operators    grpc   jian        29m



2. Create a OperatorGroup in project default.
$ cat og.yaml 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: jian-operators
  namespace: default
spec:
  targetNamespaces:
  - default

$ oc get og -n default
NAME      AGE
test-og   29m



3. Create a Subscription resource without setting the `env` field.
$ cat sub-env.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 generateName: etcd-env-
 namespace: default
spec:
 source: etcd-env-operator
 sourceNamespace: openshift-marketplace
 name: etcd
 startingCSV: etcdoperator.v0.9.2
 channel: alpha

$ oc get deployment etcd-operator -o yaml -n default|grep "\- command" -A 15
  
      - command:
        - etcd-operator
        - --create-crd=false
        - $(ARGS1)
        - $(ARGS2)
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2



4. Edit this Subscription object, add the env variable. Like below:
$ cat sub-env2.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: etcd-env-
  namespace: default
spec:
  source: etcd-env-operator
  sourceNamespace: openshift-marketplace
  name: etcd
  startingCSV: etcdoperator.v0.9.2
  channel: alpha
  config:
    env:
    - name: ARGS1
      value: '-v=4'
    - name: EMPTY_ENV



5. Check the etcd-operator env vars.
$ oc get sub
NAME             PACKAGE   SOURCE              CHANNEL
etcd-env-gm2cw   etcd      etcd-env-operator   alpha
$ oc get csv
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.2   etcd      0.9.2     etcdoperator.v0.9.0   Succeeded



Actual results:
The env value $(ARGS1) in the deployment is not replaced by the value (-v=4) that is set in the subscription. 
$ oc get deployment etcd-operator -o yaml -n default |grep "\- command" -A 15

      - command:
        - etcd-operator
        - --create-crd=false
        - $(ARGS1)                         # This env variable $(ARGS1) should be replaced by its value -v=4.
        - $(ARGS2)
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ARGS1
          value: -v=4
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2



Expected results:
The env variable "ARGS1" in the command part should be replaced by `-v=4`.
The expected result of the command should be:
$ oc get deployment etcd-operator -o yaml -n default |grep "\- command" -A 15

      - command:
        - etcd-operator
        - --create-crd=false
        - -v=4                             # It is what we expected.
        - $(ARGS2)
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ARGS1
          value: -v=4
        - name: EMPTY_ENV
        image: quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2


Thanks for your reply. Hope the information provided this time is helpful to understand. If any problems, feel free to let me know.

Comment 37 Evan Cordell 2020-03-20 14:02:52 UTC
>         - -v=4                             # It is what we expected.


Env var substitution does not happen on the Pod Spec, it happens when the container is started by the kubelet. You should not expect to see this value in the Pod, but you should expect to see its effects (i.e. more logging) in the container that is started.

https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#use-environment-variables-to-define-arguments

Comment 39 yhui 2020-03-26 09:51:37 UTC
Hi Evan,

Thanks for your reply. 
I tested the bug again and I found the env var substitution really happened when the container was started. 

Cluster version is 4.5.0-0.nightly-2020-03-25-125610.
[hui@localhost test-run]$ oc exec catalog-operator-7cc7697bbd-6zsrq -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.14.2
git commit: 915c118f5845e72265ecd9b2cf59a1a26819fb92

The test procedure is same as comment 36.

Actual results:
[hui@localhost 1751443]$ oc get pods
NAME                             READY   STATUS    RESTARTS   AGE
etcd-operator-5fbffb8446-76gwx   3/3     Running   0          2m39s

[hui@localhost 1751443]$ oc rsh etcd-operator-5fbffb8446-76gwx
Defaulting container name to etcd-operator.
Use 'oc describe pod/etcd-operator-5fbffb8446-76gwx -n default' to see all of the containers in this pod.
/ 
$ ps -elf|cat
PID   USER     TIME   COMMAND
    1 etcd-ope   0:00 etcd-operator --create-crd=false -v=4 $(ARGS2)           # The env var $(ARGS1) is replaced by its value -v=4
   11 etcd-ope   0:00 /bin/sh -c TERM="xterm-256color" /bin/sh
   17 etcd-ope   0:00 /bin/sh
   18 etcd-ope   0:00 ps -elf
   19 etcd-ope   0:00 cat

The env var $(ARGS1) is replaced by its value -v=4 in the command. It is what we expected.

Mark the bug verified.

Comment 40 yhui 2020-03-26 09:56:28 UTC
(In reply to Evan Cordell from comment #37)
> >         - -v=4                             # It is what we expected.
> 
> 
> Env var substitution does not happen on the Pod Spec, it happens when the
> container is started by the kubelet. You should not expect to see this value
> in the Pod, but you should expect to see its effects (i.e. more logging) in
> the container that is started.
> 
> https://kubernetes.io/docs/tasks/inject-data-application/define-command-
> argument-container/#use-environment-variables-to-define-arguments

Hi Evan,

I have a question about logging to consult you. Which logging file should I check? 

I used the following command to check logs. But the command is not recorded in the logs.
[hui@localhost lib]$ oc logs etcd-operator-869c4bd946-jxg2s etcd-operator
time="2020-03-26T01:14:47Z" level=info msg="etcd-operator Version: 0.9.2"
time="2020-03-26T01:14:47Z" level=info msg="Git SHA: a0032c1f"
time="2020-03-26T01:14:47Z" level=info msg="Go Version: go1.10"
time="2020-03-26T01:14:47Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-03-26T01:15:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"default\", Name:\"etcd-operator\", UID:\"6e7d0316-67b3-4e7d-8eab-4cdbb6d017c6\", APIVersion:\"v1\", ResourceVersion:\"178132\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' etcd-operator-869c4bd946-jxg2s became leader"

Thank you in advance.
Hui

Comment 42 errata-xmlrpc 2020-07-13 17:11:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.