Bug 1732613

Summary: Setting config in Subscritpion for env vars takes a long time to reflect in operator pods
Product: OpenShift Container Platform Reporter: Evan Cordell <ecordell>
Component: OLMAssignee: Evan Cordell <ecordell>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: bandrade, chezhang, chuo, jfan, scolange
Version: 4.2.0   
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:30:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1737057    
Bug Blocks:    

Description Evan Cordell 2019-07-23 21:44:28 UTC
Description of problem:

If env vars are set for pod configuration, the installed CSV can take minutes to reflect the changes.




Version-Release number of selected component (if applicable):


How reproducible: Always


Steps to Reproduce:
1. Create a Subscription
2. Set an environment variable via `spec.config.env`, such as `PROXY_HTTP`
3. Wait

Actual results:

The operator pod will not have the env variable at first, but after several minutes, it will.

Expected results:

The operator pod has the env vars set more or less immediately

Comment 2 Jian Zhang 2019-07-29 06:34:51 UTC
Verify failed.
1, Subscribe the etcd operator, like below:
mac:~ jianzhang$ oc get sub -n default
NAME   PACKAGE   SOURCE                CHANNEL
etcd   etcd      community-operators   singlenamespace-alpha
mac:~ jianzhang$ oc get csv -n default
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded
mac:~ jianzhang$ oc get pods -n default
NAME                             READY   STATUS    RESTARTS   AGE
etcd-operator-779f78d8f9-8tllg   3/3     Running   0          10m


2, Add environment variable into this etcd subscription.
mac:~ jianzhang$ oc get sub -n default etcd -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
...
spec:
  channel: singlenamespace-alpha
  config:
  - env:
    - name: PROXY_HTTP
      value: test
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

But, got below errors:
E0729 06:29:30.655645       1 reflector.go:126] pkg/lib/queueinformer/queueinformer_operator.go:152: Failed to list *v1alpha1.Subscription: v1alpha1.SubscriptionList.Items: []v1alpha1.Subscription: v1alpha1.Subscription.Spec: v1alpha1.SubscriptionSpec.Config: readObjectStart: expect { or n, but found [, error found in #10 byte of ...|"config":[{"env":[{"|..., bigger context ...|pec":{"channel":"singlenamespace-alpha","config":[{"env":[{"name":"PROXY_HTTP","value":"test"}]}],"i|...

OLM version:
mac:~ jianzhang$ oc exec catalog-operator-86f77c9666-2c5nk -- olm --version
OLM version: 0.11.0
git commit: d2209c409b35f1db4669c474044decc6995f624d

Comment 3 Evan Cordell 2019-08-02 13:24:41 UTC
Please use:

spec:
  config:
  - selector:                     
      matchLabels:
        name: etcd-operator-alm-owned
    env:
    - name: PROXY_HTTP
      value: test


as described here: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/contributors/design-proposals/operator-config.md

I will file a separate bug for the crd validation, which is missing the config block.

Comment 6 Jian Zhang 2019-08-05 06:00:49 UTC
The issue in comment 2 is tracing in bug 1737057.
For this bug, I config the subscription object according to comment 3. But the format changed after I save it.
The format of the current `config` field is wrong, it cannot be changed to JSON correctly. See below, @Evan Do you have any workaround for the YAML format? 
mac:~ jianzhang$ oc get sub etcd -o yaml
...
spec:
  channel: singlenamespace-alpha
  config:
  - env:
    - name: PROXY_HTTP
      value: test
    selector:
      matchLabels:
        name: etcd-operator-alm-owned
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
status:
  catalogHealth:
...

The correct format per comment 3, shoule be:
     config: 
      [ { selector: null,
          matchLabels: { name: 'etcd-operator-alm-owned' },
          env: [ { name: 'PROXY_HTTP', value: 'test' } ] } ],


So, still got the same errors as bug 1737057, see below:
E0805 05:37:14.495983       1 reflector.go:126] pkg/lib/queueinformer/queueinformer_operator.go:152: Failed to list *v1alpha1.Subscription: v1alpha1.SubscriptionList.Items: []v1alpha1.Subscription: v1alpha1.Subscription.Spec: v1alpha1.SubscriptionSpec.Config: readObjectStart: expect { or n, but found [, error found in #10 byte of ...|"config":[{"env":[{"|..., bigger context ...|pec":{"channel":"singlenamespace-alpha","config":[{"env":[{"name":"PROXY_HTTP","value":"test"}],"sel|...


mac:~ jianzhang$ oc exec catalog-operator-f8dffb5df-mrcw4 -- olm --version
OLM version: 0.11.0
git commit: 26e2cc30fb5a7e35df43102c331da4b5467114a2

Comment 11 Jian Zhang 2019-09-08 03:58:39 UTC
Hi, Evan

I followed your suggestion in comment 9, but failed, is it a bug?
mac:~ jianzhang$ cat sub-etcd-42-config.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-config-test
  namespace: default
spec:
  config:
  - selector:
     matchLabels:
       name: etcd-operator-alm-owned
    env:
    - name: PROXY_HTTP
      value: test
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

mac:~ jianzhang$ oc create -f sub-etcd-42-config.yaml 
The Subscription "etcd-config-test" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"operators.coreos.com/v1alpha1", "kind":"Subscription", "metadata":map[string]interface {}{"creationTimestamp":"2019-09-08T03:51:54Z", "generation":1, "name":"etcd-config-test", "namespace":"default", "uid":"ffafff5a-d1eb-11e9-b33a-02b34813d322"}, "spec":map[string]interface {}{"channel":"singlenamespace-alpha", "config":[]interface {}{map[string]interface {}{"env":[]interface {}{map[string]interface {}{"name":"PROXY_HTTP", "value":"test"}}, "selector":map[string]interface {}{"matchLabels":map[string]interface {}{"name":"etcd-operator-alm-owned"}}}}, "installPlanApproval":"Automatic", "name":"etcd", "source":"community-operators", "sourceNamespace":"openshift-marketplace", "startingCSV":"etcdoperator.v0.9.4"}}: validation failure list:
spec.config in body must be of type object: "array"


And, I'm confused, as Abu commented on https://jira.coreos.com/browse/OLM-1163, below:

'config' is not an array.
'env' is an array of object with 'name' and 'value'


I followed his suggestion, and have a try. But, failed again. So, what's the correct format for this "config" field?

mac:~ jianzhang$ cat sub-etcd-42-config.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-config-test
  namespace: default
spec:
  config:
  selector:
     matchLabels:
       name: etcd-operator-alm-owned
  env:
  - name: PROXY_HTTP
    value: test
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

mac:~ jianzhang$ oc create -f sub-etcd-42-config.yaml 
The Subscription "etcd-config-test" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"operators.coreos.com/v1alpha1", "kind":"Subscription", "metadata":map[string]interface {}{"creationTimestamp":"2019-09-08T03:55:23Z", "generation":1, "name":"etcd-config-test", "namespace":"default", "uid":"7c04fc18-d1ec-11e9-8d11-0a284293033a"}, "spec":map[string]interface {}{"channel":"singlenamespace-alpha", "config":interface {}(nil), "env":[]interface {}{map[string]interface {}{"name":"PROXY_HTTP", "value":"test"}}, "installPlanApproval":"Automatic", "name":"etcd", "selector":map[string]interface {}{"matchLabels":map[string]interface {}{"name":"etcd-operator-alm-owned"}}, "source":"community-operators", "sourceNamespace":"openshift-marketplace", "startingCSV":"etcdoperator.v0.9.4"}}: validation failure list:
spec.config in body must be of type object: "null"

Comment 12 Evan Cordell 2019-09-09 13:28:14 UTC
```
spec:
  channel: singlenamespace-alpha
  config:
    env:
      - name: PROXY_HTTP
        value: test
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
```

Sorry about that - this is the correct format to use.

Comment 14 Jian Zhang 2019-09-10 06:18:32 UTC
Hi, Evan

Thanks! But, as you mentioned in comment 9:

> a selector is required ...
> The selector tells OLM which pods in the operator to apply the config to (since operators can be made from many pods).

I didn't see the `selector` configure item in comment 12. Could you help update it? Thanks!

Comment 15 Evan Cordell 2019-09-10 13:32:14 UTC
Hi Jian,

I was mistaken - sorry. That was something we discussed in the proposal document for this feature but did not implement for the first pass (which needed to be implemented very quickly for the proxy initiative).

The syntax I posted in my last comment should allow you to test the feature as it is implemented today.

Comment 16 Jian Zhang 2019-09-11 03:05:08 UTC
It works as expected, verify it, details as below:

Cluster version is 4.2.0-0.nightly-2019-09-10-181551

1, Create an OperatorGroup in project default.
mac:~ jianzhang$ oc create -f og.yaml 
operatorgroup.operators.coreos.com/test-og created
mac:~ jianzhang$ oc get og -n default
NAME      AGE
test-og   10s

2, Subscribe etcd operator with env vars ""
mac:~ jianzhang$ cat sub-etcd-42-config.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-config-test
  namespace: default
spec:
  config:
    env:
    - name: PROXY_HTTP
      value: test
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

mac:~ jianzhang$ oc create -f sub-etcd-42-config.yaml 
subscription.operators.coreos.com/etcd-config-test created

3, Check the pods of the etcd operator immediately. The env vars can be set within the 30s. Looks good.

mac:~ jianzhang$ oc get pod  etcd-operator-5964db9449-kp8hs -o yaml|grep -i "proxy" -A 2
    - name: PROXY_HTTP
      value: test
    image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
--
    - name: PROXY_HTTP
      value: test
    image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
--
    - name: PROXY_HTTP
      value: test
    image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b

Comment 17 Jian Zhang 2019-09-11 03:07:33 UTC
mac:~ jianzhang$ oc exec catalog-operator-5544569dd7-lggqq  -- olm --version
OLM version: 0.11.0
git commit: 5a5389cb8d831e79acade535d947d4ad4a5c40a7

Comment 18 errata-xmlrpc 2019-10-16 06:30:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922