Bug 1732613 - Setting config in Subscritpion for env vars takes a long time to reflect in operator pods
Summary: Setting config in Subscritpion for env vars takes a long time to reflect in o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.2.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1737057
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-23 21:44 UTC by Evan Cordell
Modified: 2019-10-16 06:31 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:30:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 958 0 None closed Bug 1732613: Follow up for pod configuration 2020-02-25 06:13:12 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:31:00 UTC

Description Evan Cordell 2019-07-23 21:44:28 UTC
Description of problem:

If env vars are set for pod configuration, the installed CSV can take minutes to reflect the changes.




Version-Release number of selected component (if applicable):


How reproducible: Always


Steps to Reproduce:
1. Create a Subscription
2. Set an environment variable via `spec.config.env`, such as `PROXY_HTTP`
3. Wait

Actual results:

The operator pod will not have the env variable at first, but after several minutes, it will.

Expected results:

The operator pod has the env vars set more or less immediately

Comment 2 Jian Zhang 2019-07-29 06:34:51 UTC
Verify failed.
1, Subscribe the etcd operator, like below:
mac:~ jianzhang$ oc get sub -n default
NAME   PACKAGE   SOURCE                CHANNEL
etcd   etcd      community-operators   singlenamespace-alpha
mac:~ jianzhang$ oc get csv -n default
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded
mac:~ jianzhang$ oc get pods -n default
NAME                             READY   STATUS    RESTARTS   AGE
etcd-operator-779f78d8f9-8tllg   3/3     Running   0          10m


2, Add environment variable into this etcd subscription.
mac:~ jianzhang$ oc get sub -n default etcd -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
...
spec:
  channel: singlenamespace-alpha
  config:
  - env:
    - name: PROXY_HTTP
      value: test
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

But, got below errors:
E0729 06:29:30.655645       1 reflector.go:126] pkg/lib/queueinformer/queueinformer_operator.go:152: Failed to list *v1alpha1.Subscription: v1alpha1.SubscriptionList.Items: []v1alpha1.Subscription: v1alpha1.Subscription.Spec: v1alpha1.SubscriptionSpec.Config: readObjectStart: expect { or n, but found [, error found in #10 byte of ...|"config":[{"env":[{"|..., bigger context ...|pec":{"channel":"singlenamespace-alpha","config":[{"env":[{"name":"PROXY_HTTP","value":"test"}]}],"i|...

OLM version:
mac:~ jianzhang$ oc exec catalog-operator-86f77c9666-2c5nk -- olm --version
OLM version: 0.11.0
git commit: d2209c409b35f1db4669c474044decc6995f624d

Comment 3 Evan Cordell 2019-08-02 13:24:41 UTC
Please use:

spec:
  config:
  - selector:                     
      matchLabels:
        name: etcd-operator-alm-owned
    env:
    - name: PROXY_HTTP
      value: test


as described here: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/contributors/design-proposals/operator-config.md

I will file a separate bug for the crd validation, which is missing the config block.

Comment 6 Jian Zhang 2019-08-05 06:00:49 UTC
The issue in comment 2 is tracing in bug 1737057.
For this bug, I config the subscription object according to comment 3. But the format changed after I save it.
The format of the current `config` field is wrong, it cannot be changed to JSON correctly. See below, @Evan Do you have any workaround for the YAML format? 
mac:~ jianzhang$ oc get sub etcd -o yaml
...
spec:
  channel: singlenamespace-alpha
  config:
  - env:
    - name: PROXY_HTTP
      value: test
    selector:
      matchLabels:
        name: etcd-operator-alm-owned
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
status:
  catalogHealth:
...

The correct format per comment 3, shoule be:
     config: 
      [ { selector: null,
          matchLabels: { name: 'etcd-operator-alm-owned' },
          env: [ { name: 'PROXY_HTTP', value: 'test' } ] } ],


So, still got the same errors as bug 1737057, see below:
E0805 05:37:14.495983       1 reflector.go:126] pkg/lib/queueinformer/queueinformer_operator.go:152: Failed to list *v1alpha1.Subscription: v1alpha1.SubscriptionList.Items: []v1alpha1.Subscription: v1alpha1.Subscription.Spec: v1alpha1.SubscriptionSpec.Config: readObjectStart: expect { or n, but found [, error found in #10 byte of ...|"config":[{"env":[{"|..., bigger context ...|pec":{"channel":"singlenamespace-alpha","config":[{"env":[{"name":"PROXY_HTTP","value":"test"}],"sel|...


mac:~ jianzhang$ oc exec catalog-operator-f8dffb5df-mrcw4 -- olm --version
OLM version: 0.11.0
git commit: 26e2cc30fb5a7e35df43102c331da4b5467114a2

Comment 11 Jian Zhang 2019-09-08 03:58:39 UTC
Hi, Evan

I followed your suggestion in comment 9, but failed, is it a bug?
mac:~ jianzhang$ cat sub-etcd-42-config.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-config-test
  namespace: default
spec:
  config:
  - selector:
     matchLabels:
       name: etcd-operator-alm-owned
    env:
    - name: PROXY_HTTP
      value: test
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

mac:~ jianzhang$ oc create -f sub-etcd-42-config.yaml 
The Subscription "etcd-config-test" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"operators.coreos.com/v1alpha1", "kind":"Subscription", "metadata":map[string]interface {}{"creationTimestamp":"2019-09-08T03:51:54Z", "generation":1, "name":"etcd-config-test", "namespace":"default", "uid":"ffafff5a-d1eb-11e9-b33a-02b34813d322"}, "spec":map[string]interface {}{"channel":"singlenamespace-alpha", "config":[]interface {}{map[string]interface {}{"env":[]interface {}{map[string]interface {}{"name":"PROXY_HTTP", "value":"test"}}, "selector":map[string]interface {}{"matchLabels":map[string]interface {}{"name":"etcd-operator-alm-owned"}}}}, "installPlanApproval":"Automatic", "name":"etcd", "source":"community-operators", "sourceNamespace":"openshift-marketplace", "startingCSV":"etcdoperator.v0.9.4"}}: validation failure list:
spec.config in body must be of type object: "array"


And, I'm confused, as Abu commented on https://jira.coreos.com/browse/OLM-1163, below:

'config' is not an array.
'env' is an array of object with 'name' and 'value'


I followed his suggestion, and have a try. But, failed again. So, what's the correct format for this "config" field?

mac:~ jianzhang$ cat sub-etcd-42-config.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-config-test
  namespace: default
spec:
  config:
  selector:
     matchLabels:
       name: etcd-operator-alm-owned
  env:
  - name: PROXY_HTTP
    value: test
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

mac:~ jianzhang$ oc create -f sub-etcd-42-config.yaml 
The Subscription "etcd-config-test" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"operators.coreos.com/v1alpha1", "kind":"Subscription", "metadata":map[string]interface {}{"creationTimestamp":"2019-09-08T03:55:23Z", "generation":1, "name":"etcd-config-test", "namespace":"default", "uid":"7c04fc18-d1ec-11e9-8d11-0a284293033a"}, "spec":map[string]interface {}{"channel":"singlenamespace-alpha", "config":interface {}(nil), "env":[]interface {}{map[string]interface {}{"name":"PROXY_HTTP", "value":"test"}}, "installPlanApproval":"Automatic", "name":"etcd", "selector":map[string]interface {}{"matchLabels":map[string]interface {}{"name":"etcd-operator-alm-owned"}}, "source":"community-operators", "sourceNamespace":"openshift-marketplace", "startingCSV":"etcdoperator.v0.9.4"}}: validation failure list:
spec.config in body must be of type object: "null"

Comment 12 Evan Cordell 2019-09-09 13:28:14 UTC
```
spec:
  channel: singlenamespace-alpha
  config:
    env:
      - name: PROXY_HTTP
        value: test
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4
```

Sorry about that - this is the correct format to use.

Comment 14 Jian Zhang 2019-09-10 06:18:32 UTC
Hi, Evan

Thanks! But, as you mentioned in comment 9:

> a selector is required ...
> The selector tells OLM which pods in the operator to apply the config to (since operators can be made from many pods).

I didn't see the `selector` configure item in comment 12. Could you help update it? Thanks!

Comment 15 Evan Cordell 2019-09-10 13:32:14 UTC
Hi Jian,

I was mistaken - sorry. That was something we discussed in the proposal document for this feature but did not implement for the first pass (which needed to be implemented very quickly for the proxy initiative).

The syntax I posted in my last comment should allow you to test the feature as it is implemented today.

Comment 16 Jian Zhang 2019-09-11 03:05:08 UTC
It works as expected, verify it, details as below:

Cluster version is 4.2.0-0.nightly-2019-09-10-181551

1, Create an OperatorGroup in project default.
mac:~ jianzhang$ oc create -f og.yaml 
operatorgroup.operators.coreos.com/test-og created
mac:~ jianzhang$ oc get og -n default
NAME      AGE
test-og   10s

2, Subscribe etcd operator with env vars ""
mac:~ jianzhang$ cat sub-etcd-42-config.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-config-test
  namespace: default
spec:
  config:
    env:
    - name: PROXY_HTTP
      value: test
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.4

mac:~ jianzhang$ oc create -f sub-etcd-42-config.yaml 
subscription.operators.coreos.com/etcd-config-test created

3, Check the pods of the etcd operator immediately. The env vars can be set within the 30s. Looks good.

mac:~ jianzhang$ oc get pod  etcd-operator-5964db9449-kp8hs -o yaml|grep -i "proxy" -A 2
    - name: PROXY_HTTP
      value: test
    image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
--
    - name: PROXY_HTTP
      value: test
    image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
--
    - name: PROXY_HTTP
      value: test
    image: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b

Comment 17 Jian Zhang 2019-09-11 03:07:33 UTC
mac:~ jianzhang$ oc exec catalog-operator-5544569dd7-lggqq  -- olm --version
OLM version: 0.11.0
git commit: 5a5389cb8d831e79acade535d947d4ad4a5c40a7

Comment 18 errata-xmlrpc 2019-10-16 06:30:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.