Bug 1667363

Summary: config map can not be updated If ASB is deployed by operator
Product: OpenShift Container Platform Reporter: Zihan Tang <zitang>
Component: Service BrokerAssignee: jkim
Status: CLOSED ERRATA QA Contact: Zihan Tang <zitang>
Severity: high Docs Contact:
Priority: high    
Version: 4.1.0CC: aos-bugs, chezhang, dyan, jfan, jiazha, jmatthew, sponnaga
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:42:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1668534    
Bug Blocks:    

Description Zihan Tang 2019-01-18 09:19:51 UTC
Description of problem:
If ASB is deployed by ASB operator, cm broker-config can not update successfully.

Version-Release number of selected component (if applicable):
openshift-install v0.10.0
Cluster version is 4.0.0-0.1
OLM version: 0.8.1  git commit: 54a541d

How reproducible:
always

Steps to Reproduce:
1. deploy asb operator and asb using file: https://github.com/fusor/catbrokers4
2. edit registry in cm broker-config 
3.

Actual results:
cm can not update successfully. will rollback automatically.

Expected results:
Provide some ways to update cm 

Additional info:
we have tried to stop the asb operators and some other ways, sometime it brokes the asb pod, and still not update cm successfully.

Comment 1 Zihan Tang 2019-01-18 09:21:23 UTC
This bug blocks current test run, so add testbloker keyword.

Comment 2 Zhang Cheng 2019-01-18 10:51:07 UTC
We noticed the configmap of automation broker is watching by operator. And no method to change broker config at present. We tried to stop (or remove) broker operator, but asb pod crash since didn't find relate resource. 

So, we lLeave TestBlocker in here to block related TCs in current test run, and will remove it while get fix or a workaround. Thanks.


# oc rsh automation-broker-operator-66b644bfb5-xjn69
sh-4.2$ ps -ef | grep ansible-operator
ansible+      1      0  1 07:30 ?        00:01:00 /usr/local/bin/ansible-operator run ansible --watches-file=/opt/ansible/watches.yaml

sh-4.2$ cat watches.yaml
---

- name: AutomationBroker
  version: v1alpha1
  group: automationbroker.io
  kind: AutomationBroker
  playbook: /opt/ansible/deploy.yml
  watchDependentResources: false
  finalizer:
    name: finalizer.automationbroker.io
    vars:
      action: deprovision
      broker_destroy_shared_resources: true

sh-4.2$ cat /opt/ansible/deploy.yml
---

- name: automation-broker-operator
  hosts: localhost
  become: false
  gather_facts: false
  connection: local
  vars:
    state: present
    broker_name: "{{ meta.name }}"
    broker_namespace: "{{ meta.namespace }}"
    broker_image: "{{ lookup('env', 'IMAGE') }}"
  tasks:
    - name: Validation
      assert:
        that: broker_name and broker_namespace and broker_image
        fail_msg: >
          broker_name broker_namespace broker_image must be defined
          The IMAGE environment variable must provide the broker's image name
          and should be specified in the operator's deployment specification.
    - name: Run automation-broker role
      include_role:
        name: automation-broker

sh-4.2$ cat roles/automation-broker/vars/configmap.yml
---

broker_config_map:
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: broker-config
    namespace: '{{ broker_namespace }}'
    labels:
      app: '{{ broker_name }}'
  data:
    broker-config: '{{ broker_config | to_nice_yaml }}'

Comment 3 Zhang Cheng 2019-01-18 11:00:05 UTC
John, 

Could developer provide a workaround to change broker config at present? We can accept stop watching by broker operator until get formal solution since we are focus on regression test and ensuring old features/functions still work fine in OCP4.0, especially in operator of brokers is not shipped.

Comment 4 Erik Nelson 2019-01-18 13:22:20 UTC
It should be possible to do this with the current images, I'm confirming that this morning and will update this bz with the results and advice.

Comment 5 Erik Nelson 2019-01-18 15:56:54 UTC
I've tested and confirmed that the currently published ASB operator image supports broker configuration via the CR. You need to add a complete broker configuration to the CR under the "brokerConfig" variable. This will override the operator's defaults and the operator will simply take that configuration and create a ConfigMap for the broker. Additionally, it also supports changes. If you would like to change any configuration values for an operator managed broker, you should edit the broker config that is in corresponding CR. The operator will recognize that the configuration has changed, update the ConfigMap, and recreate the broker's pod with the new config mounted in.

Here is an example CR that contains a broker_config override: https://github.com/eriknelson/catbrokers4/blob/add-config-example/files/asb/05-asb-cr.yaml#L8

We are in the process of merging this to the catbrokers4 repo so that the playbook will use it.

Comment 6 Zhang Cheng 2019-01-19 02:08:04 UTC
Erik, Thanks for your quickly response.
We will checking

Comment 7 Zihan Tang 2019-01-21 07:01:59 UTC
At present, we can not deploy any operator successfully due to bug https://bugzilla.redhat.com/show_bug.cgi?id=1666225
we'll try when #bug 1666225 fixed or find workaround.

Comment 9 Zhang Cheng 2019-01-30 23:56:18 UTC
Yes. But we cannot verify it since it is blocking by another bug https://bugzilla.redhat.com/show_bug.cgi?id=1668534

So, we marked 'Depends On: 1668534'.

Comment 10 Erik Nelson 2019-02-01 20:30:09 UTC
The broker's operator has been updated with a limited set of supported configuration options that should be set in the broker's CR. All of the supported options are documented here: https://github.com/openshift/ansible-service-broker/blob/master/docs/operator.md

Manually editing the broker's config directly, in addition to setting the broker's config directly, are both no longer supported. If you must edit the broker's configuration directly, you will need to scale down the broker's operator by deleting the CSV + subscription in the operator's namespace. Then you will be able to edit the configmap directly without the operator overwriting your changes.

The updated operator image has been published upstream to docker.io/automationbroker/automation-broker-operator.

Comment 12 Zihan Tang 2019-02-19 10:24:31 UTC
Verified.
asb operator: docker.io/automationbroker/osb-operators
asb version: 1.4.4

Then cm can be updated in the following scenario:
1. recreate the automationbroker CR and add the registries etc. config in the CR.
2. edit automationbroker CR , add the config in 'spec' like: 
  brokerImage: docker.io/zitangbj/ose-ansible-service-broker:v4.0
  registries:
    - type: openshift
      name: aws
      url:  https://registry.reg-aws.openshift.com:443 
      tag:  v4.0
      images:
        - openshift/mediawiki-apb
        - openshift/postgresql-apb
        - openshift/mariadb-apb
....
   when the automationbroker work well, it will trigger update asb pod automatically, but asb operator pod often produce some error after running for some time and can not monitor the update or deletion of CR. I'll open another bug to trace it.
3. delete csv and subscription, then can edit broker-config cm directly as before v3.11

Comment 15 errata-xmlrpc 2019-06-04 10:42:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758