Bug 1661569

Summary: Upgrade Fails at TASK [ansible_service_broker : Create the Broker resource in the catalog]
Product: OpenShift Container Platform Reporter: Jay Boyd <jaboyd>
Component: Service CatalogAssignee: Jay Boyd <jaboyd>
Status: CLOSED ERRATA QA Contact: Jian Zhang <jiazha>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: pkanthal
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The 3.11.43 update added health and liveness probes for the Service Catalog pods. Install was not waiting for the update rollout to finish before proceeding to update Ansible Service Broker. Because of timing, the Service Catalog pods were unavailable when the Broker attempted to register. Consequence: Ansible Service Broker update failed with an error indicating "the server is currently unable to handle the request (post clusterservicebrokers.servicecatalog.k8s.io)" Fix: Installation was updated to wait for the Service Catalog update rollout to finish before proceeding with installing Ansible Service Broker.
Story Points: ---
Clone Of: 1656925 Environment:
Last Closed: 2019-01-30 15:13:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1656925    
Bug Blocks:    

Comment 1 Jay Boyd 2018-12-21 15:35:21 UTC
fixed by https://github.com/openshift/openshift-ansible/pull/10658  in 3.11.z.

Comment 3 Jian Zhang 2019-01-17 13:58:27 UTC
LGTM, verify it. Details as below:

1, original OCP cluster: 3.11.20
[root@ip-172-18-7-221 ~]# oc version
oc v3.11.20
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-7-221.ec2.internal:8443
openshift v3.11.20
kubernetes v1.11.0+d4cacc0

[root@ip-172-18-7-221 ~]# oc get pods -n kube-service-catalog 
NAME                       READY     STATUS    RESTARTS   AGE
apiserver-zwfqr            1/1       Running   0          1h
controller-manager-ngwzw   1/1       Running   2          1h

[root@ip-172-18-7-221 ~]# oc get pods -n kube-service-catalog apiserver-zwfqr -o yaml |grep image
    image: registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11.20

[root@ip-172-18-7-221 ~]# docker run --rm -ti registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11.20 service-catalog --version
v3.11.20;Upstream:v0.1.31

2, Upgrade it to the OCP 3.11.69 with the release-3.11 of the openshift-ansible.
[root@ip-172-18-7-221 ~]# oc version
oc v3.11.69
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-7-221.ec2.internal:8443
openshift v3.11.69
kubernetes v1.11.0+d4cacc0

[root@ip-172-18-7-221 ~]# oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
apiserver-xv5zn            1/1       Running   8          30m
controller-manager-z4lv4   1/1       Running   8          30m

[root@ip-172-18-7-221 ~]# oc get pods -o yaml |grep image
      image: registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11

[root@ip-172-18-7-221 ~]# docker run --rm -ti registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11 service-catalog --version
v3.11.69;Upstream:v0.1.35

correlating logs:
TASK [openshift_service_catalog : Wait for API Server rollout success] *********
task path: /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/start.yml:2
Thursday 17 January 2019  13:22:42 +0000 (0:00:00.279)       0:15:35.323 ****** 
ok: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"attempts": 1, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/apiserver"], "delta": "0:00:36.021360", "end": "2019-01-17 08:23:51.766271", "rc": 0, "start": "2019-01-17 08:23:15.744911", "stderr": "", "stderr_lines": [], "stdout": "Waiting for daemon set \"apiserver\" rollout to finish: 0 of 1 updated pods are available...\ndaemon set \"apiserver\" successfully rolled out", "stdout_lines": ["Waiting for daemon set \"apiserver\" rollout to finish: 0 of 1 updated pods are available...", "daemon set \"apiserver\" successfully rolled out"]}

TASK [openshift_service_catalog : Wait for Controller Manager rollout success] ***
task path: /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/start.yml:14
Thursday 17 January 2019  13:23:19 +0000 (0:00:36.414)       0:16:11.738 ****** 
ok: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"attempts": 1, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/controller-manager"], "delta": "0:00:44.357499", "end": "2019-01-17 08:24:36.525204", "rc": 0, "start": "2019-01-17 08:23:52.167705", "stderr": "", "stderr_lines": [], "stdout": "Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...\ndaemon set \"controller-manager\" successfully rolled out", "stdout_lines": ["Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...", "daemon set \"controller-manager\" successfully rolled out"]}

TASK [ansible_service_broker : Create the Broker resource in the catalog] ******
task path: /usr/share/ansible/openshift-ansible/roles/ansible_service_broker/tasks/install.yml:226
Thursday 17 January 2019  13:24:46 +0000 (0:00:01.138)       0:17:39.139 ****** 
changed: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"changed": true, "results": {"cmd": "/usr/bin/oc get ClusterServiceBroker ansible-service-broker -o json -n default", "results": [{"apiVersion": "servicecatalog.k8s.io/v1beta1", "kind": "ClusterServiceBroker", "metadata": {"creationTimestamp": "2019-01-17T11:59:39Z", "generation": 1, "name": "ansible-service-broker", "resourceVersion": "16297", "selfLink": "/apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers/ansible-service-broker", "uid": "5e2d67fb-1a4f-11e9-b3ee-0a580a800006"}, "spec": {"authInfo": {"bearer": {"secretRef": {"name": "asb-client", "namespace": "openshift-ansible-service-broker"}}}, "caBundle": "xxx", "relistBehavior": "Duration", "relistRequests": 0, "url": "https://asb.openshift-ansible-service-broker.svc:1338/osb"}, "status": {"conditions": [{"lastTransitionTime": "2019-01-17T12:00:06Z", "message": "Successfully fetched catalog entries from broker.", "reason": "FetchedCatalog", "status": "True", "type": "Ready"}], "lastCatalogRetrievalTime": "2019-01-17T13:21:03Z", "reconciledGeneration": 1}}], "returncode": 0}, "state": "present"}

Comment 5 errata-xmlrpc 2019-01-30 15:13:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0206