Bug 1661569 - Upgrade Fails at TASK [ansible_service_broker : Create the Broker resource in the catalog]
Summary: Upgrade Fails at TASK [ansible_service_broker : Create the Broker resource in...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Catalog
Version: 3.11.0
Hardware: All
OS: Linux
high
high
Target Milestone: ---
: 3.11.z
Assignee: Jay Boyd
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1656925
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-21 15:34 UTC by Jay Boyd
Modified: 2022-03-13 16:36 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The 3.11.43 update added health and liveness probes for the Service Catalog pods. Install was not waiting for the update rollout to finish before proceeding to update Ansible Service Broker. Because of timing, the Service Catalog pods were unavailable when the Broker attempted to register. Consequence: Ansible Service Broker update failed with an error indicating "the server is currently unable to handle the request (post clusterservicebrokers.servicecatalog.k8s.io)" Fix: Installation was updated to wait for the Service Catalog update rollout to finish before proceeding with installing Ansible Service Broker.
Clone Of: 1656925
Environment:
Last Closed: 2019-01-30 15:13:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0206 0 None None None 2019-01-30 15:13:25 UTC

Comment 1 Jay Boyd 2018-12-21 15:35:21 UTC
fixed by https://github.com/openshift/openshift-ansible/pull/10658  in 3.11.z.

Comment 3 Jian Zhang 2019-01-17 13:58:27 UTC
LGTM, verify it. Details as below:

1, original OCP cluster: 3.11.20
[root@ip-172-18-7-221 ~]# oc version
oc v3.11.20
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-7-221.ec2.internal:8443
openshift v3.11.20
kubernetes v1.11.0+d4cacc0

[root@ip-172-18-7-221 ~]# oc get pods -n kube-service-catalog 
NAME                       READY     STATUS    RESTARTS   AGE
apiserver-zwfqr            1/1       Running   0          1h
controller-manager-ngwzw   1/1       Running   2          1h

[root@ip-172-18-7-221 ~]# oc get pods -n kube-service-catalog apiserver-zwfqr -o yaml |grep image
    image: registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11.20

[root@ip-172-18-7-221 ~]# docker run --rm -ti registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11.20 service-catalog --version
v3.11.20;Upstream:v0.1.31

2, Upgrade it to the OCP 3.11.69 with the release-3.11 of the openshift-ansible.
[root@ip-172-18-7-221 ~]# oc version
oc v3.11.69
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-7-221.ec2.internal:8443
openshift v3.11.69
kubernetes v1.11.0+d4cacc0

[root@ip-172-18-7-221 ~]# oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
apiserver-xv5zn            1/1       Running   8          30m
controller-manager-z4lv4   1/1       Running   8          30m

[root@ip-172-18-7-221 ~]# oc get pods -o yaml |grep image
      image: registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11

[root@ip-172-18-7-221 ~]# docker run --rm -ti registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11 service-catalog --version
v3.11.69;Upstream:v0.1.35

correlating logs:
TASK [openshift_service_catalog : Wait for API Server rollout success] *********
task path: /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/start.yml:2
Thursday 17 January 2019  13:22:42 +0000 (0:00:00.279)       0:15:35.323 ****** 
ok: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"attempts": 1, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/apiserver"], "delta": "0:00:36.021360", "end": "2019-01-17 08:23:51.766271", "rc": 0, "start": "2019-01-17 08:23:15.744911", "stderr": "", "stderr_lines": [], "stdout": "Waiting for daemon set \"apiserver\" rollout to finish: 0 of 1 updated pods are available...\ndaemon set \"apiserver\" successfully rolled out", "stdout_lines": ["Waiting for daemon set \"apiserver\" rollout to finish: 0 of 1 updated pods are available...", "daemon set \"apiserver\" successfully rolled out"]}

TASK [openshift_service_catalog : Wait for Controller Manager rollout success] ***
task path: /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/start.yml:14
Thursday 17 January 2019  13:23:19 +0000 (0:00:36.414)       0:16:11.738 ****** 
ok: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"attempts": 1, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/controller-manager"], "delta": "0:00:44.357499", "end": "2019-01-17 08:24:36.525204", "rc": 0, "start": "2019-01-17 08:23:52.167705", "stderr": "", "stderr_lines": [], "stdout": "Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...\ndaemon set \"controller-manager\" successfully rolled out", "stdout_lines": ["Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...", "daemon set \"controller-manager\" successfully rolled out"]}

TASK [ansible_service_broker : Create the Broker resource in the catalog] ******
task path: /usr/share/ansible/openshift-ansible/roles/ansible_service_broker/tasks/install.yml:226
Thursday 17 January 2019  13:24:46 +0000 (0:00:01.138)       0:17:39.139 ****** 
changed: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"changed": true, "results": {"cmd": "/usr/bin/oc get ClusterServiceBroker ansible-service-broker -o json -n default", "results": [{"apiVersion": "servicecatalog.k8s.io/v1beta1", "kind": "ClusterServiceBroker", "metadata": {"creationTimestamp": "2019-01-17T11:59:39Z", "generation": 1, "name": "ansible-service-broker", "resourceVersion": "16297", "selfLink": "/apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers/ansible-service-broker", "uid": "5e2d67fb-1a4f-11e9-b3ee-0a580a800006"}, "spec": {"authInfo": {"bearer": {"secretRef": {"name": "asb-client", "namespace": "openshift-ansible-service-broker"}}}, "caBundle": "xxx", "relistBehavior": "Duration", "relistRequests": 0, "url": "https://asb.openshift-ansible-service-broker.svc:1338/osb"}, "status": {"conditions": [{"lastTransitionTime": "2019-01-17T12:00:06Z", "message": "Successfully fetched catalog entries from broker.", "reason": "FetchedCatalog", "status": "True", "type": "Ready"}], "lastCatalogRetrievalTime": "2019-01-17T13:21:03Z", "reconciledGeneration": 1}}], "returncode": 0}, "state": "present"}

Comment 5 errata-xmlrpc 2019-01-30 15:13:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0206


Note You need to log in before you can comment on or make changes to this bug.