Bug 1661569
Summary: | Upgrade Fails at TASK [ansible_service_broker : Create the Broker resource in the catalog] | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Jay Boyd <jaboyd> |
Component: | Service Catalog | Assignee: | Jay Boyd <jaboyd> |
Status: | CLOSED ERRATA | QA Contact: | Jian Zhang <jiazha> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 3.11.0 | CC: | pkanthal |
Target Milestone: | --- | ||
Target Release: | 3.11.z | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: |
Cause: The 3.11.43 update added health and liveness probes for the Service Catalog pods. Install was not waiting for the update rollout to finish before proceeding to update Ansible Service Broker. Because of timing, the Service Catalog pods were unavailable when the Broker attempted to register.
Consequence: Ansible Service Broker update failed with an error indicating "the server is currently unable to handle the request (post clusterservicebrokers.servicecatalog.k8s.io)"
Fix: Installation was updated to wait for the Service Catalog update rollout to finish before proceeding with installing Ansible Service Broker.
|
Story Points: | --- |
Clone Of: | 1656925 | Environment: | |
Last Closed: | 2019-01-30 15:13:18 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1656925 | ||
Bug Blocks: |
Comment 1
Jay Boyd
2018-12-21 15:35:21 UTC
LGTM, verify it. Details as below: 1, original OCP cluster: 3.11.20 [root@ip-172-18-7-221 ~]# oc version oc v3.11.20 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-7-221.ec2.internal:8443 openshift v3.11.20 kubernetes v1.11.0+d4cacc0 [root@ip-172-18-7-221 ~]# oc get pods -n kube-service-catalog NAME READY STATUS RESTARTS AGE apiserver-zwfqr 1/1 Running 0 1h controller-manager-ngwzw 1/1 Running 2 1h [root@ip-172-18-7-221 ~]# oc get pods -n kube-service-catalog apiserver-zwfqr -o yaml |grep image image: registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11.20 [root@ip-172-18-7-221 ~]# docker run --rm -ti registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11.20 service-catalog --version v3.11.20;Upstream:v0.1.31 2, Upgrade it to the OCP 3.11.69 with the release-3.11 of the openshift-ansible. [root@ip-172-18-7-221 ~]# oc version oc v3.11.69 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-7-221.ec2.internal:8443 openshift v3.11.69 kubernetes v1.11.0+d4cacc0 [root@ip-172-18-7-221 ~]# oc get pods NAME READY STATUS RESTARTS AGE apiserver-xv5zn 1/1 Running 8 30m controller-manager-z4lv4 1/1 Running 8 30m [root@ip-172-18-7-221 ~]# oc get pods -o yaml |grep image image: registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11 [root@ip-172-18-7-221 ~]# docker run --rm -ti registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.11 service-catalog --version v3.11.69;Upstream:v0.1.35 correlating logs: TASK [openshift_service_catalog : Wait for API Server rollout success] ********* task path: /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/start.yml:2 Thursday 17 January 2019 13:22:42 +0000 (0:00:00.279) 0:15:35.323 ****** ok: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"attempts": 1, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/apiserver"], "delta": "0:00:36.021360", "end": "2019-01-17 08:23:51.766271", "rc": 0, "start": "2019-01-17 08:23:15.744911", "stderr": "", "stderr_lines": [], "stdout": "Waiting for daemon set \"apiserver\" rollout to finish: 0 of 1 updated pods are available...\ndaemon set \"apiserver\" successfully rolled out", "stdout_lines": ["Waiting for daemon set \"apiserver\" rollout to finish: 0 of 1 updated pods are available...", "daemon set \"apiserver\" successfully rolled out"]} TASK [openshift_service_catalog : Wait for Controller Manager rollout success] *** task path: /usr/share/ansible/openshift-ansible/roles/openshift_service_catalog/tasks/start.yml:14 Thursday 17 January 2019 13:23:19 +0000 (0:00:36.414) 0:16:11.738 ****** ok: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"attempts": 1, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/controller-manager"], "delta": "0:00:44.357499", "end": "2019-01-17 08:24:36.525204", "rc": 0, "start": "2019-01-17 08:23:52.167705", "stderr": "", "stderr_lines": [], "stdout": "Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...\ndaemon set \"controller-manager\" successfully rolled out", "stdout_lines": ["Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...", "daemon set \"controller-manager\" successfully rolled out"]} TASK [ansible_service_broker : Create the Broker resource in the catalog] ****** task path: /usr/share/ansible/openshift-ansible/roles/ansible_service_broker/tasks/install.yml:226 Thursday 17 January 2019 13:24:46 +0000 (0:00:01.138) 0:17:39.139 ****** changed: [ec2-52-90-225-248.compute-1.amazonaws.com] => {"changed": true, "results": {"cmd": "/usr/bin/oc get ClusterServiceBroker ansible-service-broker -o json -n default", "results": [{"apiVersion": "servicecatalog.k8s.io/v1beta1", "kind": "ClusterServiceBroker", "metadata": {"creationTimestamp": "2019-01-17T11:59:39Z", "generation": 1, "name": "ansible-service-broker", "resourceVersion": "16297", "selfLink": "/apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers/ansible-service-broker", "uid": "5e2d67fb-1a4f-11e9-b3ee-0a580a800006"}, "spec": {"authInfo": {"bearer": {"secretRef": {"name": "asb-client", "namespace": "openshift-ansible-service-broker"}}}, "caBundle": "xxx", "relistBehavior": "Duration", "relistRequests": 0, "url": "https://asb.openshift-ansible-service-broker.svc:1338/osb"}, "status": {"conditions": [{"lastTransitionTime": "2019-01-17T12:00:06Z", "message": "Successfully fetched catalog entries from broker.", "reason": "FetchedCatalog", "status": "True", "type": "Ready"}], "lastCatalogRetrievalTime": "2019-01-17T13:21:03Z", "reconciledGeneration": 1}}], "returncode": 0}, "state": "present"} Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0206 |