Bug 1659198

Summary: Running the upgrade_control_plane playbook - (3.10.47-1 to 3.10.72-1) results in errors when trying to Upgrade Service Catalog
Product: OpenShift Container Platform Reporter: Mitchell Rollinson <mirollin>
Component: Service CatalogAssignee: Jay Boyd <jaboyd>
Status: CLOSED DUPLICATE QA Contact: Jian Zhang <jiazha>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: aos-bugs, jokerman, knakayam, ktadimar, mmccomas, rbost
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-14 04:20:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mitchell Rollinson 2018-12-13 19:40:33 UTC
Description of problem:

Running the upgrade_control_plane playbook - (3.10.47-1 to 3.10.72-1) results in errors when trying to Upgrade Service Catalog

Version-Release number of selected component (if applicable):

atomic-openshift-3.10.72-1.git.0.3cb2fdc.el7.x86_64 
atomic-openshift-clients-3.10.72-1.git.0.3cb2fdc.el7.x86_64 
atomic-openshift-docker-excluder-3.10.72-1.git.0.3cb2fdc.el7.noarch
atomic-openshift-excluder-3.10.72-1.git.0.3cb2fdc.el7.noarch 
atomic-openshift-hyperkube-3.10.72-1.git.0.3cb2fdc.el7.x86_64
atomic-openshift-node-3.10.72-1.git.0.3cb2fdc.el7.x86_64
openshift-ansible-3.10.47-1.git.0.95bc2d2.el7_5.noarch
openshift-ansible-docs-3.10.47-1.git.0.95bc2d2.el7_5.noarch
openshift-ansible-playbooks-3.10.47-1.git.0.95bc2d2.el7_5.noarch
openshift-ansible-roles-3.10.47-1.git.0.95bc2d2.el7_5.noarch

RHEL 7.5


How reproducible:

Attempt to run the respective upgrade playbook.  (3.10.47-1 to 3.10.72-1)

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Playbook logs

Hosts:    #######.#######.####.###
     Play:     Upgrade Service Catalog
     Task:     Create the Broker resource in the catalog
     Message:  {u'cmd': u'/usr/bin/oc create -f /tmp/brokerout-kxQhIH -n default', u'returncode': 1, u'results': {}, u'stderr': u'Error from server (AlreadyExists): error when creating "/tmp/brokerout-kxQhIH": clusterservicebrokers.servicecatalog.k8s.io "ansible-service-broker" already exists\n', u'stdout': u''}


TASK [ansible_service_broker : Create the Broker resource in the catalog] *************************************************************************************************************************************
fatal: [ #######.#######.####.###]: FAILED! => {"changed": false, "failed": true, "msg": {"cmd": "/usr/bin/oc create -f /tmp/brokerout-_WH2A5 -n default", "results": {}, "returncode": 1, "stderr": "Error from server (ServiceUnavailable): error when creating \"/tmp/brokerout-_WH2A5\": the server is currently unable to handle the request (post clusterservicebrokers.servicecatalog.k8s.io)\n", "stdout": ""}}
        to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade_control_plane.retry


Similar bugs have been reported.

 https://bugzilla.redhat.com/show_bug.cgi?id=1628881
 https://bugzilla.redhat.com/show_bug.cgi?id=1523625
 https://bugzilla.redhat.com/show_bug.cgi?id=1578300

Comment 2 Kenjiro Nakayama 2018-12-14 02:36:34 UTC
Not sure why the message has '"ansible-service-broker" already exists', but below error would be dup of bz#1658018 and bz#1656925

TASK [ansible_service_broker : Create the Broker resource in the catalog] *************************************************************************************************************************************
fatal: [ #######.#######.####.###]: FAILED! => {"changed": false, "failed": true, "msg": {"cmd": "/usr/bin/oc create -f /tmp/brokerout-_WH2A5 -n default", "results": {}, "returncode": 1, "stderr": "Error from server (ServiceUnavailable): error when creating \"/tmp/brokerout-_WH2A5\": the server is currently unable to handle the request (post clusterservicebrokers.servicecatalog.k8s.io)\n", "stdout": ""}}
        to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade_control_plane.retry

You can avoid the issue by removing livness&readiness probe from the template in the playbook.

Comment 3 Mitchell Rollinson 2018-12-14 04:20:06 UTC

*** This bug has been marked as a duplicate of bug 1658018 ***

Comment 4 Jay Boyd 2018-12-14 10:07:14 UTC

*** This bug has been marked as a duplicate of bug 1656925 ***