Bug 1561485

Summary: playbook to manage Service Broker fails if namespace no longer exists
Product: OpenShift Container Platform Reporter: Vladislav Walek <vwalek>
Component: InstallerAssignee: Fabian von Feilitzsch <fabian>
Status: CLOSED ERRATA QA Contact: Zihan Tang <zitang>
Severity: high Docs Contact:
Priority: medium    
Version: 3.7.0CC: aos-bugs, chezhang, dzhukous, jiazha, jokerman, mmccomas, pmorie, rleite, sdodson, vwalek, zhsun, zitang
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: When openshift-ansible deletes resources, it requires the namespace they are in to exist Consequence: If the broker is not deployed, removing it fails (because the namespace doesn't exist) Fix: If the namespace of a resource doesn't exist, neither does that resource. Count those as successful deletions. Result: Removal of ASB is now idempotent.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:11:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1523294    
Bug Blocks:    

Description Vladislav Walek 2018-03-28 12:27:06 UTC
Description of problem:

Service catalog playbook (/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/service-catalog.yml) fails if the namespace no longer exists.

TASK [ansible_service_broker : remove ansible-service-broker service] **************************************************************************************************************************************
fatal: [master1]: FAILED! => {"changed": false, "msg": {"cmd": "/usr/local/bin/oc get service asb -o json -n openshift-ansible-service-broker", "results": [{}], "returncode": 1, "stderr": "Error from server (NotFound): namespaces \"openshift-ansible-service-broker\" not found\n", "stdout": ""}}
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/service-catalog.retry

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
TASK [ansible_service_broker : remove ansible-service-broker service] **************************************************************************************************************************************
fatal: [master1]: FAILED! => {"changed": false, "msg": {"cmd": "/usr/local/bin/oc get service asb -o json -n openshift-ansible-service-broker", "results": [{}], "returncode": 1, "stderr": "Error from server (NotFound): namespaces \"openshift-ansible-service-broker\" not found\n", "stdout": ""}}
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/service-catalog.retry

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Vladislav Walek 2018-03-28 12:54:44 UTC
-- rpm -q openshift-ansible
openshift-ansible-3.7.23-1.git.0.bc406aa.el7.noarch

-- rpm -q ansible
ansible-2.4.2.0-2.el7.noarch

-- ansible --version
ansible 2.4.2.0
  config file = /root/.ansible.cfg
  configured module search path = [u'/usr/share/ansible']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

Comment 4 Jeff Peeler 2018-04-19 17:13:58 UTC
This might need to be reset to the installer component again, I just didn't know how to otherwise get this triaged to the right group.

Comment 5 Fabian von Feilitzsch 2018-04-20 19:21:32 UTC
This looks like a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1523294

This can be worked around by setting 

    ansible_service_broker_remove=false

in the inventory if the broker is not deployed. I'm not sure this is severe enough for a backport, so I think we should mention the workaround and close this in favor of the 3.10 aligned bug. Is this acceptable?

Comment 7 Fabian von Feilitzsch 2018-04-23 16:25:54 UTC
Scott, can we align this to 3.9.z?

Comment 8 Scott Dodson 2018-04-23 19:43:39 UTC
Yes, 3.9.z is fine after we confirm the fix on master.

Comment 9 Fabian von Feilitzsch 2018-05-04 16:48:23 UTC
https://github.com/openshift/openshift-ansible/pull/8270

Comment 10 openshift-github-bot 2018-05-16 19:36:09 UTC
Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/ae7d8b890ccd4244b4a0a9d7783b63be4658b973
Bug 1561485- get now returns empty instead of error when the namespace is missing

https://github.com/openshift/openshift-ansible/commit/2681fa47c5c9579c9a552ca48a32b8d7988ed258
Merge pull request #8270 from fabianvf/bz1561485

Bug 1561485- get now returns empty instead of error when the namespace is missing

Comment 11 Scott Dodson 2018-05-17 12:48:28 UTC
https://github.com/openshift/openshift-ansible/pull/8396 3.9 backport

Comment 12 Zihan Tang 2018-05-28 07:24:20 UTC
image is ready ,change it to ON_QA

Comment 13 Zihan Tang 2018-05-28 07:24:58 UTC
verified in v3.10 and v3.9
openshift-ansible-3.10.0-0.53.0  
openshift-ansible-3.9.30-1

TASK [ansible_service_broker : remove ansible-service-broker service] *************
Monday 28 May 2018  06:49:25 +0000 (0:00:01.576)       0:01:40.802 ************ 
ok: [*****.com]

Comment 14 Raul Leite 2018-07-11 19:34:10 UTC
I'm facing performance issues in client with openshift 3.9.27, deployed in 5 nodes (1x master, 1x infra, 3x nodes - rhel 7.5 x86_64).
When we try to remove any project, pod, containers some objects stuck in status "Terminating" even if try force remove or reboot node, nothing changed.

I'm facing the same issue in customer, case:
- https://access.redhat.com/support/cases/#/case/02139285

Comment 16 errata-xmlrpc 2018-07-30 19:11:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816