Bug 1667194

Description of problem:
After a successful openshift deployment 3.11 with kuryr we are trying to uninstall the openshift installation running the uninstall playbook which fails eventually on deleting the load balancers.
After an investigation by Avishai, see error http://pastebin.test.redhat.com/697603, the flow is:
DNS entries are being deleted
we try to get a list of kuryr services to delete [1]  
the oc command fails and skipped since there is no record for console.openshift.example.com
We try to delete the LB [2] but the list is empty so we do nothing actually

Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-3.11.67-1.git.0.ccfb323.el7.noarch
rpm -q ansible
ansible-2.5.14-1.el7ae.noarch
ansible --version
[cloud-user@ansible-host-0 ~]$ ansible --version

ansible 2.5.14
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:
100%

Steps to Reproduce:
1. Deploy ocp 3.11
2. Run uninstall playbook
3.

Actual results:
TASK [cleanup : Run OpenShift uninstall playbook] ***************************************************************************************************************************
fatal: [ansible_host-0]: FAILED! => {"changed": true, "cmd": "export ANSIBLE_HOST_KEY_CHECKING=False\n export ANSIBLE_RETRY_FILES_ENABLED=False\n export OPENSHIFT_CLUSTER=\"openshift.example.com\"\n\n source ~/shiftstackrc\n ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/uninstall.yml", "delta": "0:02:06.558806", "end": "2018-12-25 05:06:52.865319", "msg": "non-zero return code", "rc": 2, "start": "2018-12-25 05:04:46.306513", "stderr": " [ERROR]: /usr/lib/python2.7/site-packages/requests/__init__.py:91:\nRequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a\nsupported version!   RequestsDependencyWarning)\n [WARNING]: Unable to parse /home/cloud-user/inventory as an inventory source\n [WARNING]: Found both group and host with same name: localhost\n/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a supported version!\n  RequestsDependencyWarning)", "stderr_lines": [" [ERROR]: /usr/lib/python2.7/site-packages/requests/__init__.py:91:", "RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a", "supported version!   RequestsDependencyWarning)", " [WARNING]: Unable to parse /home/cloud-user/inventory as an inventory source", " [WARNING]: Found both group and host with same name: localhost", "/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a supported version!", "  RequestsDependencyWarning)"], "stdout": "\nPLAY [Evaluate the OpenStack groups] *******************************************\n\nTASK [Create a brand new masters group (no scaling)] ***************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nTASK [Create pre-existing masters group] ***************************************\nskipping: [localhost]\n\nTASK [Create new_masters group] ************************************************\n\nTASK [Create a brand new etcd group (no scaling)] ******************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nTASK [Create pre-existing etcd group] ******************************************\nskipping: [localhost]\n\nTASK [Create new_etcd group] ***************************************************\n\nTASK [Create a brand new nodes group (no scaling)] *****************************\nok: [localhost] => (item=app-node-0.openshift.example.com)\nok: [localhost] => (item=app-node-1.openshift.example.com)\nok: [localhost] => (item=master-0.openshift.example.com)\nok: [localhost] => (item=infra-node-0.openshift.example.com)\n\nTASK [Create pre-existing nodes group] *****************************************\nskipping: [localhost]\n\nTASK [Create new_nodes group] **************************************************\n\nTASK [Create glusterfs group] **************************************************\n\nPLAY [Populate config host groups] *********************************************\n\nTASK [Load group name mapping variables] ***************************************\nok: [localhost]\n\nTASK [Evaluate groups - g_nfs_hosts is single host] ****************************\nskipping: [localhost]\n\nTASK [Evaluate oo_all_hosts] ***************************************************\nok: [localhost] => (item=master-0.openshift.example.com)\nok: [localhost] => (item=app-node-0.openshift.example.com)\nok: [localhost] => (item=app-node-1.openshift.example.com)\nok: [localhost] => (item=infra-node-0.openshift.example.com)\n\nTASK [Evaluate oo_masters] *****************************************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nTASK [Evaluate oo_first_master] ************************************************\nok: [localhost]\n\nTASK [Evaluate oo_new_etcd_to_config] ******************************************\n\nTASK [Evaluate oo_masters_to_config] *******************************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nTASK [Evaluate oo_etcd_to_config] **********************************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nTASK [Evaluate oo_first_etcd] **************************************************\nok: [localhost]\n\nTASK [Evaluate oo_etcd_hosts_to_upgrade] ***************************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nTASK [Evaluate oo_etcd_hosts_to_backup] ****************************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nTASK [Evaluate oo_nodes_to_config] *********************************************\nok: [localhost] => (item=app-node-0.openshift.example.com)\nok: [localhost] => (item=app-node-1.openshift.example.com)\nok: [localhost] => (item=master-0.openshift.example.com)\nok: [localhost] => (item=infra-node-0.openshift.example.com)\n\nTASK [Evaluate oo_lb_to_config] ************************************************\n\nTASK [Evaluate oo_nfs_to_config] ***********************************************\n\nTASK [Evaluate oo_glusterfs_to_config] *****************************************\n\nTASK [Evaluate oo_etcd_to_migrate] *********************************************\nok: [localhost] => (item=master-0.openshift.example.com)\n\nPLAY [Unsubscribe RHEL instances] **********************************************\n\nTASK [Gathering Facts] *********************************************************\nok: [app-node-0.openshift.example.com]\nok: [app-node-1.openshift.example.com]\nok: [master-0.openshift.example.com]\nok: [infra-node-0.openshift.example.com]\n\nTASK [rhel_unsubscribe : Remove RedHat subscriptions] **************************\nskipping: [master-0.openshift.example.com]\nskipping: [app-node-0.openshift.example.com]\nskipping: [app-node-1.openshift.example.com]\nskipping: [infra-node-0.openshift.example.com]\n\nPLAY [Clean DNS entries] *******************************************************\n\nTASK [Gathering Facts] *********************************************************\nok: [localhost]\n\nTASK [openshift_openstack : Add private node records] **************************\nchanged: [localhost] => (item=app-node-0.openshift.example.com)\nchanged: [localhost] => (item=app-node-1.openshift.example.com)\nchanged: [localhost] => (item=master-0.openshift.example.com)\nchanged: [localhost] => (item=infra-node-0.openshift.example.com)\n\nTASK [openshift_openstack : Add public node records] ***************************\nchanged: [localhost] => (item=app-node-0.openshift.example.com)\nchanged: [localhost] => (item=app-node-1.openshift.example.com)\nchanged: [localhost] => (item=master-0.openshift.example.com)\nchanged: [localhost] => (item=infra-node-0.openshift.example.com)\n\nTASK [openshift_openstack : Add public wildcard record] ************************\nchanged: [localhost]\n\nTASK [openshift_openstack : Add public API record] *****************************\nchanged: [localhost]\n\nPLAY [Remove OpenStack resources] **********************************************\n\nTASK [Gathering Facts] *********************************************************\nok: [localhost]\n\nTASK [retrieve cluster name from the environment if present] *******************\nok: [localhost]\n\nTASK [openshift_openstack : Get kuryr services] ********************************\nfatal: [localhost -> 10.46.22.236]: FAILED! => {\"changed\": false, \"msg\": {\"cmd\": \"/bin/oc get endpoints -o json --all-namespaces\", \"results\": [{}], \"returncode\": 1, \"stderr\": \"Unable to connect to the server: dial tcp: lookup console.openshift.example.com on 192.168.99.6:53: no such host\\n\", \"stdout\": \"\"}}\n...ignoring\n\nTASK [openshift_openstack : Delete kuryr LoadBalancers (cascading)] ************\n\nTASK [openshift_openstack : Remove trunk subports] *****************************\nchanged: [localhost] => (item=app-node-0.openshift.example.com)\nchanged: [localhost] => (item=app-node-1.openshift.example.com)\nchanged: [localhost] => (item=master-0.openshift.example.com)\nchanged: [localhost] => (item=infra-node-0.openshift.example.com)\n\nTASK [openshift_openstack : Get kuryr net CRDs] ********************************\nfatal: [localhost -> 10.46.22.236]: FAILED! => {\"changed\": false, \"msg\": {\"cmd\": \"/bin/oc get kuryrnets -o json --all-namespaces\", \"results\": [{}], \"returncode\": 1, \"stderr\": \"Unable to connect to the server: dial tcp: lookup console.openshift.example.com on 192.168.99.6:53: no such host\\n\", \"stdout\": \"\"}}\n...ignoring\n\nTASK [openshift_openstack : Delete namespaced resources] ***********************\n\nTASK [openshift_openstack : Delete the Stack] **********************************\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Failed to delete stack openshift.example.com: Resource DELETE failed: BadRequest: resources.pod_subnet_pool: Unable to delete subnet pool: Subnet pool has existing allocations.\\nNeutron server returns request_ids: ['req-a9d48a69-368a-43e9-8862-ced5f3dbc2b9']\"}\n\nPLAY RECAP *********************************************************************\napp-node-0.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   \napp-node-1.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   \ninfra-node-0.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   \nlocalhost                  : ok=24   changed=5    unreachable=0    failed=1   \nmaster-0.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   ", "stdout_lines": ["", "PLAY [Evaluate the OpenStack groups] *******************************************", "", "TASK [Create a brand new masters group (no scaling)] ***************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "TASK [Create pre-existing masters group] ***************************************", "skipping: [localhost]", "", "TASK [Create new_masters group] ************************************************", "", "TASK [Create a brand new etcd group (no scaling)] ******************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "TASK [Create pre-existing etcd group] ******************************************", "skipping: [localhost]", "", "TASK [Create new_etcd group] ***************************************************", "", "TASK [Create a brand new nodes group (no scaling)] *****************************", "ok: [localhost] => (item=app-node-0.openshift.example.com)", "ok: [localhost] => (item=app-node-1.openshift.example.com)", "ok: [localhost] => (item=master-0.openshift.example.com)", "ok: [localhost] => (item=infra-node-0.openshift.example.com)", "", "TASK [Create pre-existing nodes group] *****************************************", "skipping: [localhost]", "", "TASK [Create new_nodes group] **************************************************", "", "TASK [Create glusterfs group] **************************************************", "", "PLAY [Populate config host groups] *********************************************", "", "TASK [Load group name mapping variables] ***************************************", "ok: [localhost]", "", "TASK [Evaluate groups - g_nfs_hosts is single host] ****************************", "skipping: [localhost]", "", "TASK [Evaluate oo_all_hosts] ***************************************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "ok: [localhost] => (item=app-node-0.openshift.example.com)", "ok: [localhost] => (item=app-node-1.openshift.example.com)", "ok: [localhost] => (item=infra-node-0.openshift.example.com)", "", "TASK [Evaluate oo_masters] *****************************************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "TASK [Evaluate oo_first_master] ************************************************", "ok: [localhost]", "", "TASK [Evaluate oo_new_etcd_to_config] ******************************************", "", "TASK [Evaluate oo_masters_to_config] *******************************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "TASK [Evaluate oo_etcd_to_config] **********************************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "TASK [Evaluate oo_first_etcd] **************************************************", "ok: [localhost]", "", "TASK [Evaluate oo_etcd_hosts_to_upgrade] ***************************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "TASK [Evaluate oo_etcd_hosts_to_backup] ****************************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "TASK [Evaluate oo_nodes_to_config] *********************************************", "ok: [localhost] => (item=app-node-0.openshift.example.com)", "ok: [localhost] => (item=app-node-1.openshift.example.com)", "ok: [localhost] => (item=master-0.openshift.example.com)", "ok: [localhost] => (item=infra-node-0.openshift.example.com)", "", "TASK [Evaluate oo_lb_to_config] ************************************************", "", "TASK [Evaluate oo_nfs_to_config] ***********************************************", "", "TASK [Evaluate oo_glusterfs_to_config] *****************************************", "", "TASK [Evaluate oo_etcd_to_migrate] *********************************************", "ok: [localhost] => (item=master-0.openshift.example.com)", "", "PLAY [Unsubscribe RHEL instances] **********************************************", "", "TASK [Gathering Facts] *********************************************************", "ok: [app-node-0.openshift.example.com]", "ok: [app-node-1.openshift.example.com]", "ok: [master-0.openshift.example.com]", "ok: [infra-node-0.openshift.example.com]", "", "TASK [rhel_unsubscribe : Remove RedHat subscriptions] **************************", "skipping: [master-0.openshift.example.com]", "skipping: [app-node-0.openshift.example.com]", "skipping: [app-node-1.openshift.example.com]", "skipping: [infra-node-0.openshift.example.com]", "", "PLAY [Clean DNS entries] *******************************************************", "", "TASK [Gathering Facts] *********************************************************", "ok: [localhost]", "", "TASK [openshift_openstack : Add private node records] **************************", "changed: [localhost] => (item=app-node-0.openshift.example.com)", "changed: [localhost] => (item=app-node-1.openshift.example.com)", "changed: [localhost] => (item=master-0.openshift.example.com)", "changed: [localhost] => (item=infra-node-0.openshift.example.com)", "", "TASK [openshift_openstack : Add public node records] ***************************", "changed: [localhost] => (item=app-node-0.openshift.example.com)", "changed: [localhost] => (item=app-node-1.openshift.example.com)", "changed: [localhost] => (item=master-0.openshift.example.com)", "changed: [localhost] => (item=infra-node-0.openshift.example.com)", "", "TASK [openshift_openstack : Add public wildcard record] ************************", "changed: [localhost]", "", "TASK [openshift_openstack : Add public API record] *****************************", "changed: [localhost]", "", "PLAY [Remove OpenStack resources] **********************************************", "", "TASK [Gathering Facts] *********************************************************", "ok: [localhost]", "", "TASK [retrieve cluster name from the environment if present] *******************", "ok: [localhost]", "", "TASK [openshift_openstack : Get kuryr services] ********************************", "fatal: [localhost -> 10.46.22.236]: FAILED! => {\"changed\": false, \"msg\": {\"cmd\": \"/bin/oc get endpoints -o json --all-namespaces\", \"results\": [{}], \"returncode\": 1, \"stderr\": \"Unable to connect to the server: dial tcp: lookup console.openshift.example.com on 192.168.99.6:53: no such host\\n\", \"stdout\": \"\"}}", "...ignoring", "", "TASK [openshift_openstack : Delete kuryr LoadBalancers (cascading)] ************", "", "TASK [openshift_openstack : Remove trunk subports] *****************************", "changed: [localhost] => (item=app-node-0.openshift.example.com)", "changed: [localhost] => (item=app-node-1.openshift.example.com)", "changed: [localhost] => (item=master-0.openshift.example.com)", "changed: [localhost] => (item=infra-node-0.openshift.example.com)", "", "TASK [openshift_openstack : Get kuryr net CRDs] ********************************", "fatal: [localhost -> 10.46.22.236]: FAILED! => {\"changed\": false, \"msg\": {\"cmd\": \"/bin/oc get kuryrnets -o json --all-namespaces\", \"results\": [{}], \"returncode\": 1, \"stderr\": \"Unable to connect to the server: dial tcp: lookup console.openshift.example.com on 192.168.99.6:53: no such host\\n\", \"stdout\": \"\"}}", "...ignoring", "", "TASK [openshift_openstack : Delete namespaced resources] ***********************", "", "TASK [openshift_openstack : Delete the Stack] **********************************", "fatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Failed to delete stack openshift.example.com: Resource DELETE failed: BadRequest: resources.pod_subnet_pool: Unable to delete subnet pool: Subnet pool has existing allocations.\\nNeutron server returns request_ids: ['req-a9d48a69-368a-43e9-8862-ced5f3dbc2b9']\"}", "", "PLAY RECAP *********************************************************************", "app-node-0.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   ", "app-node-1.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   ", "infra-node-0.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   ", "localhost                  : ok=24   changed=5    unreachable=0    failed=1   ", "master-0.openshift.example.com : ok=1    changed=0    unreachable=0    failed=0   "]}

Expected results:
Uninstall pass
Additional info:
Please attach logs from ansible-playbook with the -vvv flag