Bug 1611839

Summary: Requirement of Floating IPs for OpenShift Deployment exposes cluster unnecessarily to security vulnerabilities
Product: OpenShift Container Platform Reporter: rlopez
Component: InstallerAssignee: Tomas Sedovic <tsedovic>
Installer sub component: OpenShift on OpenStack QA Contact: weiwei jiang <wjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, bschmaus, eminguez, jialiu, jokerman, kschinck, ltomasbo, mmccomas, tzumainn, wjiang, wsun
Version: 3.10.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Feature: Control the assignment of Floating IP addresses for OpenStack cloud provisioning Reason: The playbook responsible for creating the OpenStack virtual servers would always associate a Floating IP address with each VM (i.e. each OpenShift node). This had two negative implications: 1. The OpenShift cluster size was limited by the number of Floating IPs available to the OpenStack user 2. All OpenShift nodes were directly accessible from the outside, increasing the potential attack surface. Result: We have introduced a role-based control over which nodes get Floating IPs and which don't. This is controlled by the following inventory variables: * openshift_openstack_master_floating_ip * openshift_openstack_infra_floating_ip * openshift_openstack_compute_floating_ip * openshift_openstack_load_balancer_floating_ip They are all boolean and all default to `true`. This allows for usecases such as: * Cluster where all the master and infra nodes have Floating IPs but the compute nodes don't * Cluster where none of the nodes have Floating IPs, but the load balancers do (so OpenShift is used through the load balancers, but none of the nodes are directly accessible) Note that if some of the nodes don't have Floating IPs (e.g. by setting `openshift_openstack_compute_floating_ip = false`), the openshift-ansible playbooks must be run from inside the node network. This is because a server without a floating IP is only accessible from the network it is in. A common way to do this is to pre-create the node network and subnet, create a "bastion" host in it and run Ansible there. $ openstack network create openshift $ openstack subnet create --subnet-range 192.168.0.0/24 --dns-nameserver 10.20.30.40 --network openshift openshift $ openstack router create openshift-router $ openstack router set --external-gateway public openshift-router $ openstack router add subnet openshift-router openshift $ openstack server create --wait --image RHEL7 --flavor m1.medium --key-name openshift --network openshift bastion $ openstack floating ip create public $ openstack server add floating ip bastion 172.24.4.10 $ ping 172.24.4.10 $ ssh cloud-user@172.24.4.10 Then install openshift-ansible and add the following to the inventory like so: (in inventory/group_vars/all.yml): openshift_openstack_node_network_name: openshift openshift_openstack_router_name: openshift-router openshift_openstack_node_subnet_name: openshift openshift_openstack_master_floating_ip: false openshift_openstack_infra_floating_ip: false openshift_openstack_compute_floating_ip: false openshift_openstack_load_balancer_floating_ip: false
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-16 07:46:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description rlopez 2018-08-02 21:11:54 UTC 
Description of problem:

Currently using the deployment process of deploying using the openshift-ansible
RPM requires the use of floating IPs for all master, infra, app nodes. By enabling floating IPs for all nodes you are unnecessarily exposing the openshift cluster to the outside world creating a security vulnerability. 

Aside from the security aspect, requiring environments to have floating IPs
available might not be feasible in certain customer environments. What if a
customer wants to deploy a very large environment? Are we going to force 
they have a public IP for each node?

Recommendation:

- Make the creation of the bastion host as part of the provision.yml. Since the
bastion host is an optional step today it is not part of the same internal network as created by the provision.yml thus the optional bastion host can only talk to the openshift cluster via the public subnet. The purpose of the bastion host is to act as a jumpbox that can be accessed remotely via SSH that can then be used to login to other instances via their private subnets. 

- Ensure that the installation steps and other important playbooks can run via the internal network i.e.
provision.yml, install.yml, uninstall.yml 

- Apply best practices to the the bastion host to ensure it is properly secured. I'd recommend discussing with someone within the RH security group to ensure their best practices are applied when the bastion host is created. 


NOTE: Simply removing the floating IP after the deployment of the openshift
cluster is not enough in the current state of openshift ansible deployment as 
any future updates or upgrades would require floating IPs to be re-added to the 
existing cluster.
Comment 1 Tomas Sedovic 2018-08-03 09:22:19 UTC 
We have been tracking the need for a more fine-grained floating IP address and general access control to the nodes. Thanks for creating this BZ.

It is be possible to deploy OpenShift without floating IP addresses today though.

Here's what you can do:
1. Create a private network + subnet
2. Launch a bastion VM inside that subnet
3. Connect to the bastion VM and secure it as you wish
4. Install Ansible, etc. and configure the openshift-ansible inventory inside the bastion VM
5. Make sure `openshift_openstack_external_network_name` and `openshift_openstack_private_network_name` are NOT set in inventory/group_vars/all.yml
6. Set `openshift_openstack_provider_network_name` to the network you created in step 1.
7. Run the openshift-ansible/playbooks/openstack/openshift-cluster/*.yml playbooks as usual

This will cause all OpenShift nodes to be put into the network and subnet you've created in step 1 (instead of creating a new one) and they will not have floating IP addresses assigned. But the bastion VM will be able to keep managing the cluster using the private addresses.

Alternatively, you could run the `provision-resources.yml` playbook from the outside, have it create the networks and nodes and then remove all the floating IPs, put a bastion inside that network and finish the deployment.

In either case, you must then provide a way to make the cluster accessible to your users by either assigning floating IPs to the master and infra nodes or by putting a proxy / load balancer in front of them.


Now, this is not ideal, because neither process is well documented and it is still rather involved, but it's there.

Introducing the bastion into openshift-ansible would make things nicer and less error-prone. It would however complicate the openshift-ansible playbooks.

We should definitely document this better and help at least partially automate this. I'm a bit wary of doing the full end to end bastion management, but we can definitely help at least with some bits.

Also of note is that as far as I understand it the 4.0 installer will not use floating IPs at all.
Comment 3 rlopez 2018-08-08 18:31:32 UTC 
Attempting to install with instructions above gets the following error:

TASK [openshift_openstack : validate the Heat template] *****************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openstack", "orchestration", "template", "validate", "-t", "/tmp/openshift-ansibleZMPVX0/stack.yaml"], "delta":
 "0:00:02.514699", "end": "2018-08-08 14:05:10.126005", "msg": "non-zero return code", "rc": 1, "start": "2018-08-08 14:05:07.611306", "stderr": "ERROR: The specified re
ference \"subnet\" (in api_lb.Properties.vip_subnet) is incorrect.", "stderr_lines": ["ERROR: The specified reference \"subnet\" (in api_lb.Properties.vip_subnet) is inc
orrect."], "stdout": "", "stdout_lines": []}

Speaking to Tzu-Mainn, he mentioned pointed to 2 sections of the heat_stack.yaml

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_openstack/templates/heat_stack.yaml.j2#L276

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_openstack/templates/heat_stack.yaml.j2#L196

He mentioned the above is the conditional that controls where that piece is generaeted. However, this would only work if you do not specify a provider network. 

In this particular, case we want a provider network to be specified so we can have bastion host and OCP instances that are provisioned on the same internal network.
Comment 4 Tzu-Mainn Chen 2018-09-05 16:11:02 UTC 
Tomas's excellent PR https://github.com/openshift/openshift-ansible/pull/9862 should also take care of this issue; as a reward I am re-assigning this BZ to him.
Comment 5 Luis Tomas Bolivar 2018-09-10 15:31:23 UTC 
Kuryr support was not fully working as there was some information missing on the kuryr-conf side -- such as ids of the created subnet and router. A new PR has been created to fix this (https://github.com/openshift/openshift-ansible/pull/9976)
Comment 7 N. Harrison Ripps 2018-09-21 20:15:08 UTC 
Per OCP program call on 21-SEP-2018 we are deferring Kuryr-related bugs to 3.11.z
Comment 8 Eduardo Minguez 2018-11-05 15:08:40 UTC 
Following the bz instructions, provision fails with 


TASK [openshift_openstack : validate the Heat template] ************************
Monday 05 November 2018  09:32:14 -0500 (0:00:00.680)       0:00:05.490 *******
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openstack", "orchestration", "template", "validate", "-t", "/tmp/openshift-ansible77m1pZ/stack.yaml"], "delta": "0:00:04.633918", "end": "2018-11-05 09:32:18.897989", "msg": "non-zero return code", "rc": 1, "start": "2018-11-05 09:32:14.264071", "stderr": "ERROR: The specified reference \"subnet\" (in interface.Properties.subnet_id) is incorrect.", "stderr_lines": ["ERROR: The specified reference \"subnet\" (in interface.Properties.subnet_id) is incorrect."], "stdout": "", "stdout_lines": []}

It seems the router points to an unknown subnet[1]:

    120   interface:
    121     type: OS::Neutron::RouterInterface
    122     properties:
    123       router_id: { get_resource: router }
    124       subnet_id: { get_resource: subnet }

So after adding the openshift_openstack_router_name variable in the inventory file, it works.

TL;DR.- I think the "openshift_openstack_router_name: openshift-router" variable should be added to the instructions.

[1] https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_openstack/templates/heat_stack.yaml.j2#L394-L411
Comment 9 Tzu-Mainn Chen 2018-11-05 15:40:50 UTC 
Odd, I don't remember having that issue. Could you attach your inventory files?
Comment 10 Eduardo Minguez 2018-11-05 16:07:34 UTC 
http://pastebin.test.redhat.com/666267
Comment 11 Tzu-Mainn Chen 2018-11-05 17:07:07 UTC 
Upon further checking, you're right that it's required; and it looks like it's already in the documentation:

https://github.com/openshift/openshift-ansible/commit/5c5c3dd12fc8d4fb184b335dfab2d6fdd58df492

It was added after the initial PR, so perhaps the note wasn't there in your build of 3.11, but it looks like this followup was backported to 3.11.
Comment 12 Eduardo Minguez 2018-11-05 17:48:23 UTC 
Can we modify the bugzilla doc just in case? Thanks!
Comment 13 Tzu-Mainn Chen 2018-11-05 18:11:28 UTC 
Ah, missed the bugzilla doc. Updated now, thanks for pointing it out!
Comment 15 weiwei jiang 2019-01-03 02:23:04 UTC 
Hi gcheresh:

Help check this, since this is not easy to check from OCP side.

Thanks
Comment 16 weiwei jiang 2019-01-04 10:52:11 UTC 
Tried to verify with openshift-ansible-3.11.65-1 but got the following errors, and all instances is up, so no idea if this is really working well:

TASK [Gather Cluster facts] *************************************************************************************************************************************************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:27
Friday 04 January 2019  05:43:05 -0500 (0:00:00.508)       0:01:43.310 ********
Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py
<172.16.122.6> ESTABLISH SSH CONNECTION FOR USER: openshift
<172.16.122.6> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.6 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-qewdvtlrdscngtbdbguzcqaynocpvvbd; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py
<172.16.122.59> ESTABLISH SSH CONNECTION FOR USER: openshift
<172.16.122.59> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.59 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-gblcgvqyfzihjjmgwbmijahwxrhliheb; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py                                                                                                                                                                                                                      
<172.16.122.51> ESTABLISH SSH CONNECTION FOR USER: openshift
<172.16.122.51> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.51 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-pkouygkyqoicawmknmsyruipxbjtwsne; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py                                                                                                                                                                                                                      
<172.16.122.44> ESTABLISH SSH CONNECTION FOR USER: openshift                       
<172.16.122.44> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.44 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-akvyzmuvxwuekqncspjehkxbiinukupt; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py        
<172.16.122.56> ESTABLISH SSH CONNECTION FOR USER: openshift
<172.16.122.56> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.56 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-kdaamngypidqdhespyighywnkzhgumjk; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''                                                                                                                                                                                                                                                                                                                                                                                                                 
Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py        
<172.16.122.41> ESTABLISH SSH CONNECTION FOR USER: openshift
<172.16.122.41> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.41 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-didnwroqitclkekgyfrhmwzxrydzasmd; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''                                                                                                                                                                                                                                                                                                                                                                                                                 
Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py        
<172.16.122.62> ESTABLISH SSH CONNECTION FOR USER: openshift
<172.16.122.62> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.62 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ifwwsgwdomnkfnwijzvqsvlykqjjhydb; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''                                                                                                                                                                                                                                                                                                                                                                                                                 
Escalation succeeded                                                                                           
Escalation succeeded                                        
Escalation succeeded                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
Escalation succeeded                                                                                           
Escalation succeeded                                        
Escalation succeeded                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
Escalation succeeded
<172.16.122.51> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n  File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1300, in <module>\n    main()\n  File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1287, in main\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n    provider_info.get(\'metadata\')\n  File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 310, in nor
malize_openstack_facts\n    if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n')
fatal: [infra-node-0.wjiang-ocp.example.com]: FAILED! => {
    "changed": false,                                                                                          
    "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n  File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1300, in <module>\n    main()\n  File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1287, in main\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n    provider_info.get('metadata')\n  File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 310,
 in normalize_openstack_facts\n    if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n",                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
    "module_stdout": "", 
    "msg": "MODULE FAILURE",                                                                                   
    "rc": 1                                                 
}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
<172.16.122.44> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n  File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1300, in <module>\n    main()\n  File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1287, in main\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n    provider_info.get(\'metadata\')\n  File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 310, in nor
malize_openstack_facts\n    if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n')
fatal: [etcd-0.wjiang-ocp.example.com]: FAILED! => {                                                                                              
    "changed": false,                                                                                                                             
    "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n  File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1300, in <module>\n    main()\n  File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1287, in main\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n    provider_info.get('metadata')\n  File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 310,
 in normalize_openstack_facts\n    if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1             
}                                                    
<172.16.122.56> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n  File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1300, in <module>\n    main()\n  File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1287, in main\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n    provider_info.get(\'metadata\')\n  File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 310, in nor
malize_openstack_facts\n    if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n')
fatal: [lb-0.wjiang-ocp.example.com]: FAILED! => {
    "changed": false, 
    "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n  File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1300, in <module>\n    main()\n  File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1287, in main\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n    provider_info.get('metadata')\n  File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 310,
 in normalize_openstack_facts\n    if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1             
}                                                    
<172.16.122.59> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n  File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1300, in <module>\n    main()\n  File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1287, in main\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n    provider_info.get(\'metadata\')\n  File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 310, in nor
malize_openstack_facts\n    if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n')
fatal: [app-node-0.wjiang-ocp.example.com]: FAILED! => {
    "changed": false,                       
    "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n  File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1300, in <module>\n    main()\n  File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1287, in main\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n    provider_info.get('metadata')\n  File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 310,
 in normalize_openstack_facts\n    if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1                                          
}        
<172.16.122.6> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n  File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1300, in <module>\n    main()\n  File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1287, in main\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n    provider_info.get(\'metadata\')\n  File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 310, in norm
alize_openstack_facts\n    if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n')
fatal: [master-1.wjiang-ocp.example.com]: FAILED! => {
    "changed": false,    
    "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n  File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1300, in <module>\n    main()\n  File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1287, in main\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n    provider_info.get('metadata')\n  File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 310,
 in normalize_openstack_facts\n    if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE",                         
    "rc": 1
}    
<172.16.122.41> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n  File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1300, in <module>\n    main()\n  File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1287, in main\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n    provider_info.get(\'metadata\')\n  File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 310, in nor
malize_openstack_facts\n    if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n')
fatal: [etcd-1.wjiang-ocp.example.com]: FAILED! => {
    "changed": false,    
    "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n  File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1300, in <module>\n    main()\n  File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1287, in main\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n    provider_info.get('metadata')\n  File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 310,
 in normalize_openstack_facts\n    if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE",                         
    "rc": 1
}    
<172.16.122.62> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n  File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1300, in <module>\n    main()\n  File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1287, in main\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n    provider_info.get(\'metadata\')\n  File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 310, in nor
malize_openstack_facts\n    if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n')
fatal: [master-0.wjiang-ocp.example.com]: FAILED! => {                                                                                            
    "changed": false,                   
    "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n  File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1300, in <module>\n    main()\n  File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1287, in main\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1039, in __init__\n    additive_facts_to_overwrite)\n  File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n    provider_facts = self.init_provider_facts()\n  File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n    provider_info.get('metadata')\n  File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n    facts = normalize_openstack_facts(metadata, facts)\n  File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 310,
 in normalize_openstack_facts\n    if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n",                                                                                                                                                                                                                                                                                                                                                
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
}        
     
PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************************************************************************                                                                                                                                                                                                    
app-node-0.wjiang-ocp.example.com : ok=18   changed=0    unreachable=0    failed=1                                                                
etcd-0.wjiang-ocp.example.com : ok=17   changed=0    unreachable=0    failed=1   
etcd-1.wjiang-ocp.example.com : ok=17   changed=0    unreachable=0    failed=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
infra-node-0.wjiang-ocp.example.com : ok=17   changed=0    unreachable=0    failed=1   
lb-0.wjiang-ocp.example.com : ok=17   changed=0    unreachable=0    failed=1   
localhost                  : ok=61   changed=14   unreachable=0    failed=0                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
master-0.wjiang-ocp.example.com : ok=21   changed=0    unreachable=0    failed=1   
master-1.wjiang-ocp.example.com : ok=19   changed=0    unreachable=0    failed=1   
     
 
INSTALLER STATUS ************************************************************************************************************************************************************************************************************************************************************************************************************
Initialization  : In Progress (0:00:22)
Friday 04 January 2019  05:43:09 -0500 (0:00:03.793)       0:01:47.104 ********
Comment 18 Tzu-Mainn Chen 2019-01-09 15:57:05 UTC 
Issue should be fixed by https://github.com/openshift/openshift-ansible/pull/10974
Comment 19 Tzu-Mainn Chen 2019-01-15 20:55:11 UTC 
PR merged
Comment 21 weiwei jiang 2019-01-25 07:23:01 UTC 
Checked with openshift3/ose-ansible:v3.11.69, and this patch is still not in https://github.com/openshift/openshift-ansible/pull/10974 .

So issue in https://bugzilla.redhat.com/show_bug.cgi?id=1611839#c16 is still not fixed in this version which the errata https://errata.devel.redhat.com/advisory/38936 will be delivered.
Comment 22 weiwei jiang 2019-01-25 08:43:08 UTC 
openshift3/ose-ansible:v3.11.69 is equal to

sh-4.2$ rpm -qa|grep -i openshift
openshift-ansible-docs-3.11.69-1.git.0.2ff281f.el7.noarch
openshift-ansible-3.11.69-1.git.0.2ff281f.el7.noarch
openshift-ansible-roles-3.11.69-1.git.0.2ff281f.el7.noarch
openshift-ansible-playbooks-3.11.69-1.git.0.2ff281f.el7.noarch
atomic-openshift-clients-3.11.69-1.git.0.7478b86.el7.x86_64
Comment 23 weiwei jiang 2019-01-25 09:31:11 UTC 
And Checked with the 
# rpm -qa|grep -i openshift
openshift-ansible-roles-3.11.73-1.git.0.89d3763.el7.noarch
openshift-ansible-3.11.73-1.git.0.89d3763.el7.noarch
openshift-ansible-playbooks-3.11.73-1.git.0.89d3763.el7.noarch
openshift-ansible-docs-3.11.73-1.git.0.89d3763.el7.noarch
atomic-openshift-clients-3.11.73-1.git.0.8ae9af6.el7.x86_64

also can not work with following parameters, and failed at "TASK [Approve node certificates when bootstrapping]".
Seems like openshift_openstack_dns_nameservers does not take effect with use_provider_network: True.

# cat ~/inventory/group_vars/all.yml |grep -v ^# | grep -v ^$
---
openshift_openstack_use_neutron_internal_dns: False
openshift_openstack_use_no_floating_ip: True
openshift_openstack_use_nsupdate: True
openshift_openstack_use_provider_network: True
openshift_openstack_clusterid: "wjiang-ocp"
openshift_openstack_public_dns_domain: "example.com"
openshift_openstack_dns_nameservers: ["10.8.249.68"]
openshift_openstack_keypair_name: "libra"
openshift_openstack_provider_network_name: "openshift-qe-jenkins"
openshift_openstack_default_image_name: "qe-rhel-7-release"
openshift_openstack_num_masters: 2
openshift_openstack_num_infra: 1
openshift_openstack_num_cns: 0
openshift_openstack_num_nodes: 1
openshift_openstack_num_etcd: 0
openshift_openstack_master_floating_ip: false
openshift_openstack_infra_floating_ip: false
openshift_openstack_etcd_floating_ip: false
openshift_openstack_load_balancer_floating_ip: false
openshift_openstack_compute_floating_ip: false
openshift_openstack_default_flavor: "m1.medium"
openshift_openstack_use_lbaas_load_balancer: false
openshift_openstack_use_vm_load_balancer: true
openshift_openstack_docker_volume_size: "15"
ansible_user: openshift
openshift_openstack_disable_root: true
openshift_openstack_user: openshift
openshift_openstack_heat_template_version: newton
openshift_openstack_nsupdate_zone: wjiang-ocp.example.com
openshift_openstack_external_nsupdate_keys:
  private:
    key_secret: 'U3521fvPGgp1l73K5XXAzRnfM/jYiZ06+9BXSYp7Rqf3s4+K/4YpSplfo9CW8Jmy8iEFEaT1J18j2BYntmHS7w=='
    key_algorithm: 'hmac-md5'
    server: '10.8.249.68'
    key_name: 'wjiang-ocp.example.com'
openshift_openstack_private_hostname_suffix: ""
openshift_openstack_router_name: default2-router
openshift_openstack_node_subnet_name: openshift-qe-jenkins


[openshift@master-0 ~]$ oc get nodes -o wide
NAME                                  STATUS    ROLES     AGE       VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                      KERNEL-VERSION              CONTAINER-RUNTIME
app-node-0.wjiang-ocp.example.com     Ready     compute   3m        v1.11.0+d4cacc0   172.16.122.48   <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-957.1.3.el7.x86_64   docker://1.13.1
infra-node-0.wjiang-ocp.example.com   Ready     infra     3m        v1.11.0+d4cacc0   172.16.122.67   <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-957.1.3.el7.x86_64   docker://1.13.1
master-0.wjiang-ocp.example.com       Ready     master    8m        v1.11.0+d4cacc0   172.16.122.80   <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-957.1.3.el7.x86_64   docker://1.13.1
master-1.wjiang-ocp.example.com       Ready     master    8m        v1.11.0+d4cacc0   172.16.122.76   <none>        Red Hat Enterprise Linux Server 7.6 (Maipo)   3.10.0-957.1.3.el7.x86_64   docker://1.13.1
[openshift@master-0 ~]$ oc get --raw /api/v1/nodes/app-node-0.wjiang-ocp.example.com/proxy/healthz --loglevel=8
I0125 04:27:51.284324    8946 loader.go:359] Config loaded from file /home/openshift/.kube/config
I0125 04:27:51.285117    8946 round_trippers.go:383] GET https://172.16.122.74:8443/api/v1/nodes/app-node-0.wjiang-ocp.example.com/proxy/healthz
I0125 04:27:51.285142    8946 round_trippers.go:390] Request Headers:
I0125 04:27:51.285151    8946 round_trippers.go:393]     User-Agent: oc/v1.11.0+d4cacc0 (linux/amd64) kubernetes/d4cacc0
I0125 04:27:51.285163    8946 round_trippers.go:393]     Accept: application/json, */*
I0125 04:27:51.300967    8946 round_trippers.go:408] Response Status: 503 Service Unavailable in 15 milliseconds
I0125 04:27:51.301007    8946 round_trippers.go:411] Response Headers:
I0125 04:27:51.301018    8946 round_trippers.go:414]     Cache-Control: no-store
I0125 04:27:51.301033    8946 round_trippers.go:414]     Content-Type: text/plain; charset=utf-8
I0125 04:27:51.301043    8946 round_trippers.go:414]     Content-Length: 168
I0125 04:27:51.301050    8946 round_trippers.go:414]     Date: Fri, 25 Jan 2019 09:27:51 GMT
I0125 04:27:51.301084    8946 request.go:897] Response Body: Error: 'dial tcp: lookup app-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host'
Trying to reach: 'https://app-node-0.wjiang-ocp.example.com:10250/healthz'
I0125 04:27:51.301172    8946 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server is currently unable to handle the request",
  "reason": "ServiceUnavailable",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "Error: 'dial tcp: lookup app-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host'\nTrying to reach: 'https://app-node-0.wjiang-ocp.example.com:10250/healthz'"
      }
    ]
  },
  "code": 503
}]
F0125 04:27:51.301276    8946 helpers.go:119] Error from server (ServiceUnavailable): the server is currently unable to handle the request
[openshift@master-0 ~]$ oc get --raw /api/v1/nodes/infra-node-0.wjiang-ocp.example.com/proxy/healthz --loglevel=8
I0125 04:28:06.681860    9082 loader.go:359] Config loaded from file /home/openshift/.kube/config
I0125 04:28:06.682562    9082 round_trippers.go:383] GET https://172.16.122.74:8443/api/v1/nodes/infra-node-0.wjiang-ocp.example.com/proxy/healthz
I0125 04:28:06.682587    9082 round_trippers.go:390] Request Headers:
I0125 04:28:06.682596    9082 round_trippers.go:393]     Accept: application/json, */*
I0125 04:28:06.682604    9082 round_trippers.go:393]     User-Agent: oc/v1.11.0+d4cacc0 (linux/amd64) kubernetes/d4cacc0
I0125 04:28:06.700471    9082 round_trippers.go:408] Response Status: 503 Service Unavailable in 17 milliseconds
I0125 04:28:06.700586    9082 round_trippers.go:411] Response Headers:
I0125 04:28:06.700706    9082 round_trippers.go:414]     Cache-Control: no-store
I0125 04:28:06.700776    9082 round_trippers.go:414]     Content-Type: text/plain; charset=utf-8
I0125 04:28:06.700787    9082 round_trippers.go:414]     Content-Length: 172
I0125 04:28:06.700794    9082 round_trippers.go:414]     Date: Fri, 25 Jan 2019 09:28:06 GMT
I0125 04:28:06.700824    9082 request.go:897] Response Body: Error: 'dial tcp: lookup infra-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host'
Trying to reach: 'https://infra-node-0.wjiang-ocp.example.com:10250/healthz'
I0125 04:28:06.700995    9082 helpers.go:201] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server is currently unable to handle the request",
  "reason": "ServiceUnavailable",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "Error: 'dial tcp: lookup infra-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host'\nTrying to reach: 'https://infra-node-0.wjiang-ocp.example.com:10250/healthz'"
      }
    ]
  },
  "code": 503
}]
F0125 04:28:06.701223    9082 helpers.go:119] Error from server (ServiceUnavailable): the server is currently unable to handle the request
Comment 25 Tzu-Mainn Chen 2019-01-25 14:16:52 UTC 
Hi! Is it possible that you didn't update the Neutron subnet to include the desired dns nameserver? That's a required step, as per https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/configuration.md#floating-ip-address-configuration
Comment 26 Luis Tomas Bolivar 2019-04-09 06:32:54 UTC 
To deploy without FIPs it is not needed to deploy with "use_provider_network: True", that is a different setting. Moving to ON_QA as this is already on the rpm and working
Comment 27 Tomas Sedovic 2019-10-24 12:10:45 UTC 
The Needinfo had been addressed already, clearing the flag.
Comment 32 errata-xmlrpc 2020-09-16 07:46:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 3.11.286 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3695