Description of problem: Currently using the deployment process of deploying using the openshift-ansible RPM requires the use of floating IPs for all master, infra, app nodes. By enabling floating IPs for all nodes you are unnecessarily exposing the openshift cluster to the outside world creating a security vulnerability. Aside from the security aspect, requiring environments to have floating IPs available might not be feasible in certain customer environments. What if a customer wants to deploy a very large environment? Are we going to force they have a public IP for each node? Recommendation: - Make the creation of the bastion host as part of the provision.yml. Since the bastion host is an optional step today it is not part of the same internal network as created by the provision.yml thus the optional bastion host can only talk to the openshift cluster via the public subnet. The purpose of the bastion host is to act as a jumpbox that can be accessed remotely via SSH that can then be used to login to other instances via their private subnets. - Ensure that the installation steps and other important playbooks can run via the internal network i.e. provision.yml, install.yml, uninstall.yml - Apply best practices to the the bastion host to ensure it is properly secured. I'd recommend discussing with someone within the RH security group to ensure their best practices are applied when the bastion host is created. NOTE: Simply removing the floating IP after the deployment of the openshift cluster is not enough in the current state of openshift ansible deployment as any future updates or upgrades would require floating IPs to be re-added to the existing cluster.
We have been tracking the need for a more fine-grained floating IP address and general access control to the nodes. Thanks for creating this BZ. It is be possible to deploy OpenShift without floating IP addresses today though. Here's what you can do: 1. Create a private network + subnet 2. Launch a bastion VM inside that subnet 3. Connect to the bastion VM and secure it as you wish 4. Install Ansible, etc. and configure the openshift-ansible inventory inside the bastion VM 5. Make sure `openshift_openstack_external_network_name` and `openshift_openstack_private_network_name` are NOT set in inventory/group_vars/all.yml 6. Set `openshift_openstack_provider_network_name` to the network you created in step 1. 7. Run the openshift-ansible/playbooks/openstack/openshift-cluster/*.yml playbooks as usual This will cause all OpenShift nodes to be put into the network and subnet you've created in step 1 (instead of creating a new one) and they will not have floating IP addresses assigned. But the bastion VM will be able to keep managing the cluster using the private addresses. Alternatively, you could run the `provision-resources.yml` playbook from the outside, have it create the networks and nodes and then remove all the floating IPs, put a bastion inside that network and finish the deployment. In either case, you must then provide a way to make the cluster accessible to your users by either assigning floating IPs to the master and infra nodes or by putting a proxy / load balancer in front of them. Now, this is not ideal, because neither process is well documented and it is still rather involved, but it's there. Introducing the bastion into openshift-ansible would make things nicer and less error-prone. It would however complicate the openshift-ansible playbooks. We should definitely document this better and help at least partially automate this. I'm a bit wary of doing the full end to end bastion management, but we can definitely help at least with some bits. Also of note is that as far as I understand it the 4.0 installer will not use floating IPs at all.
Attempting to install with instructions above gets the following error: TASK [openshift_openstack : validate the Heat template] ***************************************************************************************************************** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openstack", "orchestration", "template", "validate", "-t", "/tmp/openshift-ansibleZMPVX0/stack.yaml"], "delta": "0:00:02.514699", "end": "2018-08-08 14:05:10.126005", "msg": "non-zero return code", "rc": 1, "start": "2018-08-08 14:05:07.611306", "stderr": "ERROR: The specified re ference \"subnet\" (in api_lb.Properties.vip_subnet) is incorrect.", "stderr_lines": ["ERROR: The specified reference \"subnet\" (in api_lb.Properties.vip_subnet) is inc orrect."], "stdout": "", "stdout_lines": []} Speaking to Tzu-Mainn, he mentioned pointed to 2 sections of the heat_stack.yaml https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_openstack/templates/heat_stack.yaml.j2#L276 https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_openstack/templates/heat_stack.yaml.j2#L196 He mentioned the above is the conditional that controls where that piece is generaeted. However, this would only work if you do not specify a provider network. In this particular, case we want a provider network to be specified so we can have bastion host and OCP instances that are provisioned on the same internal network.
Tomas's excellent PR https://github.com/openshift/openshift-ansible/pull/9862 should also take care of this issue; as a reward I am re-assigning this BZ to him.
Kuryr support was not fully working as there was some information missing on the kuryr-conf side -- such as ids of the created subnet and router. A new PR has been created to fix this (https://github.com/openshift/openshift-ansible/pull/9976)
Per OCP program call on 21-SEP-2018 we are deferring Kuryr-related bugs to 3.11.z
Following the bz instructions, provision fails with TASK [openshift_openstack : validate the Heat template] ************************ Monday 05 November 2018 09:32:14 -0500 (0:00:00.680) 0:00:05.490 ******* fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openstack", "orchestration", "template", "validate", "-t", "/tmp/openshift-ansible77m1pZ/stack.yaml"], "delta": "0:00:04.633918", "end": "2018-11-05 09:32:18.897989", "msg": "non-zero return code", "rc": 1, "start": "2018-11-05 09:32:14.264071", "stderr": "ERROR: The specified reference \"subnet\" (in interface.Properties.subnet_id) is incorrect.", "stderr_lines": ["ERROR: The specified reference \"subnet\" (in interface.Properties.subnet_id) is incorrect."], "stdout": "", "stdout_lines": []} It seems the router points to an unknown subnet[1]: 120 interface: 121 type: OS::Neutron::RouterInterface 122 properties: 123 router_id: { get_resource: router } 124 subnet_id: { get_resource: subnet } So after adding the openshift_openstack_router_name variable in the inventory file, it works. TL;DR.- I think the "openshift_openstack_router_name: openshift-router" variable should be added to the instructions. [1] https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_openstack/templates/heat_stack.yaml.j2#L394-L411
Odd, I don't remember having that issue. Could you attach your inventory files?
http://pastebin.test.redhat.com/666267
Upon further checking, you're right that it's required; and it looks like it's already in the documentation: https://github.com/openshift/openshift-ansible/commit/5c5c3dd12fc8d4fb184b335dfab2d6fdd58df492 It was added after the initial PR, so perhaps the note wasn't there in your build of 3.11, but it looks like this followup was backported to 3.11.
Can we modify the bugzilla doc just in case? Thanks!
Ah, missed the bugzilla doc. Updated now, thanks for pointing it out!
Hi gcheresh: Help check this, since this is not easy to check from OCP side. Thanks
Tried to verify with openshift-ansible-3.11.65-1 but got the following errors, and all instances is up, so no idea if this is really working well: TASK [Gather Cluster facts] ************************************************************************************************************************************************************************************************************************************************************************************************* task path: /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:27 Friday 04 January 2019 05:43:05 -0500 (0:00:00.508) 0:01:43.310 ******** Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py <172.16.122.6> ESTABLISH SSH CONNECTION FOR USER: openshift <172.16.122.6> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.6 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-qewdvtlrdscngtbdbguzcqaynocpvvbd; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py <172.16.122.59> ESTABLISH SSH CONNECTION FOR USER: openshift <172.16.122.59> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.59 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-gblcgvqyfzihjjmgwbmijahwxrhliheb; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py <172.16.122.51> ESTABLISH SSH CONNECTION FOR USER: openshift <172.16.122.51> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.51 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-pkouygkyqoicawmknmsyruipxbjtwsne; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py <172.16.122.44> ESTABLISH SSH CONNECTION FOR USER: openshift <172.16.122.44> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.44 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-akvyzmuvxwuekqncspjehkxbiinukupt; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py <172.16.122.56> ESTABLISH SSH CONNECTION FOR USER: openshift <172.16.122.56> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.56 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-kdaamngypidqdhespyighywnkzhgumjk; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py <172.16.122.41> ESTABLISH SSH CONNECTION FOR USER: openshift <172.16.122.41> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.41 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-didnwroqitclkekgyfrhmwzxrydzasmd; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Using module file /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py <172.16.122.62> ESTABLISH SSH CONNECTION FOR USER: openshift <172.16.122.62> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openshift -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/%h-%r 172.16.122.62 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ifwwsgwdomnkfnwijzvqsvlykqjjhydb; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Escalation succeeded Escalation succeeded Escalation succeeded Escalation succeeded Escalation succeeded Escalation succeeded Escalation succeeded <172.16.122.51> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1300, in <module>\n main()\n File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1287, in main\n additive_facts_to_overwrite)\n File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1039, in __init__\n additive_facts_to_overwrite)\n File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n provider_info.get(\'metadata\')\n File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File "/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py", line 310, in nor malize_openstack_facts\n if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n') fatal: [infra-node-0.wjiang-ocp.example.com]: FAILED! => { "changed": false, "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1300, in <module>\n main()\n File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1287, in main\n additive_facts_to_overwrite)\n File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1039, in __init__\n additive_facts_to_overwrite)\n File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n provider_info.get('metadata')\n File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File \"/tmp/ansible_Fyuwdw/ansible_module_openshift_facts.py\", line 310, in normalize_openstack_facts\n if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1 } <172.16.122.44> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1300, in <module>\n main()\n File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1287, in main\n additive_facts_to_overwrite)\n File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1039, in __init__\n additive_facts_to_overwrite)\n File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n provider_info.get(\'metadata\')\n File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File "/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py", line 310, in nor malize_openstack_facts\n if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n') fatal: [etcd-0.wjiang-ocp.example.com]: FAILED! => { "changed": false, "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1300, in <module>\n main()\n File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1287, in main\n additive_facts_to_overwrite)\n File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1039, in __init__\n additive_facts_to_overwrite)\n File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n provider_info.get('metadata')\n File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File \"/tmp/ansible_sHXdZW/ansible_module_openshift_facts.py\", line 310, in normalize_openstack_facts\n if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1 } <172.16.122.56> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1300, in <module>\n main()\n File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1287, in main\n additive_facts_to_overwrite)\n File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1039, in __init__\n additive_facts_to_overwrite)\n File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n provider_info.get(\'metadata\')\n File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File "/tmp/ansible_za4nsm/ansible_module_openshift_facts.py", line 310, in nor malize_openstack_facts\n if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n') fatal: [lb-0.wjiang-ocp.example.com]: FAILED! => { "changed": false, "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1300, in <module>\n main()\n File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1287, in main\n additive_facts_to_overwrite)\n File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1039, in __init__\n additive_facts_to_overwrite)\n File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n provider_info.get('metadata')\n File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File \"/tmp/ansible_za4nsm/ansible_module_openshift_facts.py\", line 310, in normalize_openstack_facts\n if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1 } <172.16.122.59> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1300, in <module>\n main()\n File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1287, in main\n additive_facts_to_overwrite)\n File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1039, in __init__\n additive_facts_to_overwrite)\n File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n provider_info.get(\'metadata\')\n File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File "/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py", line 310, in nor malize_openstack_facts\n if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n') fatal: [app-node-0.wjiang-ocp.example.com]: FAILED! => { "changed": false, "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1300, in <module>\n main()\n File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1287, in main\n additive_facts_to_overwrite)\n File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1039, in __init__\n additive_facts_to_overwrite)\n File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n provider_info.get('metadata')\n File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File \"/tmp/ansible_uDlDM4/ansible_module_openshift_facts.py\", line 310, in normalize_openstack_facts\n if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1 } <172.16.122.6> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1300, in <module>\n main()\n File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1287, in main\n additive_facts_to_overwrite)\n File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1039, in __init__\n additive_facts_to_overwrite)\n File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n provider_info.get(\'metadata\')\n File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File "/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py", line 310, in norm alize_openstack_facts\n if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n') fatal: [master-1.wjiang-ocp.example.com]: FAILED! => { "changed": false, "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1300, in <module>\n main()\n File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1287, in main\n additive_facts_to_overwrite)\n File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1039, in __init__\n additive_facts_to_overwrite)\n File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n provider_info.get('metadata')\n File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File \"/tmp/ansible_FrFYCt/ansible_module_openshift_facts.py\", line 310, in normalize_openstack_facts\n if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1 } <172.16.122.41> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1300, in <module>\n main()\n File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1287, in main\n additive_facts_to_overwrite)\n File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1039, in __init__\n additive_facts_to_overwrite)\n File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n provider_info.get(\'metadata\')\n File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File "/tmp/ansible_lI7umj/ansible_module_openshift_facts.py", line 310, in nor malize_openstack_facts\n if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n') fatal: [etcd-1.wjiang-ocp.example.com]: FAILED! => { "changed": false, "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1300, in <module>\n main()\n File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1287, in main\n additive_facts_to_overwrite)\n File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1039, in __init__\n additive_facts_to_overwrite)\n File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n provider_info.get('metadata')\n File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File \"/tmp/ansible_lI7umj/ansible_module_openshift_facts.py\", line 310, in normalize_openstack_facts\n if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1 } <172.16.122.62> (1, '', 'KeyError(\'ansible_os_family\',)\nTraceback (most recent call last):\n File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1300, in <module>\n main()\n File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1287, in main\n additive_facts_to_overwrite)\n File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1039, in __init__\n additive_facts_to_overwrite)\n File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 1194, in init_provider_facts\n provider_info.get(\'metadata\')\n File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File "/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py", line 310, in nor malize_openstack_facts\n if socket.gethostbyname(metadata[\'ec2_compat\'][h_var]) == metadata[\'ec2_compat\'][ip_var].split(\',\')[0]:\nAttributeError: \'list\' object has no attribute \'split\'\n') fatal: [master-0.wjiang-ocp.example.com]: FAILED! => { "changed": false, "module_stderr": "KeyError('ansible_os_family',)\nTraceback (most recent call last):\n File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1300, in <module>\n main()\n File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1287, in main\n additive_facts_to_overwrite)\n File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1039, in __init__\n additive_facts_to_overwrite)\n File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1061, in generate_facts\n provider_facts = self.init_provider_facts()\n File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 1194, in init_provider_facts\n provider_info.get('metadata')\n File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 345, in normalize_provider_facts\n facts = normalize_openstack_facts(metadata, facts)\n File \"/tmp/ansible_nuW1LW/ansible_module_openshift_facts.py\", line 310, in normalize_openstack_facts\n if socket.gethostbyname(metadata['ec2_compat'][h_var]) == metadata['ec2_compat'][ip_var].split(',')[0]:\nAttributeError: 'list' object has no attribute 'split'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1 } PLAY RECAP ****************************************************************************************************************************************************************************************************************************************************************************************************************** app-node-0.wjiang-ocp.example.com : ok=18 changed=0 unreachable=0 failed=1 etcd-0.wjiang-ocp.example.com : ok=17 changed=0 unreachable=0 failed=1 etcd-1.wjiang-ocp.example.com : ok=17 changed=0 unreachable=0 failed=1 infra-node-0.wjiang-ocp.example.com : ok=17 changed=0 unreachable=0 failed=1 lb-0.wjiang-ocp.example.com : ok=17 changed=0 unreachable=0 failed=1 localhost : ok=61 changed=14 unreachable=0 failed=0 master-0.wjiang-ocp.example.com : ok=21 changed=0 unreachable=0 failed=1 master-1.wjiang-ocp.example.com : ok=19 changed=0 unreachable=0 failed=1 INSTALLER STATUS ************************************************************************************************************************************************************************************************************************************************************************************************************ Initialization : In Progress (0:00:22) Friday 04 January 2019 05:43:09 -0500 (0:00:03.793) 0:01:47.104 ********
Issue should be fixed by https://github.com/openshift/openshift-ansible/pull/10974
PR merged
Checked with openshift3/ose-ansible:v3.11.69, and this patch is still not in https://github.com/openshift/openshift-ansible/pull/10974 . So issue in https://bugzilla.redhat.com/show_bug.cgi?id=1611839#c16 is still not fixed in this version which the errata https://errata.devel.redhat.com/advisory/38936 will be delivered.
openshift3/ose-ansible:v3.11.69 is equal to sh-4.2$ rpm -qa|grep -i openshift openshift-ansible-docs-3.11.69-1.git.0.2ff281f.el7.noarch openshift-ansible-3.11.69-1.git.0.2ff281f.el7.noarch openshift-ansible-roles-3.11.69-1.git.0.2ff281f.el7.noarch openshift-ansible-playbooks-3.11.69-1.git.0.2ff281f.el7.noarch atomic-openshift-clients-3.11.69-1.git.0.7478b86.el7.x86_64
And Checked with the # rpm -qa|grep -i openshift openshift-ansible-roles-3.11.73-1.git.0.89d3763.el7.noarch openshift-ansible-3.11.73-1.git.0.89d3763.el7.noarch openshift-ansible-playbooks-3.11.73-1.git.0.89d3763.el7.noarch openshift-ansible-docs-3.11.73-1.git.0.89d3763.el7.noarch atomic-openshift-clients-3.11.73-1.git.0.8ae9af6.el7.x86_64 also can not work with following parameters, and failed at "TASK [Approve node certificates when bootstrapping]". Seems like openshift_openstack_dns_nameservers does not take effect with use_provider_network: True. # cat ~/inventory/group_vars/all.yml |grep -v ^# | grep -v ^$ --- openshift_openstack_use_neutron_internal_dns: False openshift_openstack_use_no_floating_ip: True openshift_openstack_use_nsupdate: True openshift_openstack_use_provider_network: True openshift_openstack_clusterid: "wjiang-ocp" openshift_openstack_public_dns_domain: "example.com" openshift_openstack_dns_nameservers: ["10.8.249.68"] openshift_openstack_keypair_name: "libra" openshift_openstack_provider_network_name: "openshift-qe-jenkins" openshift_openstack_default_image_name: "qe-rhel-7-release" openshift_openstack_num_masters: 2 openshift_openstack_num_infra: 1 openshift_openstack_num_cns: 0 openshift_openstack_num_nodes: 1 openshift_openstack_num_etcd: 0 openshift_openstack_master_floating_ip: false openshift_openstack_infra_floating_ip: false openshift_openstack_etcd_floating_ip: false openshift_openstack_load_balancer_floating_ip: false openshift_openstack_compute_floating_ip: false openshift_openstack_default_flavor: "m1.medium" openshift_openstack_use_lbaas_load_balancer: false openshift_openstack_use_vm_load_balancer: true openshift_openstack_docker_volume_size: "15" ansible_user: openshift openshift_openstack_disable_root: true openshift_openstack_user: openshift openshift_openstack_heat_template_version: newton openshift_openstack_nsupdate_zone: wjiang-ocp.example.com openshift_openstack_external_nsupdate_keys: private: key_secret: 'U3521fvPGgp1l73K5XXAzRnfM/jYiZ06+9BXSYp7Rqf3s4+K/4YpSplfo9CW8Jmy8iEFEaT1J18j2BYntmHS7w==' key_algorithm: 'hmac-md5' server: '10.8.249.68' key_name: 'wjiang-ocp.example.com' openshift_openstack_private_hostname_suffix: "" openshift_openstack_router_name: default2-router openshift_openstack_node_subnet_name: openshift-qe-jenkins [openshift@master-0 ~]$ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME app-node-0.wjiang-ocp.example.com Ready compute 3m v1.11.0+d4cacc0 172.16.122.48 <none> Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 infra-node-0.wjiang-ocp.example.com Ready infra 3m v1.11.0+d4cacc0 172.16.122.67 <none> Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 master-0.wjiang-ocp.example.com Ready master 8m v1.11.0+d4cacc0 172.16.122.80 <none> Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 master-1.wjiang-ocp.example.com Ready master 8m v1.11.0+d4cacc0 172.16.122.76 <none> Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 [openshift@master-0 ~]$ oc get --raw /api/v1/nodes/app-node-0.wjiang-ocp.example.com/proxy/healthz --loglevel=8 I0125 04:27:51.284324 8946 loader.go:359] Config loaded from file /home/openshift/.kube/config I0125 04:27:51.285117 8946 round_trippers.go:383] GET https://172.16.122.74:8443/api/v1/nodes/app-node-0.wjiang-ocp.example.com/proxy/healthz I0125 04:27:51.285142 8946 round_trippers.go:390] Request Headers: I0125 04:27:51.285151 8946 round_trippers.go:393] User-Agent: oc/v1.11.0+d4cacc0 (linux/amd64) kubernetes/d4cacc0 I0125 04:27:51.285163 8946 round_trippers.go:393] Accept: application/json, */* I0125 04:27:51.300967 8946 round_trippers.go:408] Response Status: 503 Service Unavailable in 15 milliseconds I0125 04:27:51.301007 8946 round_trippers.go:411] Response Headers: I0125 04:27:51.301018 8946 round_trippers.go:414] Cache-Control: no-store I0125 04:27:51.301033 8946 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I0125 04:27:51.301043 8946 round_trippers.go:414] Content-Length: 168 I0125 04:27:51.301050 8946 round_trippers.go:414] Date: Fri, 25 Jan 2019 09:27:51 GMT I0125 04:27:51.301084 8946 request.go:897] Response Body: Error: 'dial tcp: lookup app-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host' Trying to reach: 'https://app-node-0.wjiang-ocp.example.com:10250/healthz' I0125 04:27:51.301172 8946 helpers.go:201] server response object: [{ "metadata": {}, "status": "Failure", "message": "the server is currently unable to handle the request", "reason": "ServiceUnavailable", "details": { "causes": [ { "reason": "UnexpectedServerResponse", "message": "Error: 'dial tcp: lookup app-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host'\nTrying to reach: 'https://app-node-0.wjiang-ocp.example.com:10250/healthz'" } ] }, "code": 503 }] F0125 04:27:51.301276 8946 helpers.go:119] Error from server (ServiceUnavailable): the server is currently unable to handle the request [openshift@master-0 ~]$ oc get --raw /api/v1/nodes/infra-node-0.wjiang-ocp.example.com/proxy/healthz --loglevel=8 I0125 04:28:06.681860 9082 loader.go:359] Config loaded from file /home/openshift/.kube/config I0125 04:28:06.682562 9082 round_trippers.go:383] GET https://172.16.122.74:8443/api/v1/nodes/infra-node-0.wjiang-ocp.example.com/proxy/healthz I0125 04:28:06.682587 9082 round_trippers.go:390] Request Headers: I0125 04:28:06.682596 9082 round_trippers.go:393] Accept: application/json, */* I0125 04:28:06.682604 9082 round_trippers.go:393] User-Agent: oc/v1.11.0+d4cacc0 (linux/amd64) kubernetes/d4cacc0 I0125 04:28:06.700471 9082 round_trippers.go:408] Response Status: 503 Service Unavailable in 17 milliseconds I0125 04:28:06.700586 9082 round_trippers.go:411] Response Headers: I0125 04:28:06.700706 9082 round_trippers.go:414] Cache-Control: no-store I0125 04:28:06.700776 9082 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I0125 04:28:06.700787 9082 round_trippers.go:414] Content-Length: 172 I0125 04:28:06.700794 9082 round_trippers.go:414] Date: Fri, 25 Jan 2019 09:28:06 GMT I0125 04:28:06.700824 9082 request.go:897] Response Body: Error: 'dial tcp: lookup infra-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host' Trying to reach: 'https://infra-node-0.wjiang-ocp.example.com:10250/healthz' I0125 04:28:06.700995 9082 helpers.go:201] server response object: [{ "metadata": {}, "status": "Failure", "message": "the server is currently unable to handle the request", "reason": "ServiceUnavailable", "details": { "causes": [ { "reason": "UnexpectedServerResponse", "message": "Error: 'dial tcp: lookup infra-node-0.wjiang-ocp.example.com on 172.16.122.80:53: no such host'\nTrying to reach: 'https://infra-node-0.wjiang-ocp.example.com:10250/healthz'" } ] }, "code": 503 }] F0125 04:28:06.701223 9082 helpers.go:119] Error from server (ServiceUnavailable): the server is currently unable to handle the request
Hi! Is it possible that you didn't update the Neutron subnet to include the desired dns nameserver? That's a required step, as per https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/configuration.md#floating-ip-address-configuration
To deploy without FIPs it is not needed to deploy with "use_provider_network: True", that is a different setting. Moving to ON_QA as this is already on the rpm and working
The Needinfo had been addressed already, clearing the flag.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.286 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3695