Created attachment 1537228 [details] ansible-log Description of problem: Unable to deploy with Octavia and Provider network on OSP14, OCP3.11 using openshift ansible installer TASK [openshift_openstack : validate the Heat template] ******************************************************************************************* Thursday 21 February 2019 14:52:35 -0500 (0:00:01.087) 0:00:04.841 ***** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openstack", "orchestration", "template", "validate", "-t", "/tmp/openshift-ansibleuhuktt/stack.yaml"], "delta": "0:00:02.580420", "end": "2019-02-21 14:52:38.706341", "msg": "non-zero return code", "rc": 1, "start": "2019-02-21 14:52:36.125921", "stderr": "ERROR: The specified reference \"subnet\" (in api_lb.Properties.vip_subnet) is incorrect.", "stderr_lines": ["ERROR: The specified reference \"subnet\" (in api_lb.Properties.vip_subnet) is incorrect."], "stdout": "", "stdout_lines": []} The provider network that is being used: (chrisj-osp14) [stack@undercloud-osp14 ~]$ openstack network show provider1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2019-02-05T22:14:21Z | | description | | | dns_domain | None | | id | 4a030c51-2a82-4e17-ab10-2464f6fa1a6d | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1500 | | name | provider1 | | port_security_enabled | True | | project_id | 7ad8b8ac7c3a43beaa75042e46379230 | | provider:network_type | vlan | | provider:physical_network | fast | | provider:segmentation_id | 315 | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | True | | status | ACTIVE | | subnets | aed181a9-95a0-4117-bf17-0d37d9e57bc2 | | tags | | | updated_at | 2019-02-05T22:14:24Z | +---------------------------+--------------------------------------+ (chrisj-osp14) [stack@undercloud-osp14 ~]$ openstack subnet show provider1-sub1 +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 172.31.5.10-172.31.5.199 | | cidr | 172.31.5.0/24 | | created_at | 2019-02-05T22:14:24Z | | description | | | dns_nameservers | 172.31.8.1 | | enable_dhcp | True | | gateway_ip | 172.31.5.254 | | host_routes | | | id | aed181a9-95a0-4117-bf17-0d37d9e57bc2 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | provider1-sub1 | | network_id | 4a030c51-2a82-4e17-ab10-2464f6fa1a6d | | project_id | 7ad8b8ac7c3a43beaa75042e46379230 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2019-02-05T22:14:24Z | +-------------------+--------------------------------------+ Relevant configuration: openshift_openstack_use_lbaas_load_balancer: true openshift_openstack_provider_network_name: "provider1" openshift_openstack_use_provider_network: True I am not entirely sure if Octavia is supported with provider networks (due to a nature of provider networks), but I couldn't find any documentation that would say it's not Version-Release number of the following components: ansible-2.5.14-1.el7ae.noarch [cloud-user@bastion ~]$ ansible --version ansible 2.5.14 config file = /home/cloud-user/ansible.cfg configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] openshift-ansible synced yesterday with https://github.com/openshift/openshift-ansible How reproducible: Every time Steps to Reproduce: 1. enable provider networks 2. enable octavia 3. ansible-playbook --user openshift -i openshift-ansible/playbooks/openstack/inventory.py -i inventory openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml -e openshift_version=3.11 Actual results: TASK [openshift_openstack : validate the Heat template] ******************************************************************************************* Thursday 21 February 2019 15:13:41 -0500 (0:00:01.090) 0:00:04.829 ***** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openstack", "orchestration", "template", "validate", "-t", "/tmp/openshift-ansibleicSb5T/stack.yaml"], "delta": "0:00:02.561637", "end": "2019-02-21 15:13:44.337486", "msg": "non-zero return code", "rc": 1, "start": "2019-02-21 15:13:41.775849", "stderr": "ERROR: The specified reference \"subnet\" (in api_lb.Properties.vip_subnet) is incorrect.", "stderr_lines": ["ERROR: The specified reference \"subnet\" (in api_lb.Properties.vip_subnet) is incorrect."], "stdout": "", "stdout_lines": []} NO MORE HOSTS LEFT ******************************************************************************************************************************** PLAY RECAP **************************************************************************************************************************************** localhost : ok=11 changed=7 unreachable=0 failed=1 Thursday 21 February 2019 15:13:44 -0500 (0:00:02.769) 0:00:07.598 ***** =============================================================================== openshift_openstack : validate the Heat template ------------------------------------------------------------------------------------------- 2.77s openshift_openstack : check for openstack client ------------------------------------------------------------------------------------------- 1.09s openshift_openstack : generate HOT stack template from jinja2 template --------------------------------------------------------------------- 0.88s openshift_openstack : generate HOT server template from jinja2 template -------------------------------------------------------------------- 0.48s openshift_openstack : generate HOT nested subports template from jinja2 template ----------------------------------------------------------- 0.46s openshift_openstack : generate user_data from jinja2 template ------------------------------------------------------------------------------ 0.46s openshift_openstack : generate HOT nested segmentation ids template from jinja2 template --------------------------------------------------- 0.46s openshift_openstack : create HOT stack template prefix ------------------------------------------------------------------------------------- 0.34s openshift_openstack : Set custom router id ------------------------------------------------------------------------------------------------- 0.09s openshift_openstack : Generate the templates ----------------------------------------------------------------------------------------------- 0.07s openshift_openstack : set template paths --------------------------------------------------------------------------------------------------- 0.06s openshift_openstack : Print out the Heat template directory -------------------------------------------------------------------------------- 0.06s retrieve cluster name from the environment if present -------------------------------------------------------------------------------------- 0.05s openshift_openstack : Get custom router id ------------------------------------------------------------------------------------------------- 0.04s openshift_openstack : Set custom network id ------------------------------------------------------------------------------------------------ 0.04s openshift_openstack : Set custom subnet id ------------------------------------------------------------------------------------------------- 0.04s openshift_openstack : Set custom subnet cidr ----------------------------------------------------------------------------------------------- 0.04s openshift_openstack : Get subnet facts when using a custom subnet -------------------------------------------------------------------------- 0.04s Expected results: it either should work or be documented that it's not supported Additional info: The same configuration with openshift_openstack_use_vm_load_balancer: true would work.
Hi! We haven't tested this specific configuration, but looking at the relevant heat template code - https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_openstack/templates/heat_stack.yaml.j2#L179-L181 - could you try setting openshift_openstack_node_subnet_name?
Small update: a similar configuration (which uses no floating IP) works for me with octavia,if I set openshift_openstack_node_subnet_name.
I have added following to my all.yml inventory file: openshift_openstack_use_lbaas_load_balancer: true #openshift_openstack_use_vm_load_balancer: true openshift_openstack_node_subnet_name: "provider1-sub1" but still get the same error on deployment. TASK [openshift_openstack : validate the Heat template] ****************************************************************************************************************************************************************************************** Tuesday 26 February 2019 13:18:58 -0500 (0:00:01.027) 0:00:06.672 ****** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["openstack", "orchestration", "template", "validate", "-t", "/tmp/openshift-ansiblekBfAvU/stack.yaml"], "delta": "0:00:04.296256", "end": "2019-02-26 13:19:02.892572", "msg": "non-zero return code", "rc": 1, "start": "2019-02-26 13:18:58.596316", "stderr": "ERROR: outputs.public_api_ip.value.get_attr: The specified reference \"api_lb_floating_ip\" (in unknown) is incorrect.", "stderr_lines": ["ERROR: outputs.public_api_ip.value.get_attr: The specified reference \"api_lb_floating_ip\" (in unknown) is incorrect."], "stdout": "", "stdout_lines": []} My subnets are as follows: (shiftstack) [stack@undercloud-osp14 ~]$ openstack subnet list +--------------------------------------+-------------------+--------------------------------------+-----------------+ | ID | Name | Network | Subnet | +--------------------------------------+-------------------+--------------------------------------+-----------------+ | 0b4bce88-2843-406b-a946-d6c67e5a14ef | openshift-dns | 69824114-6d71-49e8-8f30-8bc22d48e953 | 192.168.23.0/24 | | 6d3ee7f1-275a-4a4f-86d2-d60f4f192717 | baremetal-sub | c219dd21-ad3d-41fc-a3f3-819dc656614a | 172.31.10.0/24 | | 958aca3e-f128-47c1-ade5-36968a03f208 | public_subnet-int | eeebf1e6-2361-4575-86de-0a8a59ca2712 | 172.31.8.0/24 | | aed181a9-95a0-4117-bf17-0d37d9e57bc2 | provider1-sub1 | 4a030c51-2a82-4e17-ab10-2464f6fa1a6d | 172.31.5.0/24 | | b49a4559-8d35-4b85-94a9-55555e7e3643 | provider-sub1 | 961258b0-8510-4a36-9889-190d35ad5bd0 | 172.31.6.0/24 | +--------------------------------------+-------------------+--------------------------------------+-----------------+
I just realized this looks like a different error
Hi! Is openshift_openstack_load_balancer_floating_ip set to false?
If I add the openshift_openstack_load_balancer_floating_ip false flag I get past the "validate the Heat template" step, but fail at: openshift_openstack : Handle the Stack (create/delete) step The stack creation fails with the following: (shiftstack) [stack@undercloud-osp14 ~]$ openstack stack failures list chris-openshift chris-openshift.infra_nodes.2: resource_type: file:///tmp/openshift-ansible_jUc9t/server.yaml physical_resource_id: status: CREATE_FAILED status_reason: | resources[2]: Property error: resources.router_lb_pool_member_http.properties.subnet: The Parameter (subnet) was not provided.
Ahh - apologies, I missed the fact that you were using a provider network. Looking at the heat template - specifically https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_openstack/templates/heat_stack_server.yaml.j2#L64-L69 - it looks like this just might not work. Luis, could you comment on whether this configuration is supposed to work or not?
I'm actually not sure here, probably we are missing the same if at the router loadbalancer than at the api lb one when using octavia. So something like this https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_openstack/templates/heat_stack_server.yaml.j2#L345-L347 at this: https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_openstack/templates/heat_stack_server.yaml.j2#L357
Hi! Thanks for the pointer. I think I may have misdirected you; the actual error is the one Chris reported in the first comment (everything after is confusion due to me not realizing that a provider network was involved). The error seems to take place here: https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_openstack/templates/heat_stack.yaml.j2#L167-L183 Looking at the conditional, it looks like the api_lb is created when a lbaas load balancer is specified; but it also looks like maybe it shouldn't be if a provider network is also specified? Do you know if that's supposed to work? I don't think a subnet will exist when using a provider network.
Chris, I got the provisioning playbook to work with the following: a) set openshift_openstack_node_subnet_name b) set openshift_openstack_load_balancer_floating_ip to false c) apply https://github.com/openshift/openshift-ansible/pull/11285 I don't have a full setup to test for a provider network though; is this a fix you'd be able to test?
Thanks. That worked for me as well.
@Tzu-Mainn: If the setup was with octavia, I don't understand how PR 11285 helps here. It should only be needed in case lbaasv2 is used, right? And that is not the case. Did you try with only steps a) and b)
Luis and I have resolved the somewhat confusing conditional in the PR and agreed that this solution can work! @Chris, could you confirm that the solution works for you in full, including the installation of openshift? If so, I'll update the PR to include docs changes and submit it.
It works after performing all the steps from comment #10 .. I think there might be some issue on how octavia handles load balancing in this provider network setup, but this is outside the scope of this BZ. Thanks for providing the solutions to this BZ. Please let me know if I can help reviewing docs.
OpenShift have no resources to verify the scenarios with provider_network, this need enable neurton internal dns. And also we have no octivia enabled OSP can do testing.
Moving to verified as it was reported that the solution described in comment #10 worked.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.286 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3695