Created attachment 1471885 [details] vlan floating ip not reachable - full scenario example Description of problem: On vlan environment, creating multiple VMs on two different networks, with external network, and IPv4 + IPv6 subnets, sometimes gets an unreachable floating ip, only on one of the networks. Version-Release number of selected component (if applicable): OSP version: 13 -p 2018-07-13.1 How reproducible: sometimes Steps to Reproduce: Please see attachment. Actual results: (overcloud) [stack@undercloud-0 ~]$ openstack network list --external +--------------------------------------+------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+------+--------------------------------------+ | bc8634e6-658d-4e70-bcc4-1788dac3c37b | nova | 26953cf1-9ee0-47ac-a8a3-91d8c49e9c64 | +--------------------------------------+------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack network show nova +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2018-07-31T15:28:49Z | | description | | | dns_domain | None | | id | bc8634e6-658d-4e70-bcc4-1788dac3c37b | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1500 | | name | nova | | port_security_enabled | True | | project_id | d475a218e0ae41948d6549b5bee66b44 | | provider:network_type | vlan | | provider:physical_network | datacentre | | provider:segmentation_id | 181 | | qos_policy_id | None | | revision_number | 7 | | router:external | External | | segments | None | | shared | False | | status | ACTIVE | | subnets | 26953cf1-9ee0-47ac-a8a3-91d8c49e9c64 | | tags | | | updated_at | 2018-07-31T15:28:56Z | +---------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack subnet show 26953cf1-9ee0-47ac-a8a3-91d8c49e9c64 +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 10.35.166.100-10.35.166.140 | | cidr | 10.35.166.0/24 | | created_at | 2018-07-31T15:28:56Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 10.35.166.254 | | host_routes | | | id | 26953cf1-9ee0-47ac-a8a3-91d8c49e9c64 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | nova | | network_id | bc8634e6-658d-4e70-bcc4-1788dac3c37b | | project_id | d475a218e0ae41948d6549b5bee66b44 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2018-07-31T15:28:56Z | +-------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack router list +--------------------------------------+-------------+--------+-------+-------------+------+----------------------------------+ | ID | Name | Status | State | Distributed | HA | Project | +--------------------------------------+-------------+--------+-------+-------------+------+----------------------------------+ | 60df45f8-7209-46b0-a084-7ca1e2aa2202 | Router_eNet | ACTIVE | UP | False | True | d475a218e0ae41948d6549b5bee66b44 | +--------------------------------------+-------------+--------+-------+-------------+------+----------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+----------------+--------+-----------------------------------------------------------------+----------+---------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------------+--------+-----------------------------------------------------------------+----------+---------------+ | ce5fee85-d885-48a9-8f99-ccb4c006b76b | VM_cirros35_21 | ACTIVE | net_ipv64_2=2002::f816:3eff:feb1:3687, 10.0.2.12, 10.35.166.108 | cirros35 | cirros_flavor | | 7a9ddfa9-c72d-440e-995a-25dc4436a80c | VM_cirros35_12 | ACTIVE | net_ipv64_1=10.0.1.10, 2001::f816:3eff:fe1b:285d, 10.35.166.107 | cirros35 | cirros_flavor | | e3e17c0c-e2bc-48d3-a042-68faa793ca4b | VM_cirros35_11 | ACTIVE | net_ipv64_1=10.0.1.5, 2001::f816:3eff:fe51:3fc1, 10.35.166.103 | cirros35 | cirros_flavor | +--------------------------------------+----------------+--------+-----------------------------------------------------------------+----------+---------------+ (overcloud) [stack@undercloud-0 ~]$ ping 10.35.166.107 PING 10.35.166.101 (10.35.166.101) 56(84) bytes of data. 64 bytes from 10.35.166.107: icmp_seq=1 ttl=61 time=2003 ms 64 bytes from 10.35.166.107: icmp_seq=2 ttl=61 time=1004 ms 64 bytes from 10.35.166.107: icmp_seq=3 ttl=61 time=4.18 ms 64 bytes from 10.35.166.107: icmp_seq=4 ttl=61 time=0.772 ms 64 bytes from 10.35.166.107: icmp_seq=5 ttl=61 time=0.811 ms 64 bytes from 10.35.166.107: icmp_seq=6 ttl=61 time=0.781 ms --- 10.35.166.107 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5001ms rtt min/avg/max/mdev = 0.772/502.346/2003.415/764.617 ms, pipe 3 (overcloud) [stack@undercloud-0 ~]$ ping 10.35.166.108 PING 10.35.166.108 (10.35.166.108) 56(84) bytes of data. From 10.35.166.108 icmp_seq=1 Destination Host Unreachable From 10.35.166.108 icmp_seq=2 Destination Host Unreachable From 10.35.166.108 icmp_seq=3 Destination Host Unreachable From 10.35.166.108 icmp_seq=4 Destination Host Unreachable --- 10.35.166.108 ping statistics --- 7 packets transmitted, 0 received, +4 errors, 100% packet loss, time 6001ms (overcloud) [stack@undercloud-0 ~]$ openstack port list +--------------------------------------+-------------------------------------------------+-------------------+------------------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+-------------------------------------------------+-------------------+------------------------------------------------------------------------------------------+--------+ | 03b83cbd-1b35-4890-b462-49611bf4858a | | fa:16:3e:10:6b:69 | ip_address='10.35.166.107', subnet_id='26953cf1-9ee0-47ac-a8a3-91d8c49e9c64' | N/A | | 055b0d0a-1b3e-4f80-9ac7-c396090dc07b | HA port tenant d475a218e0ae41948d6549b5bee66b44 | fa:16:3e:53:ab:45 | ip_address='169.254.192.7', subnet_id='e50b56f3-e60b-4a4d-a021-427b270bfbcb' | ACTIVE | | 193aee20-7168-4e0c-95a5-427b2f7bebdf | | fa:16:3e:57:07:ce | ip_address='2001::1', subnet_id='8d3bd293-2d7b-47be-8ae8-1af11ebc4ffd' | ACTIVE | | 3a287945-d218-4b7a-b1eb-4f0b805349f1 | VM_cirros35_12_10.35.166.107 | fa:16:3e:1b:28:5d | ip_address='10.0.1.10', subnet_id='675db1a0-0201-4d77-8b9f-58c40898f167' | ACTIVE | | | | | ip_address='2001::f816:3eff:fe1b:285d', subnet_id='8d3bd293-2d7b-47be-8ae8-1af11ebc4ffd' | | | 44885935-c129-4b37-a3ba-eabe5779988d | | fa:16:3e:e2:42:17 | ip_address='2002::1', subnet_id='212674e2-af08-4605-8f5b-5951c032753a' | ACTIVE | | 4e322d44-5cdf-4f70-be00-06bc57a8d038 | | fa:16:3e:86:3f:0c | ip_address='10.0.2.2', subnet_id='f3b825af-082b-4df1-a73d-5aca0ef8cb1b' | ACTIVE | | | | | ip_address='2002::f816:3eff:fe86:3f0c', subnet_id='212674e2-af08-4605-8f5b-5951c032753a' | | | 5a2a0f37-1e20-43a5-b5d3-df62361b4fe7 | | fa:16:3e:9d:0b:67 | ip_address='10.0.1.2', subnet_id='675db1a0-0201-4d77-8b9f-58c40898f167' | ACTIVE | | | | | ip_address='2001::f816:3eff:fe9d:b67', subnet_id='8d3bd293-2d7b-47be-8ae8-1af11ebc4ffd' | | | 5a811e07-45f8-4443-8d96-792a6fe51973 | | fa:16:3e:4e:93:90 | ip_address='10.0.1.4', subnet_id='675db1a0-0201-4d77-8b9f-58c40898f167' | ACTIVE | | | | | ip_address='2001::f816:3eff:fe4e:9390', subnet_id='8d3bd293-2d7b-47be-8ae8-1af11ebc4ffd' | | | 6cb42b67-9ce2-48c4-bf9d-310463e36193 | | fa:16:3e:ed:09:88 | ip_address='10.35.166.103', subnet_id='26953cf1-9ee0-47ac-a8a3-91d8c49e9c64' | N/A | | 73ddd76c-6469-477c-9049-504620caf25e | | fa:16:3e:0c:dc:5e | ip_address='10.0.2.4', subnet_id='f3b825af-082b-4df1-a73d-5aca0ef8cb1b' | ACTIVE | | | | | ip_address='2002::f816:3eff:fe0c:dc5e', subnet_id='212674e2-af08-4605-8f5b-5951c032753a' | | | 856cf0f7-3a48-4c8d-bd81-4287182a3c72 | | fa:16:3e:ff:c5:b1 | ip_address='10.0.2.1', subnet_id='f3b825af-082b-4df1-a73d-5aca0ef8cb1b' | ACTIVE | | 9839ddbb-762f-47b0-b318-e44dc008d4e9 | HA port tenant d475a218e0ae41948d6549b5bee66b44 | fa:16:3e:6d:c2:67 | ip_address='169.254.192.6', subnet_id='e50b56f3-e60b-4a4d-a021-427b270bfbcb' | ACTIVE | | a026c5e4-3c2b-49f5-89bb-858ff40d6434 | | fa:16:3e:0d:cc:f1 | ip_address='10.0.1.1', subnet_id='675db1a0-0201-4d77-8b9f-58c40898f167' | ACTIVE | | a1ac7963-da43-4c5f-8e16-a52e95ddd860 | HA port tenant d475a218e0ae41948d6549b5bee66b44 | fa:16:3e:6b:4e:72 | ip_address='169.254.192.9', subnet_id='e50b56f3-e60b-4a4d-a021-427b270bfbcb' | ACTIVE | | abee98c2-09c2-48b5-aed9-b715aa2c6eae | VM_cirros35_11_10.35.166.103 | fa:16:3e:51:3f:c1 | ip_address='10.0.1.5', subnet_id='675db1a0-0201-4d77-8b9f-58c40898f167' | ACTIVE | | | | | ip_address='2001::f816:3eff:fe51:3fc1', subnet_id='8d3bd293-2d7b-47be-8ae8-1af11ebc4ffd' | | | c5b3541a-260e-438f-a06d-eb75493ac291 | | fa:16:3e:87:ea:35 | ip_address='10.35.166.100', subnet_id='26953cf1-9ee0-47ac-a8a3-91d8c49e9c64' | ACTIVE | | d6ea0a47-c40c-49d4-8b14-1339b854b72d | | fa:16:3e:7f:f3:91 | ip_address='10.0.1.3', subnet_id='675db1a0-0201-4d77-8b9f-58c40898f167' | ACTIVE | | | | | ip_address='2001::f816:3eff:fe7f:f391', subnet_id='8d3bd293-2d7b-47be-8ae8-1af11ebc4ffd' | | | dc09f246-0aed-47d0-b5ef-7fc1acdc1a62 | VM_cirros35_21_10.35.166.108 | fa:16:3e:b1:36:87 | ip_address='10.0.2.12', subnet_id='f3b825af-082b-4df1-a73d-5aca0ef8cb1b' | ACTIVE | | | | | ip_address='2002::f816:3eff:feb1:3687', subnet_id='212674e2-af08-4605-8f5b-5951c032753a' | | | e738931b-aa28-4490-a96d-55c3510c1e43 | | fa:16:3e:25:5b:92 | ip_address='10.35.166.108', subnet_id='26953cf1-9ee0-47ac-a8a3-91d8c49e9c64' | N/A | | eccf18b4-11df-439f-a97a-e0142924cca6 | | fa:16:3e:bc:c3:36 | ip_address='10.0.2.3', subnet_id='f3b825af-082b-4df1-a73d-5aca0ef8cb1b' | ACTIVE | | | | | ip_address='2002::f816:3eff:febc:c336', subnet_id='212674e2-af08-4605-8f5b-5951c032753a' | | +--------------------------------------+-------------------------------------------------+-------------------+------------------------------------------------------------------------------------------+--------+ (overcloud) [stack@undercloud-0 ~]$ openstack port list | grep -A1 10.0.2.12 | dc09f246-0aed-47d0-b5ef-7fc1acdc1a62 | VM_cirros35_21_10.35.166.108 | fa:16:3e:b1:36:87 | ip_address='10.0.2.12', subnet_id='f3b825af-082b-4df1-a73d-5aca0ef8cb1b' | ACTIVE | | | | | ip_address='2002::f816:3eff:feb1:3687', subnet_id='212674e2-af08-4605-8f5b-5951c032753a' | | (overcloud) [stack@undercloud-0 ~]$ openstack port show dc09f246-0aed-47d0-b5ef-7fc1acdc1a62 +-----------------------+------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computesriov-1.localdomain | | binding_profile | | | binding_vif_details | datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2018-07-31T15:37:06Z | | data_plane_status | None | | description | | | device_id | ce5fee85-d885-48a9-8f99-ccb4c006b76b | | device_owner | compute:nova | | dns_assignment | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.0.2.12', subnet_id='f3b825af-082b-4df1-a73d-5aca0ef8cb1b' | | | ip_address='2002::f816:3eff:feb1:3687', subnet_id='212674e2-af08-4605-8f5b-5951c032753a' | | id | dc09f246-0aed-47d0-b5ef-7fc1acdc1a62 | | ip_address | None | | mac_address | fa:16:3e:b1:36:87 | | name | VM_cirros35_21_10.35.166.108 | | network_id | 8fdf42a4-e76a-4ad4-b7a1-c69927954dfe | | option_name | None | | option_value | None | | port_security_enabled | True | | project_id | d475a218e0ae41948d6549b5bee66b44 | | qos_policy_id | None | | revision_number | 13 | | security_group_ids | 662322aa-a72b-4645-8d89-deea20cbf09e | | status | ACTIVE | | subnet_id | None | | tags | | | trunk_details | None | | updated_at | 2018-07-31T15:38:02Z | +-----------------------+------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack security group show 662322aa-a72b-4645-8d89-deea20cbf09e +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2018-07-31T15:31:52Z | | description | sec_group | | id | 662322aa-a72b-4645-8d89-deea20cbf09e | | name | sec_group | | project_id | d475a218e0ae41948d6549b5bee66b44 | | revision_number | 6 | | rules | created_at='2018-07-31T15:31:58Z', direction='ingress', ethertype='IPv4', id='84eaf3e2-4c88-4d8b-89a0-dcef2ac1466d', port_range_max='80', port_range_min='80', protocol='tcp', remote_ip_prefix='0.0.0.0/0', updated_at='2018-07-31T15:31:58Z' | | | created_at='2018-07-31T15:31:52Z', direction='egress', ethertype='IPv6', id='c3b66bd0-eca3-48e2-88e5-94b25f1e26f6', updated_at='2018-07-31T15:31:52Z' | | | created_at='2018-07-31T15:32:03Z', direction='ingress', ethertype='IPv4', id='c46958d8-7023-4b5b-88fa-009d4ff88182', port_range_max='22', port_range_min='22', protocol='tcp', remote_ip_prefix='0.0.0.0/0', updated_at='2018-07-31T15:32:03Z' | | | created_at='2018-07-31T15:32:13Z', direction='ingress', ethertype='IPv4', id='ccf31045-2b07-44f5-9e8e-7b3e1e36379e', protocol='icmp', remote_ip_prefix='0.0.0.0/0', updated_at='2018-07-31T15:32:13Z' | | | created_at='2018-07-31T15:31:52Z', direction='egress', ethertype='IPv4', id='f20aaa06-6927-4652-84ca-612d88a9f39e', updated_at='2018-07-31T15:31:52Z' | | | created_at='2018-07-31T15:32:08Z', direction='ingress', ethertype='IPv4', id='f32f6814-a207-4cea-a8f3-f386a3bbd16c', port_range_max='443', port_range_min='443', protocol='tcp', remote_ip_prefix='0.0.0.0/0', updated_at='2018-07-31T15:32:08Z' | | updated_at | 2018-07-31T15:32:13Z | +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ ping 10.35.166.108 PING 10.35.166.108 (10.35.166.108) 56(84) bytes of data. From 10.35.166.108 icmp_seq=1 Destination Host Unreachable From 10.35.166.108 icmp_seq=2 Destination Host Unreachable From 10.35.166.108 icmp_seq=3 Destination Host Unreachable From 10.35.166.108 icmp_seq=4 Destination Host Unreachable --- 10.35.166.108 ping statistics --- 7 packets transmitted, 0 received, +4 errors, 100% packet loss, time 6001ms Expected results: The floating IP should be reachable Additional info:
Created attachment 1471889 [details] Same scenario on FLAT network - error does not happen Please note - running the same scenario, on an environment with FLAT (not vlan) network, all VMs fip are reachable.
You can use this script to reproduce scenario: https://code.engineering.redhat.com/gerrit/#/c/145740/1/Scripts/create_multi_topology.sh For example, you can run this script to create external vlan network, with 2 internal networks, where on each Network there will be 2 cirros vms. You may run it as is - and then supply input params, or with predefined params, like: ./create_multi_topology.sh -i cirros35 -t M -n 2 -m 2 -e vlan -c YES
sos reports from puma07 (vlan network): http://rhos-release.virt.bos.redhat.com/log/bz1610468
Slawek will deploy with the script given and see if he can reproduce the issue.
It may be the same issue as in upstream: https://bugs.launchpad.net/neutron/+bug/1787919
https://review.openstack.org/#/c/613396/ is the proposed upstream patch for this, in testing.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:0935