Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1662680 - Floating IP not reachable on second network interface
Summary: Floating IP not reachable on second network interface
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Slawek Kaplonski
QA Contact: Roee Agiman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-31 14:54 UTC by Noam Manos
Modified: 2019-01-09 15:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-09 15:03:12 UTC
Target Upstream Version:


Attachments (Terms of Use)
full console output + log traces of the non-pingable fip port (89.75 KB, text/plain)
2018-12-31 14:54 UTC, Noam Manos
no flags Details

Description Noam Manos 2018-12-31 14:54:00 UTC
Created attachment 1517666 [details]
full console output + log traces of the non-pingable fip port

Description of problem:
On OSP 14 (and probably on OSP13 as well), launching instance with 2 NICs, with 2 floating ips, will have only the first FIP reachable, but not the second FIP.

Version-Release number of selected component (if applicable):
OSP14   -p 2018-12-20.4

How reproducible:
Always

Steps to Reproduce:
1. Install OSP
2. Launch a VM with 2 NICs
3. Assign FIP to each NIC
4. Ping each FIP

You can use this script to do steps 2-4: 
https://code.engineering.redhat.com/gerrit/gitweb?p=Neutron-QE.git;a=blob;f=Scripts/create_multi_topology.sh

./create_multi_topology.sh -q -i cirros35 -t mni -e skip -n 2 -v 1 -c YES

Actual results:


[2018-12-31 09:09:19] Creating Router and 2 Networks - each one with both IPv4 and IPv6 Subnets:

+-------------------------+--------------------------------------+
| Field                   | Value                                |
+-------------------------+--------------------------------------+
| admin_state_up          | UP                                   |
| availability_zone_hints | None                                 |
| availability_zones      | None                                 |
| created_at              | 2018-12-31T14:09:21Z                 |
| description             |                                      |
| distributed             | None                                 |
| external_gateway_info   | None                                 |
| flavor_id               | None                                 |
| ha                      | None                                 |
| id                      | 1ab38f3f-5c45-4b30-a47a-cee93cc9cffc |
| name                    | Router_eNet                          |
| project_id              | a820cebba90f4bd296e6e846e9f03d17     |
| revision_number         | 0                                    |
| routes                  |                                      |
| status                  | ACTIVE                               |
| tags                    |                                      |
| updated_at              | 2018-12-31T14:09:21Z                 |
+-------------------------+--------------------------------------+

[2018-12-31 09:09:25] Creating Network net_ipv64_1:

+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-12-31T14:09:28Z                 |
| description               |                                      |
| dns_domain                |                                      |
| id                        | 8078ff43-69f3-4633-8c65-7240d199963a |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1442                                 |
| name                      | net_ipv64_1                          |
| port_security_enabled     | True                                 |
| project_id                | a820cebba90f4bd296e6e846e9f03d17     |
| provider:network_type     | None                                 |
| provider:physical_network | None                                 |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 2                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2018-12-31T14:09:28Z                 |
+---------------------------+--------------------------------------+

[2018-12-31 09:09:29] Creating ipv4 Subnet on net_ipv64_1 - subnet_ipv4_1:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 10.0.1.2-10.0.1.254                  |
| cidr              | 10.0.1.0/24                          |
| created_at        | 2018-12-31T14:09:33Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 10.0.1.1                             |
| host_routes       |                                      |
| id                | bcb38bbf-86e2-4754-b51a-503967b1f251 |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | subnet_ipv4_1                        |
| network_id        | 8078ff43-69f3-4633-8c65-7240d199963a |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-31T14:09:33Z                 |
+-------------------+--------------------------------------+

[2018-12-31 09:09:35] Creating ipv6 Subnet on net_ipv64_1 - subnet_ipv6_1:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 2001::2-2001::ffff:ffff:ffff:ffff    |
| cidr              | 2001::/64                            |
| created_at        | 2018-12-31T14:09:39Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 2001::1                              |
| host_routes       |                                      |
| id                | 5cb796bc-c136-4f3c-ba89-4bf4c19beb54 |
| ip_version        | 6                                    |
| ipv6_address_mode | slaac                                |
| ipv6_ra_mode      | slaac                                |
| name              | subnet_ipv6_1                        |
| network_id        | 8078ff43-69f3-4633-8c65-7240d199963a |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-31T14:09:39Z                 |
+-------------------+--------------------------------------+

[2018-12-31 09:09:40] Adding subnet_ipv4_1 and subnet_ipv6_1 to the router.


[2018-12-31 09:09:56] Creating Network net_ipv64_2:

+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-12-31T14:09:59Z                 |
| description               |                                      |
| dns_domain                |                                      |
| id                        | 3505cd39-e760-46fc-9a33-58f7efd4cc6c |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1442                                 |
| name                      | net_ipv64_2                          |
| port_security_enabled     | True                                 |
| project_id                | a820cebba90f4bd296e6e846e9f03d17     |
| provider:network_type     | None                                 |
| provider:physical_network | None                                 |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 2                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2018-12-31T14:09:59Z                 |
+---------------------------+--------------------------------------+

[2018-12-31 09:10:00] Creating ipv4 Subnet on net_ipv64_2 - subnet_ipv4_2:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 10.0.2.2-10.0.2.254                  |
| cidr              | 10.0.2.0/24                          |
| created_at        | 2018-12-31T14:10:04Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 10.0.2.1                             |
| host_routes       |                                      |
| id                | 23d79776-addc-436d-88e4-8458eb2ee98f |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | subnet_ipv4_2                        |
| network_id        | 3505cd39-e760-46fc-9a33-58f7efd4cc6c |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-31T14:10:04Z                 |
+-------------------+--------------------------------------+

[2018-12-31 09:10:06] Creating ipv6 Subnet on net_ipv64_2 - subnet_ipv6_2:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 2002::2-2002::ffff:ffff:ffff:ffff    |
| cidr              | 2002::/64                            |
| created_at        | 2018-12-31T14:10:10Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 2002::1                              |
| host_routes       |                                      |
| id                | 24ec6c15-a7fc-47db-bffc-29228d458329 |
| ip_version        | 6                                    |
| ipv6_address_mode | slaac                                |
| ipv6_ra_mode      | slaac                                |
| name              | subnet_ipv6_2                        |
| network_id        | 3505cd39-e760-46fc-9a33-58f7efd4cc6c |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-31T14:10:10Z                 |
+-------------------+--------------------------------------+

[2018-12-31 09:10:12] Adding subnet_ipv4_2 and subnet_ipv6_2 to the router.


[2018-12-31 09:10:28] Connecting the router to the external network "public"


[2018-12-31 09:10:37] Creating security group rules for group "sec_group"

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-31T14:10:43Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | 0d21d7bf-e8b3-46b8-a78c-682cdd889dca |
| name              | None                                 |
| port_range_max    | 80                                   |
| port_range_min    | 80                                   |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| updated_at        | 2018-12-31T14:10:43Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-31T14:10:47Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | da7dd7ea-7e1b-4cd4-a027-7baf585e5d22 |
| name              | None                                 |
| port_range_max    | 22                                   |
| port_range_min    | 22                                   |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| updated_at        | 2018-12-31T14:10:47Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-31T14:10:50Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | 83c7e770-9923-4c0d-8f2b-361461d76e89 |
| name              | None                                 |
| port_range_max    | 443                                  |
| port_range_min    | 443                                  |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| updated_at        | 2018-12-31T14:10:50Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-31T14:10:54Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | 09b9cebd-4d3e-4993-a7a2-703fc4dfa111 |
| name              | None                                 |
| port_range_max    | None                                 |
| port_range_min    | None                                 |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| protocol          | icmp                                 |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| updated_at        | 2018-12-31T14:10:54Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-31T14:10:57Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv6                                 |
| id                | b89e0c8f-42ed-40e0-a527-f63ace21503a |
| name              | None                                 |
| port_range_max    | None                                 |
| port_range_min    | None                                 |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | None                                 |
| revision_number   | 0                                    |
| security_group_id | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| updated_at        | 2018-12-31T14:10:57Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-31T14:11:01Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv6                                 |
| id                | 9b27777d-dc06-45de-b90b-b5f09e78745b |
| name              | None                                 |
| port_range_max    | None                                 |
| port_range_min    | None                                 |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| protocol          | udp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | None                                 |
| revision_number   | 0                                    |
| security_group_id | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| updated_at        | 2018-12-31T14:11:01Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-31T14:11:04Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv6                                 |
| id                | abb91775-fa54-4688-9b32-042065784fa3 |
| name              | None                                 |
| port_range_max    | None                                 |
| port_range_min    | None                                 |
| project_id        | a820cebba90f4bd296e6e846e9f03d17     |
| protocol          | icmp                                 |
| remote_group_id   | None                                 |
| remote_ip_prefix  | None                                 |
| revision_number   | 0                                    |
| security_group_id | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| updated_at        | 2018-12-31T14:11:04Z                 |
+-------------------+--------------------------------------+
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range  | Port Range | Remote Security Group                | Security Group                       |
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+
| 09b9cebd-4d3e-4993-a7a2-703fc4dfa111 | icmp        | 0.0.0.0/0 |            | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| 0d21d7bf-e8b3-46b8-a78c-682cdd889dca | tcp         | 0.0.0.0/0 | 80:80      | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| 32adbc95-bf38-411f-88e0-6cb0d2f12b3f | None        | None      |            | None                                 | cbea8754-8fff-4358-b099-365161ad748d |
| 55081afd-3c16-4a58-a4b5-86f5f42e0f8d | None        | None      |            | None                                 | cbea8754-8fff-4358-b099-365161ad748d |
| 83c7e770-9923-4c0d-8f2b-361461d76e89 | tcp         | 0.0.0.0/0 | 443:443    | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| 8abe0993-d058-4817-9151-63928462bc1d | None        | None      |            | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| 9b27777d-dc06-45de-b90b-b5f09e78745b | udp         | None      |            | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| 9d5f5431-2971-448c-99e8-2b85db64c1ae | None        | None      |            | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| abb91775-fa54-4688-9b32-042065784fa3 | icmp        | None      |            | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| b89e0c8f-42ed-40e0-a527-f63ace21503a | tcp         | None      |            | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
| cd54dacc-bd16-4378-bfcd-2ecfec716f2e | None        | None      |            | cbea8754-8fff-4358-b099-365161ad748d | cbea8754-8fff-4358-b099-365161ad748d |
| d4e8ed7d-d1b9-47b0-a78f-92afbc560606 | None        | None      |            | cbea8754-8fff-4358-b099-365161ad748d | cbea8754-8fff-4358-b099-365161ad748d |
| da7dd7ea-7e1b-4cd4-a027-7baf585e5d22 | tcp         | 0.0.0.0/0 | 22:22      | None                                 | eaf10726-f9f4-43ce-a40f-63b9e60c5735 |
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+

[2018-12-31 09:11:07] Creating openstack key pair to easily login into VMs:

+-------------+-------------------------------------------------+
| Field       | Value                                           |
+-------------+-------------------------------------------------+
| fingerprint | ba:f6:07:22:e7:42:66:8d:5b:83:83:0b:a5:6d:64:b7 |
| name        | tester-key                                      |
| user_id     | 6c0e03c0e66c4288b0fa03592faec0fc                |
+-------------+-------------------------------------------------+
+------------+-------------------------------------------------+
| Name       | Fingerprint                                     |
+------------+-------------------------------------------------+
| tester-key | ba:f6:07:22:e7:42:66:8d:5b:83:83:0b:a5:6d:64:b7 |
+------------+-------------------------------------------------+

[2018-12-31 09:11:16] Creating 1 VM instanses.


[2018-12-31 09:11:21] Creating and booting VM instance with 2 NICs: cirros35_vm1

+-----------------------------+------------------------------------------------------+
| Field                       | Value                                                |
+-----------------------------+------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                               |
| OS-EXT-AZ:availability_zone |                                                      |
| OS-EXT-STS:power_state      | NOSTATE                                              |
| OS-EXT-STS:task_state       | scheduling                                           |
| OS-EXT-STS:vm_state         | building                                             |
| OS-SRV-USG:launched_at      | None                                                 |
| OS-SRV-USG:terminated_at    | None                                                 |
| accessIPv4                  |                                                      |
| accessIPv6                  |                                                      |
| addresses                   |                                                      |
| adminPass                   | YcTkfpHG4iyy                                         |
| config_drive                |                                                      |
| created                     | 2018-12-31T14:11:35Z                                 |
| flavor                      | cirros_flavor (8c351db5-3e7c-443c-80c4-477c513b54e6) |
| hostId                      |                                                      |
| id                          | 2db24404-1fa2-4e76-8bfd-a8389bd4ccf4                 |
| image                       | cirros35 (101123d6-f0d8-4970-b1c2-92ee996410a1)      |
| key_name                    | tester-key                                           |
| name                        | cirros35_vm1                                         |
| progress                    | 0                                                    |
| project_id                  | a820cebba90f4bd296e6e846e9f03d17                     |
| properties                  |                                                      |
| security_groups             | name='eaf10726-f9f4-43ce-a40f-63b9e60c5735'          |
| status                      | BUILD                                                |
| updated                     | 2018-12-31T14:11:36Z                                 |
| user_id                     | 6c0e03c0e66c4288b0fa03592faec0fc                     |
| volumes_attached            |                                                      |
+-----------------------------+------------------------------------------------------+
| progress                    | 0                                                                                                 |
| project_id                  | a820cebba90f4bd296e6e846e9f03d17                                                                  |
| properties                  |                                                                                                   |
| security_groups             | name='sec_group'                                                                                  |
|                             | name='sec_group'                                                                                  |
| status                      | ACTIVE                                                                                            |

[2018-12-31 09:12:17] Configuring Networks for each of the 2 NICs: net_ipv64_2=10.0.2.3,  net_ipv64_1= 10.0.1.10


[2018-12-31 09:12:25] Adding floating ip 10.0.0.210 to cirros35_vm1, and connecting to IP 10.0.1.10 (Port 99a072af-08f1-44f1-8e72-e45881a74db0).


[2018-12-31 09:12:41] Setting a NAME to the new floating ip port: "cirros35_vm1_10.0.0.210"

+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                                    |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                                                                                       |
| allowed_address_pairs |                                                                                                                                          |
| binding_host_id       | None                                                                                                                                     |
| binding_profile       | None                                                                                                                                     |
| binding_vif_details   | None                                                                                                                                     |
| binding_vif_type      | None                                                                                                                                     |
| binding_vnic_type     | normal                                                                                                                                   |
| created_at            | 2018-12-31T14:11:41Z                                                                                                                     |
| data_plane_status     | None                                                                                                                                     |
| description           |                                                                                                                                          |
| device_id             | 2db24404-1fa2-4e76-8bfd-a8389bd4ccf4                                                                                                     |
| device_owner          | compute:nova                                                                                                                             |
| dns_assignment        | fqdn='host-10-0-1-10.openstacklocal.', hostname='host-10-0-1-10', ip_address='10.0.1.10'                                                 |
|                       | fqdn='host-2001--f816-3eff-fec3-c859.openstacklocal.', hostname='host-2001--f816-3eff-fec3-c859', ip_address='2001::f816:3eff:fec3:c859' |
| dns_domain            | None                                                                                                                                     |
| dns_name              |                                                                                                                                          |
| extra_dhcp_opts       |                                                                                                                                          |
| fixed_ips             | ip_address='10.0.1.10', subnet_id='bcb38bbf-86e2-4754-b51a-503967b1f251'                                                                 |
|                       | ip_address='2001::f816:3eff:fec3:c859', subnet_id='5cb796bc-c136-4f3c-ba89-4bf4c19beb54'                                                 |
| id                    | 99a072af-08f1-44f1-8e72-e45881a74db0                                                                                                     |
| mac_address           | fa:16:3e:c3:c8:59                                                                                                                        |
| name                  | cirros35_vm1_10.0.0.210                                                                                                                  |
| network_id            | 8078ff43-69f3-4633-8c65-7240d199963a                                                                                                     |
| port_security_enabled | True                                                                                                                                     |
| project_id            | a820cebba90f4bd296e6e846e9f03d17                                                                                                         |
| qos_policy_id         | None                                                                                                                                     |
| revision_number       | 6                                                                                                                                        |
| security_group_ids    | eaf10726-f9f4-43ce-a40f-63b9e60c5735                                                                                                     |
| status                | ACTIVE                                                                                                                                   |
| tags                  |                                                                                                                                          |
| trunk_details         | None                                                                                                                                     |
| updated_at            | 2018-12-31T14:12:45Z                                                                                                                     |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+

[2018-12-31 09:12:48] Waiting for port status to be ACTIVE on cirros35_vm1, with internal IP address 10.0.1.10:


| dns_domain            | None                                                                                                                                     |
| dns_name              |                                                                                                                                          |
| extra_dhcp_opts       |                                                                                                                                          |
| fixed_ips             | ip_address='10.0.1.10', subnet_id='bcb38bbf-86e2-4754-b51a-503967b1f251'                                                                 |
|                       | ip_address='2001::f816:3eff:fec3:c859', subnet_id='5cb796bc-c136-4f3c-ba89-4bf4c19beb54'                                                 |
| id                    | 99a072af-08f1-44f1-8e72-e45881a74db0                                                                                                     |
| mac_address           | fa:16:3e:c3:c8:59                                                                                                                        |
| name                  | cirros35_vm1_10.0.0.210                                                                                                                  |
| network_id            | 8078ff43-69f3-4633-8c65-7240d199963a                                                                                                     |
| port_security_enabled | True                                                                                                                                     |
| project_id            | a820cebba90f4bd296e6e846e9f03d17                                                                                                         |
| qos_policy_id         | None                                                                                                                                     |
| revision_number       | 6                                                                                                                                        |
| security_group_ids    | eaf10726-f9f4-43ce-a40f-63b9e60c5735                                                                                                     |
| status                | ACTIVE                                                                                                                                   |

[2018-12-31 09:12:51] Pinging the new FIP 10.0.0.210 from Undercloud (North-South test):

PING 10.0.0.210 (10.0.0.210) 56(84) bytes of data.
64 bytes from 10.0.0.210: icmp_seq=2 ttl=63 time=2.13 ms
64 bytes from 10.0.0.210: icmp_seq=3 ttl=63 time=1.36 ms
64 bytes from 10.0.0.210: icmp_seq=4 ttl=63 time=0.930 ms
64 bytes from 10.0.0.210: icmp_seq=5 ttl=63 time=0.969 ms
64 bytes from 10.0.0.210: icmp_seq=6 ttl=63 time=0.887 ms

--- 10.0.0.210 ping statistics ---
6 packets transmitted, 5 received, 16% packet loss, time 5003ms
rtt min/avg/max/mdev = 0.887/1.256/2.135/0.472 ms
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to 10.0.0.210:80; Connection refused

[2018-12-31 09:12:56] Generating ssh key to access cirros35_vm1 on 10.0.0.210:

ssh-keygen -f ~/.ssh/known_hosts -R $fip
Host 10.0.0.210 not found in /home/stack/.ssh/known_hosts

[2018-12-31 09:12:56] Checking inside cirros35_vm1 uptime and pinging 8.8.8.8 (SNAT test):

COUNT=0; until ssh -i $KEY_FILE -o StrictHostKeyChecking=no $ssh_user@$fip "ping 8.8.8.8 -c 1 && uptime" || [ $COUNT -eq 5 ]; do echo $(( COUNT++ )); done
Warning: Permanently added '10.0.0.210' (RSA) to the list of known hosts.
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=50.331 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 50.331/50.331/50.331 ms
 14:12:56 up 0 min,  0 users,  load average: 0.00, 0.00, 0.00

[2018-12-31 09:13:05] Adding floating ip 10.0.0.237 to cirros35_vm1, and connecting to IP 10.0.2.3 (Port 73b6ee55-39f5-41f8-af03-b37109d7a758).


[2018-12-31 09:13:20] Setting a NAME to the new floating ip port: "cirros35_vm1_10.0.0.237"

+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                                    |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                                                                                       |
| allowed_address_pairs |                                                                                                                                          |
| binding_host_id       | None                                                                                                                                     |
| binding_profile       | None                                                                                                                                     |
| binding_vif_details   | None                                                                                                                                     |
| binding_vif_type      | None                                                                                                                                     |
| binding_vnic_type     | normal                                                                                                                                   |
| created_at            | 2018-12-31T14:11:43Z                                                                                                                     |
| data_plane_status     | None                                                                                                                                     |
| description           |                                                                                                                                          |
| device_id             | 2db24404-1fa2-4e76-8bfd-a8389bd4ccf4                                                                                                     |
| device_owner          | compute:nova                                                                                                                             |
| dns_assignment        | fqdn='host-10-0-2-3.openstacklocal.', hostname='host-10-0-2-3', ip_address='10.0.2.3'                                                    |
|                       | fqdn='host-2002--f816-3eff-fe0b-678b.openstacklocal.', hostname='host-2002--f816-3eff-fe0b-678b', ip_address='2002::f816:3eff:fe0b:678b' |
| dns_domain            | None                                                                                                                                     |
| dns_name              |                                                                                                                                          |
| extra_dhcp_opts       |                                                                                                                                          |
| fixed_ips             | ip_address='10.0.2.3', subnet_id='23d79776-addc-436d-88e4-8458eb2ee98f'                                                                  |
|                       | ip_address='2002::f816:3eff:fe0b:678b', subnet_id='24ec6c15-a7fc-47db-bffc-29228d458329'                                                 |
| id                    | 73b6ee55-39f5-41f8-af03-b37109d7a758                                                                                                     |
| mac_address           | fa:16:3e:0b:67:8b                                                                                                                        |
| name                  | cirros35_vm1_10.0.0.237                                                                                                                  |
| network_id            | 3505cd39-e760-46fc-9a33-58f7efd4cc6c                                                                                                     |
| port_security_enabled | True                                                                                                                                     |
| project_id            | a820cebba90f4bd296e6e846e9f03d17                                                                                                         |
| qos_policy_id         | None                                                                                                                                     |
| revision_number       | 6                                                                                                                                        |
| security_group_ids    | eaf10726-f9f4-43ce-a40f-63b9e60c5735                                                                                                     |
| status                | ACTIVE                                                                                                                                   |
| tags                  |                                                                                                                                          |
| trunk_details         | None                                                                                                                                     |
| updated_at            | 2018-12-31T14:13:24Z                                                                                                                     |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------+

[2018-12-31 09:13:27] Waiting for port status to be ACTIVE on cirros35_vm1, with internal IP address 10.0.2.3:


| dns_domain            | None                                                                                                                                     |
| dns_name              |                                                                                                                                          |
| extra_dhcp_opts       |                                                                                                                                          |
| fixed_ips             | ip_address='10.0.2.3', subnet_id='23d79776-addc-436d-88e4-8458eb2ee98f'                                                                  |
|                       | ip_address='2002::f816:3eff:fe0b:678b', subnet_id='24ec6c15-a7fc-47db-bffc-29228d458329'                                                 |
| id                    | 73b6ee55-39f5-41f8-af03-b37109d7a758                                                                                                     |
| mac_address           | fa:16:3e:0b:67:8b                                                                                                                        |
| name                  | cirros35_vm1_10.0.0.237                                                                                                                  |
| network_id            | 3505cd39-e760-46fc-9a33-58f7efd4cc6c                                                                                                     |
| port_security_enabled | True                                                                                                                                     |
| project_id            | a820cebba90f4bd296e6e846e9f03d17                                                                                                         |
| qos_policy_id         | None                                                                                                                                     |
| revision_number       | 6                                                                                                                                        |
| security_group_ids    | eaf10726-f9f4-43ce-a40f-63b9e60c5735                                                                                                     |
| status                | ACTIVE                                                                                                                                   |

[2018-12-31 09:13:29] Pinging the new FIP 10.0.0.237 from Undercloud (North-South test):

PING 10.0.0.237 (10.0.0.237) 56(84) bytes of data.

--- 10.0.0.237 ping statistics ---
30 packets transmitted, 0 received, 100% packet loss, time 29003ms


Expected results:
Both NIC should get reachable FIPs

Additional info:
It was also reproduced on OSP13.

Comment 1 Noam Manos 2018-12-31 15:00:13 UTC
sos reports:
http://rhos-release.virt.bos.redhat.com/log/bz1662680

Comment 2 Brian Haley 2019-01-03 16:30:46 UTC
This might be a problem with routing on the instance where packets are coming/going from the "wrong" interfaces.  Slawek will try and reproduce it.

Comment 3 Slawek Kaplonski 2019-01-09 13:09:33 UTC
Hi, I tested it locally on devstack running with Neutron from master branch. All worked fine for me. Only problem which I had was that eth1 interface wasn't UP and configured by default. So I had to SSH to my VM and run dhclient on eth1 interface. See below:

Interfaces on my vm:

[14:01:13] vagrant@devstack-ubuntu-ovs ~ $ nova interface-list vm2_nics
+------------+--------------------------------------+--------------------------------------+----------------------+-------------------+
| Port State | Port ID                              | Net ID                               | IP addresses         | MAC Addr          |
+------------+--------------------------------------+--------------------------------------+----------------------+-------------------+
| ACTIVE     | 4b4178e4-7b14-443c-83ac-ac61d66f1bcb | 99a4d1f6-ba2a-4084-8287-09ecabce546e | 10.0.2.110,2002::1ef | fa:16:3e:40:d5:ec |
| ACTIVE     | ea04a6ba-d595-4c6d-a512-d47bc28fe05d | 62b3974e-7e6a-43b5-9f3a-b045f22ea794 | 10.0.1.88,2001::92   | fa:16:3e:03:06:08 |
+------------+--------------------------------------+--------------------------------------+----------------------+-------------------+

Floating IPs:
[14:07:39] vagrant@devstack-ubuntu-ovs ~ $ neutron floatingip-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+----------------------------------+------------------+---------------------+--------------------------------------+
| id                                   | tenant_id                        | fixed_ip_address | floating_ip_address | port_id                              |
+--------------------------------------+----------------------------------+------------------+---------------------+--------------------------------------+
| 50722b42-3150-4bfc-9954-c72c5d619872 | 72df2ab5d6544e1984feddd08e24cac6 | 10.0.2.110       | 10.10.0.212         | 4b4178e4-7b14-443c-83ac-ac61d66f1bcb |
| c47070f9-8312-4fb8-9a94-5749e3953409 | 72df2ab5d6544e1984feddd08e24cac6 | 10.0.1.88        | 10.10.0.192         | ea04a6ba-d595-4c6d-a512-d47bc28fe05d |
+--------------------------------------+----------------------------------+------------------+---------------------+--------------------------------------+



Network config on cirros vm:
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:03:06:08 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.88/24 brd 10.0.1.255 scope global eth0
    inet6 fe80::f816:3eff:fe03:608/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:40:d5:ec brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fe40:d5ec/64 scope link
       valid_lft forever preferred_lft forever
$ sudo /sbin/cirros-dhcpc up eth1
udhcpc (v1.20.1) started
Sending discover...
Sending select for 10.0.2.110...
Lease of 10.0.2.110 obtained, lease time 86400
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.0.2.1"
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:03:06:08 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.88/24 brd 10.0.1.255 scope global eth0
    inet6 fe80::f816:3eff:fe03:608/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:40:d5:ec brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.110/24 brd 10.0.2.255 scope global eth1
    inet6 fe80::f816:3eff:fe40:d5ec/64 scope link
       valid_lft forever preferred_lft forever
$


And now ping to each of FIPs:

[14:08:04] vagrant@devstack-ubuntu-ovs ~ $ ping 10.10.0.212 -c 1
PING 10.10.0.212 (10.10.0.212) 56(84) bytes of data.
64 bytes from 10.10.0.212: icmp_seq=1 ttl=63 time=1.59 ms

--- 10.10.0.212 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.596/1.596/1.596/0.000 ms
[14:08:08] vagrant@devstack-ubuntu-ovs ~ $ ping 10.10.0.192 -c 1
PING 10.10.0.192 (10.10.0.192) 56(84) bytes of data.
64 bytes from 10.10.0.192: icmp_seq=1 ttl=63 time=1.44 ms

--- 10.10.0.192 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.445/1.445/1.445/0.000 ms


I think we can close this as not a bug because it's depends on guest OS configuration how it will work.

Comment 4 Bernard Cafarelli 2019-01-09 15:03:12 UTC
Closing per comment 3 I had similar issues in the past (which I workaround with additional SSH commands to Cirros VMs where I setup eth1)


Note You need to log in before you can comment on or make changes to this bug.