Bug 1662042 - [Netvirt] Floating ip unreachable: port admin_state_up=True while status=DOWN
Summary: [Netvirt] Floating ip unreachable: port admin_state_up=True while status=DOWN
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 14.0 (Rocky)
Assignee: Aswin Suryanarayanan
QA Contact: Noam Manos
URL:
Whiteboard: Netvirt
Depends On:
Blocks: 1656091
TreeView+ depends on / blocked
 
Reported: 2018-12-25 13:45 UTC by Noam Manos
Modified: 2019-03-06 16:17 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Release Note
Doc Text:
OpenDaylight does not support IPv6 for tenant or provider networks. Therefore, use only IPv4 networks. You may experience issues related to floating IPs if IPv6 networks are used along with IPv4 networks.
Clone Of:
Environment:
Last Closed: 2019-03-06 16:14:37 UTC
Target Upstream Version:


Attachments (Terms of Use)
Bug was reproduced on a new OSP14 env - full output + log traces attached. (197.77 KB, text/plain)
2019-01-02 19:26 UTC, Noam Manos
no flags Details

Description Noam Manos 2018-12-25 13:45:46 UTC
Description of problem:
On OSP14 with ODL, trying to ping a new cirros instance fails. dhcp-agent.log shows: "port admin_state_up=True" while "status=DOWN".

Version-Release number of selected component (if applicable):
14   -p 2018-12-20.4

How reproducible:
Sometimes

Steps to Reproduce:
1. Install OSP + ODL
2. Launch cirros VM with floating ip, and ping it.

You can use following script to do so:
https://code.engineering.redhat.com/gerrit/gitweb?p=Neutron-QE.git;a=blob;f=Scripts/create_multi_topology.sh

./create_multi_topology.sh -q -i cirros35 -t mvi -e skip -n 2 -v 2


Actual results:

(overcloud) Deleting Tenant test project and user

No project test_cloud
No user tester
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-12-25T10:23:50Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 4f53e5d4-4132-44fb-8465-bbf12c4fd682 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1500                                 |
| name                      | nova                                 |
| port_security_enabled     | True                                 |
| project_id                | 70b82a4868bd42ccb36261613ac4d178     |
| provider:network_type     | flat                                 |
| provider:physical_network | datacentre                           |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 1                                    |
| router:external           | External                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2018-12-25T10:23:50Z                 |
+---------------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 10.0.0.210-10.0.0.250                |
| cidr              | 10.0.0.0/24                          |
| created_at        | 2018-12-25T10:23:54Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | False                                |
| gateway_ip        | 10.0.0.1                             |
| host_routes       |                                      |
| id                | 8bb7610c-af1f-4f31-a285-501971aadd11 |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | nova                                 |
| network_id        | 4f53e5d4-4132-44fb-8465-bbf12c4fd682 |
| project_id        | 70b82a4868bd42ccb36261613ac4d178     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-25T10:23:54Z                 |
+-------------------+--------------------------------------+

(overcloud) Creating CirrOS 0.3.5 Image:

--2018-12-25 05:23:58--  https://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
Resolving download.cirros-cloud.net (download.cirros-cloud.net)... 2607:f298:6:a036::bd6:a72a, 64.90.42.85
Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|2607:f298:6:a036::bd6:a72a|:443... failed: No route to host.
Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|64.90.42.85|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13267968 (13M) [text/plain]
Saving to: ‘cirros-0.3.5-x86_64-disk.img’

2018-12-25 05:24:02 (4.35 MB/s) - ‘cirros-0.3.5-x86_64-disk.img’ saved [13267968/13267968]

+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                                                                                                                    |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | f8ab98ff5e73ebab884d80c9dc9c7290                                                                                                                                                                                                                                         |
| container_format | bare                                                                                                                                                                                                                                                                     |
| created_at       | 2018-12-25T10:24:06Z                                                                                                                                                                                                                                                     |
| disk_format      | qcow2                                                                                                                                                                                                                                                                    |
| file             | /v2/images/d59b98b8-c136-4c27-a6ea-9c0b4ff80da6/file                                                                                                                                                                                                                     |
| id               | d59b98b8-c136-4c27-a6ea-9c0b4ff80da6                                                                                                                                                                                                                                     |
| min_disk         | 0                                                                                                                                                                                                                                                                        |
| min_ram          | 0                                                                                                                                                                                                                                                                        |
| name             | cirros35                                                                                                                                                                                                                                                                 |
| owner            | 70b82a4868bd42ccb36261613ac4d178                                                                                                                                                                                                                                         |
| properties       | direct_url='swift+config://ref1/glance/d59b98b8-c136-4c27-a6ea-9c0b4ff80da6', os_hash_algo='sha512', os_hash_value='f0fd1b50420dce4ca382ccfbb528eef3a38bbeff00b54e95e3876b9bafe7ed2d6f919ca35d9046d437c6d2d8698b1174a335fbd66035bb3edc525d2cdb187232', os_hidden='False' |
| protected        | False                                                                                                                                                                                                                                                                    |
| schema           | /v2/schemas/image                                                                                                                                                                                                                                                        |
| size             | 13267968                                                                                                                                                                                                                                                                 |
| status           | active                                                                                                                                                                                                                                                                   |
| tags             |                                                                                                                                                                                                                                                                          |
| updated_at       | 2018-12-25T10:24:09Z                                                                                                                                                                                                                                                     |
| virtual_size     | None                                                                                                                                                                                                                                                                     |
| visibility       | public                                                                                                                                                                                                                                                                   |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

(overcloud) Creating CirrOS Flavor:

No flavor with a name or ID of 'cirros_flavor' exists.
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 1                                    |
| id                         | e6096fca-de55-4a82-8522-0d9c81b91aad |
| name                       | cirros_flavor                        |
| os-flavor-access:is_public | True                                 |
| properties                 |                                      |
| ram                        | 512                                  |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 1                                    |
+----------------------------+--------------------------------------+


[05:24:39.063] (tester) Creating Router and 2 Networks - each one with both IPv4 and IPv6 Subnets:

+-------------------------+--------------------------------------+
| Field                   | Value                                |
+-------------------------+--------------------------------------+
| admin_state_up          | UP                                   |
| availability_zone_hints | None                                 |
| availability_zones      | None                                 |
| created_at              | 2018-12-25T10:24:41Z                 |
| description             |                                      |
| distributed             | None                                 |
| external_gateway_info   | None                                 |
| flavor_id               | None                                 |
| ha                      | None                                 |
| id                      | 4d04585a-1cf5-436c-9f7f-05c9b8546dbf |
| name                    | Router_eNet                          |
| project_id              | 09ee7dd1bff648b099a8fca98f49738c     |
| revision_number         | 0                                    |
| routes                  |                                      |
| status                  | ACTIVE                               |
| tags                    |                                      |
| updated_at              | 2018-12-25T10:24:41Z                 |
+-------------------------+--------------------------------------+

[05:24:44.884] (tester) Creating Network net_ipv64_1:

+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-12-25T10:24:47Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | ca9e10da-65c4-4deb-bab5-5481fae07f7a |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1450                                 |
| name                      | net_ipv64_1                          |
| port_security_enabled     | True                                 |
| project_id                | 09ee7dd1bff648b099a8fca98f49738c     |
| provider:network_type     | None                                 |
| provider:physical_network | None                                 |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 1                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2018-12-25T10:24:47Z                 |
+---------------------------+--------------------------------------+

[05:24:48.250] (tester) Creating ipv4 Subnet on net_ipv64_1 - subnet_ipv4_1:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 10.0.1.2-10.0.1.254                  |
| cidr              | 10.0.1.0/24                          |
| created_at        | 2018-12-25T10:24:52Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 10.0.1.1                             |
| host_routes       |                                      |
| id                | b23b3221-0df2-41c7-bec5-02f0cd700f52 |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | subnet_ipv4_1                        |
| network_id        | ca9e10da-65c4-4deb-bab5-5481fae07f7a |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-25T10:24:52Z                 |
+-------------------+--------------------------------------+

[05:24:53.190] (tester) Creating ipv6 Subnet on net_ipv64_1 - subnet_ipv6_1:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 2001::2-2001::ffff:ffff:ffff:ffff    |
| cidr              | 2001::/64                            |
| created_at        | 2018-12-25T10:24:58Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 2001::1                              |
| host_routes       |                                      |
| id                | 51bcd214-3431-467b-8261-a90a59aff0bc |
| ip_version        | 6                                    |
| ipv6_address_mode | slaac                                |
| ipv6_ra_mode      | slaac                                |
| name              | subnet_ipv6_1                        |
| network_id        | ca9e10da-65c4-4deb-bab5-5481fae07f7a |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-25T10:24:58Z                 |
+-------------------+--------------------------------------+

[05:25:00.327] (tester) Adding subnet_ipv4_1 and subnet_ipv6_1 to the router.


[05:25:14.713] (tester) Creating Network net_ipv64_2:

+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-12-25T10:25:17Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | c83a0fbc-558e-4f26-bf13-1607d9ef7de3 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1450                                 |
| name                      | net_ipv64_2                          |
| port_security_enabled     | True                                 |
| project_id                | 09ee7dd1bff648b099a8fca98f49738c     |
| provider:network_type     | None                                 |
| provider:physical_network | None                                 |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | 1                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2018-12-25T10:25:17Z                 |
+---------------------------+--------------------------------------+

[05:25:17.742] (tester) Creating ipv4 Subnet on net_ipv64_2 - subnet_ipv4_2:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 10.0.2.2-10.0.2.254                  |
| cidr              | 10.0.2.0/24                          |
| created_at        | 2018-12-25T10:25:22Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 10.0.2.1                             |
| host_routes       |                                      |
| id                | 6073882b-a028-48fa-b224-68746aa4ac52 |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | subnet_ipv4_2                        |
| network_id        | c83a0fbc-558e-4f26-bf13-1607d9ef7de3 |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-25T10:25:22Z                 |
+-------------------+--------------------------------------+

[05:25:22.655] (tester) Creating ipv6 Subnet on net_ipv64_2 - subnet_ipv6_2:

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 2002::2-2002::ffff:ffff:ffff:ffff    |
| cidr              | 2002::/64                            |
| created_at        | 2018-12-25T10:25:27Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 2002::1                              |
| host_routes       |                                      |
| id                | ef96249a-b921-44c7-afc5-3d9e091d783c |
| ip_version        | 6                                    |
| ipv6_address_mode | slaac                                |
| ipv6_ra_mode      | slaac                                |
| name              | subnet_ipv6_2                        |
| network_id        | c83a0fbc-558e-4f26-bf13-1607d9ef7de3 |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| revision_number   | 0                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | 2018-12-25T10:25:27Z                 |
+-------------------+--------------------------------------+

[05:25:29.843] (tester) Adding subnet_ipv4_2 and subnet_ipv6_2 to the router.


[05:25:43.973] (tester) Connecting the router to the external network "nova"


[05:25:51.231] (tester) Creating security group rules for group "sec_group"

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-25T10:25:57Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | 87c19417-a1ef-431a-8857-9e9ecea72050 |
| name              | None                                 |
| port_range_max    | 80                                   |
| port_range_min    | 80                                   |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| updated_at        | 2018-12-25T10:25:57Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-25T10:26:01Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | b9bb5850-c8df-4ced-9667-9b6e75727e26 |
| name              | None                                 |
| port_range_max    | 22                                   |
| port_range_min    | 22                                   |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| updated_at        | 2018-12-25T10:26:01Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-25T10:26:04Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | 949af6b5-e707-4d50-9ff8-7ccd40084f4d |
| name              | None                                 |
| port_range_max    | 443                                  |
| port_range_min    | 443                                  |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| updated_at        | 2018-12-25T10:26:04Z                 |
+-------------------+--------------------------------------+
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | 2018-12-25T10:26:07Z                 |
| description       |                                      |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | 0e7d036e-0e43-4f20-be08-76bb4b18923a |
| name              | None                                 |
| port_range_max    | None                                 |
| port_range_min    | None                                 |
| project_id        | 09ee7dd1bff648b099a8fca98f49738c     |
| protocol          | icmp                                 |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | 0                                    |
| security_group_id | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| updated_at        | 2018-12-25T10:26:07Z                 |
+-------------------+--------------------------------------+
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range  | Port Range | Remote Security Group                | Security Group                       |
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+
| 0e7d036e-0e43-4f20-be08-76bb4b18923a | icmp        | 0.0.0.0/0 |            | None                                 | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| 2bf61797-539e-407c-a667-cc83c1ecacd7 | None        | None      |            | 856ed828-0321-47c4-a880-b71775cb0cce | 856ed828-0321-47c4-a880-b71775cb0cce |
| 6ecc0f8e-d3d2-4174-b73a-9e0d1262a121 | None        | None      |            | None                                 | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| 87c19417-a1ef-431a-8857-9e9ecea72050 | tcp         | 0.0.0.0/0 | 80:80      | None                                 | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| 8ba47d9d-795d-488f-aad6-b96aa81c9b4c | None        | None      |            | None                                 | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| 949af6b5-e707-4d50-9ff8-7ccd40084f4d | tcp         | 0.0.0.0/0 | 443:443    | None                                 | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| b9bb5850-c8df-4ced-9667-9b6e75727e26 | tcp         | 0.0.0.0/0 | 22:22      | None                                 | c4a46802-8aa5-4e21-84cc-3ff96a079582 |
| c2fee30c-b071-4064-b2cc-dc8a78c6880a | None        | None      |            | None                                 | 856ed828-0321-47c4-a880-b71775cb0cce |
| ca06ba9e-8af7-41aa-9e5d-aeeee76cc28a | None        | None      |            | 856ed828-0321-47c4-a880-b71775cb0cce | 856ed828-0321-47c4-a880-b71775cb0cce |
| cd45c64b-b6b1-4ee2-a64c-dde8b75b8d63 | None        | None      |            | None                                 | 856ed828-0321-47c4-a880-b71775cb0cce |
+--------------------------------------+-------------+-----------+------------+--------------------------------------+--------------------------------------+

[05:26:10.954] (tester) Creating openstack key pair to easily login into VMs:

+-------------+-------------------------------------------------+
| Field       | Value                                           |
+-------------+-------------------------------------------------+
| fingerprint | 87:82:5c:5f:ac:33:e3:2a:fa:02:96:6b:6b:90:f5:ac |
| name        | tester-key                                      |
| user_id     | b53a8059553341a5b41bd165e9ac03c3                |
+-------------+-------------------------------------------------+
+------------+-------------------------------------------------+
| Name       | Fingerprint                                     |
+------------+-------------------------------------------------+
| tester-key | 87:82:5c:5f:ac:33:e3:2a:fa:02:96:6b:6b:90:f5:ac |
+------------+-------------------------------------------------+

[05:26:20.418] (tester) For each Network - creating 2 VM instances:


[05:26:29.989] (tester) Creating and booting VM instance: cirros35_vm1_net1, connected to network net_ipv64_1:

+-----------------------------+------------------------------------------------------+
| Field                       | Value                                                |
+-----------------------------+------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                               |
| OS-EXT-AZ:availability_zone |                                                      |
| OS-EXT-STS:power_state      | NOSTATE                                              |
| OS-EXT-STS:task_state       | scheduling                                           |
| OS-EXT-STS:vm_state         | building                                             |
| OS-SRV-USG:launched_at      | None                                                 |
| OS-SRV-USG:terminated_at    | None                                                 |
| accessIPv4                  |                                                      |
| accessIPv6                  |                                                      |
| addresses                   |                                                      |
| adminPass                   | npMk7acgFok4                                         |
| config_drive                |                                                      |
| created                     | 2018-12-25T10:26:45Z                                 |
| flavor                      | cirros_flavor (e6096fca-de55-4a82-8522-0d9c81b91aad) |
| hostId                      |                                                      |
| id                          | edf174a4-50d2-4b4c-bb54-cae74ccdb874                 |
| image                       | cirros35 (d59b98b8-c136-4c27-a6ea-9c0b4ff80da6)      |
| key_name                    | tester-key                                           |
| name                        | cirros35_vm1_net1                                        |
| progress                    | 0                                                        |
| project_id                  | 09ee7dd1bff648b099a8fca98f49738c                         |
| properties                  |                                                          |
| security_groups             | name='sec_group'                                         |
| status                      | ACTIVE                                                   |

[05:27:09.277] (tester) Adding floating ip 10.0.0.223 to cirros35_vm1_net1, and checking connectivity:


[05:27:37.869] (tester) Setting a name to the port of the new floating ip: "cirros35_vm1_net1_10.0.0.223"

+-----------------------+------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                    |
+-----------------------+------------------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                                       |
| allowed_address_pairs |                                                                                          |
| binding_host_id       | None                                                                                     |
| binding_profile       | None                                                                                     |
| binding_vif_details   | None                                                                                     |
| binding_vif_type      | None                                                                                     |
| binding_vnic_type     | normal                                                                                   |
| created_at            | 2018-12-25T10:26:50Z                                                                     |
| data_plane_status     | None                                                                                     |
| description           |                                                                                          |
| device_id             | edf174a4-50d2-4b4c-bb54-cae74ccdb874                                                     |
| device_owner          | compute:nova                                                                             |
| dns_assignment        | None                                                                                     |
| dns_domain            | None                                                                                     |
| dns_name              | None                                                                                     |
| extra_dhcp_opts       |                                                                                          |
| fixed_ips             | ip_address='10.0.1.22', subnet_id='b23b3221-0df2-41c7-bec5-02f0cd700f52'                 |
|                       | ip_address='2001::f816:3eff:fedc:8600', subnet_id='51bcd214-3431-467b-8261-a90a59aff0bc' |
| id                    | 8328ebfe-c2ef-41b8-a8e6-35ee853aaccd                                                     |
| mac_address           | fa:16:3e:dc:86:00                                                                        |
| name                  | cirros35_vm1_net1_10.0.0.223                                                             |
| network_id            | ca9e10da-65c4-4deb-bab5-5481fae07f7a                                                     |
| port_security_enabled | True                                                                                     |
| project_id            | 09ee7dd1bff648b099a8fca98f49738c                                                         |
| qos_policy_id         | None                                                                                     |
| revision_number       | 5                                                                                        |
| security_group_ids    | c4a46802-8aa5-4e21-84cc-3ff96a079582                                                     |
| status                | ACTIVE                                                                                   |
PING 10.0.0.223 (10.0.0.223) 56(84) bytes of data.

--- 10.0.0.223 ping statistics ---
30 packets transmitted, 0 received, 100% packet loss, time 29004ms



[stack@undercloud-0 ~]$ openstack server list --all
+--------------------------------------+-------------------+--------+--------------------------------------------------------------+----------+---------------+
| ID                                   | Name              | Status | Networks                                                     | Image    | Flavor        |
+--------------------------------------+-------------------+--------+--------------------------------------------------------------+----------+---------------+
| edf174a4-50d2-4b4c-bb54-cae74ccdb874 | cirros35_vm1_net1 | ACTIVE | net_ipv64_1=2001::f816:3eff:fedc:8600, 10.0.1.22, 10.0.0.223 | cirros35 | cirros_flavor |
+--------------------------------------+-------------------+--------+--------------------------------------------------------------+----------+---------------+

(overcloud) [stack@undercloud-0 ~]$ ping 10.0.0.223
PING 10.0.0.223 (10.0.0.223) 56(84) bytes of data.
^C
--- 10.0.0.223 ping statistics ---
74 packets transmitted, 0 received, 100% packet loss, time 73012ms

(overcloud) [stack@undercloud-0 ~]$
(overcloud) [stack@undercloud-0 ~]$ openstack server show edf174a4-50d2-4b4c-bb54-cae74ccdb874
+-------------------------------------+--------------------------------------------------------------+
| Field                               | Value                                                        |
+-------------------------------------+--------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                       |
| OS-EXT-AZ:availability_zone         | nova                                                         |
| OS-EXT-SRV-ATTR:host                | compute-1.localdomain                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.localdomain                                        |
| OS-EXT-SRV-ATTR:instance_name       | instance-0000006b                                            |
| OS-EXT-STS:power_state              | Running                                                      |
| OS-EXT-STS:task_state               | None                                                         |
| OS-EXT-STS:vm_state                 | active                                                       |
| OS-SRV-USG:launched_at              | 2018-12-25T10:27:02.000000                                   |
| OS-SRV-USG:terminated_at            | None                                                         |
| accessIPv4                          |                                                              |
| accessIPv6                          |                                                              |
| addresses                           | net_ipv64_1=2001::f816:3eff:fedc:8600, 10.0.1.22, 10.0.0.223 |
| config_drive                        |                                                              |
| created                             | 2018-12-25T10:26:45Z                                         |
| flavor                              | cirros_flavor (e6096fca-de55-4a82-8522-0d9c81b91aad)         |
| hostId                              | 35f4a10db2322853d61c3b5b6016895e18bb6130a0c5f4e34dfcebe9     |
| id                                  | edf174a4-50d2-4b4c-bb54-cae74ccdb874                         |
| image                               | cirros35 (d59b98b8-c136-4c27-a6ea-9c0b4ff80da6)              |
| key_name                            | tester-key                                                   |
| name                                | cirros35_vm1_net1                                            |
| progress                            | 0                                                            |
| project_id                          | 09ee7dd1bff648b099a8fca98f49738c                             |
| properties                          |                                                              |
| security_groups                     | name='sec_group'                                             |
| status                              | ACTIVE                                                       |
| updated                             | 2018-12-25T10:27:03Z                                         |
| user_id                             | b53a8059553341a5b41bd165e9ac03c3                             |
| volumes_attached                    |                                                              |
+-------------------------------------+--------------------------------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ openstack port list
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------------------+--------+
| ID                                   | Name                         | MAC Address       | Fixed IP Addresses                                                                       | Status |
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------------------+--------+
| 0232bdc0-2ca9-4466-a68f-076ecbcfa227 |                              | fa:16:3e:9e:5e:4f | ip_address='10.0.0.220', subnet_id='8bb7610c-af1f-4f31-a285-501971aadd11'                | DOWN   |
| 0896b966-6fcf-41df-baaa-d45e1425db04 |                              | fa:16:3e:97:84:fa | ip_address='10.0.1.2', subnet_id='b23b3221-0df2-41c7-bec5-02f0cd700f52'                  | ACTIVE |
|                                      |                              |                   | ip_address='2001::f816:3eff:fe97:84fa', subnet_id='51bcd214-3431-467b-8261-a90a59aff0bc' |        |
| 3c823d7d-b016-4aef-9bcc-d202a091ff8b |                              | fa:16:3e:96:4c:84 | ip_address='10.0.1.1', subnet_id='b23b3221-0df2-41c7-bec5-02f0cd700f52'                  | DOWN   |
| 48be5042-9017-42a6-bc4f-4be56c0a82e9 |                              | fa:16:3e:82:49:21 | ip_address='10.0.1.4', subnet_id='b23b3221-0df2-41c7-bec5-02f0cd700f52'                  | ACTIVE |
|                                      |                              |                   | ip_address='2001::f816:3eff:fe82:4921', subnet_id='51bcd214-3431-467b-8261-a90a59aff0bc' |        |
| 535e64c6-6fef-4a99-84f6-f265ba8546cf |                              | fa:16:3e:ae:97:63 | ip_address='10.0.1.3', subnet_id='b23b3221-0df2-41c7-bec5-02f0cd700f52'                  | ACTIVE |
|                                      |                              |                   | ip_address='2001::f816:3eff:feae:9763', subnet_id='51bcd214-3431-467b-8261-a90a59aff0bc' |        |
| 5a991911-5bcb-4609-8ee0-87a49db2f832 |                              | fa:16:3e:8f:29:27 | ip_address='10.0.2.1', subnet_id='6073882b-a028-48fa-b224-68746aa4ac52'                  | DOWN   |
| 670e0169-75cf-4936-b448-c0e3cb2c344d |                              | fa:16:3e:72:ae:46 | ip_address='2002::1', subnet_id='ef96249a-b921-44c7-afc5-3d9e091d783c'                   | DOWN   |
| 8328ebfe-c2ef-41b8-a8e6-35ee853aaccd | cirros35_vm1_net1_10.0.0.223 | fa:16:3e:dc:86:00 | ip_address='10.0.1.22', subnet_id='b23b3221-0df2-41c7-bec5-02f0cd700f52'                 | ACTIVE |
|                                      |                              |                   | ip_address='2001::f816:3eff:fedc:8600', subnet_id='51bcd214-3431-467b-8261-a90a59aff0bc' |        |
| 8a1a5aef-fa71-4adc-b862-4fd159d51f1a |                              | fa:16:3e:94:86:94 | ip_address='10.0.2.2', subnet_id='6073882b-a028-48fa-b224-68746aa4ac52'                  | ACTIVE |
|                                      |                              |                   | ip_address='2002::f816:3eff:fe94:8694', subnet_id='ef96249a-b921-44c7-afc5-3d9e091d783c' |        |
| a8c54698-a9f1-4e0f-bf91-546820e79a1a |                              | fa:16:3e:62:1b:44 | ip_address='10.0.2.3', subnet_id='6073882b-a028-48fa-b224-68746aa4ac52'                  | ACTIVE |
|                                      |                              |                   | ip_address='2002::f816:3eff:fe62:1b44', subnet_id='ef96249a-b921-44c7-afc5-3d9e091d783c' |        |
| d0b6071e-348d-418e-8ab9-0ec2df5b65ae |                              | fa:16:3e:65:5c:75 | ip_address='2001::1', subnet_id='51bcd214-3431-467b-8261-a90a59aff0bc'                   | DOWN   |
| d91a715b-1f88-456f-9ac6-956a803bf263 |                              | fa:16:3e:85:27:a4 | ip_address='10.0.0.223', subnet_id='8bb7610c-af1f-4f31-a285-501971aadd11'                | N/A    |
| dd2b6f61-a7a6-4a39-901d-0db4de087f89 |                              | fa:16:3e:24:f7:c8 | ip_address='10.0.2.4', subnet_id='6073882b-a028-48fa-b224-68746aa4ac52'                  | ACTIVE |
|                                      |                              |                   | ip_address='2002::f816:3eff:fe24:f7c8', subnet_id='ef96249a-b921-44c7-afc5-3d9e091d783c' |        |
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------------------+--------+

(overcloud) [stack@undercloud-0 ~]$ openstack port show cirros35_vm1_net1_10.0.0.223
+-----------------------+------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                    |
+-----------------------+------------------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                                       |
| allowed_address_pairs |                                                                                          |
| binding_host_id       | compute-1.localdomain                                                                    |
| binding_profile       |                                                                                          |
| binding_vif_details   |                                                                                          |
| binding_vif_type      | ovs                                                                                      |
| binding_vnic_type     | normal                                                                                   |
| created_at            | 2018-12-25T10:26:50Z                                                                     |
| data_plane_status     | None                                                                                     |
| description           |                                                                                          |
| device_id             | edf174a4-50d2-4b4c-bb54-cae74ccdb874                                                     |
| device_owner          | compute:nova                                                                             |
| dns_assignment        | None                                                                                     |
| dns_domain            | None                                                                                     |
| dns_name              | None                                                                                     |
| extra_dhcp_opts       |                                                                                          |
| fixed_ips             | ip_address='10.0.1.22', subnet_id='b23b3221-0df2-41c7-bec5-02f0cd700f52'                 |
|                       | ip_address='2001::f816:3eff:fedc:8600', subnet_id='51bcd214-3431-467b-8261-a90a59aff0bc' |
| id                    | 8328ebfe-c2ef-41b8-a8e6-35ee853aaccd                                                     |
| mac_address           | fa:16:3e:dc:86:00                                                                        |
| name                  | cirros35_vm1_net1_10.0.0.223                                                             |
| network_id            | ca9e10da-65c4-4deb-bab5-5481fae07f7a                                                     |
| port_security_enabled | True                                                                                     |
| project_id            | 09ee7dd1bff648b099a8fca98f49738c                                                         |
| qos_policy_id         | None                                                                                     |
| revision_number       | 5                                                                                        |
| security_group_ids    | c4a46802-8aa5-4e21-84cc-3ff96a079582                                                     |
| status                | ACTIVE                                                                                   |
| tags                  |                                                                                          |
| trunk_details         | None                                                                                     |
| updated_at            | 2018-12-25T10:27:41Z                                                                     |
+-----------------------+------------------------------------------------------------------------------------------+




*** controller-1: /var/log/containers/neutron/server.log ***

2018-12-25 10:26:51.076 38 INFO networking_odl.journal.journal [req-de942de0-41cf-4806-9dd1-289d56174c0a - - - - -] Processing (Entry ID: 2840) - create port 8328ebfe-c2ef-41b8-a8e6-35ee853aaccd (Time stamp: 63681330411.1)
2018-12-25 10:26:51.077 38 DEBUG networking_odl.common.client [req-de942de0-41cf-4806-9dd1-289d56174c0a - - - - -] Sending METHOD (post) URL (http://172.17.1.28:8081/controller/nb/v2/neutron/ports) JSON ({
2018-12-25 10:26:51.095 38 DEBUG networking_odl.common.client [req-de942de0-41cf-4806-9dd1-289d56174c0a - - - - -] Got response:
    "admin_state_up": true,
    "allowed_address_pairs": [],
    "binding:host_id": "",
    "binding:profile": "{}",
    "binding:vif_details": {},
    "binding:vif_type": "unbound",
    "binding:vnic_type": "normal",
    "device_id": "edf174a4-50d2-4b4c-bb54-cae74ccdb874"
    "device_owner": "",
    "fixed_ips": [
    "id": "8328ebfe-c2ef-41b8-a8e6-35ee853aaccd",
        "id": "c4a46802-8aa5-4e21-84cc-3ff96a079582"
        "ip_address": "10.0.1.22"
        "ip_address": "2001::f816:3eff:fedc:8600"
    "mac_address": "fa:16:3e:dc:86:00",
    "name": "",
    "network_id": "ca9e10da-65c4-4deb-bab5-5481fae07f7a",
   "port" : {
  "port": {
    "port_security_enabled": true,
    "project_id": "09ee7dd1bff648b099a8fca98f49738c",
}) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89
    "security_groups": [
        "subnet_id": "51bcd214-3431-467b-8261-a90a59aff0bc",
        "subnet_id": "b23b3221-0df2-41c7-bec5-02f0cd700f52",
    "tenant_id": "09ee7dd1bff648b099a8fca98f49738c",


*** controller-1: /var/log/containers/neutron/dhcp-agent.log ***

2018-12-25 10:26:51.124 124733 INFO neutron.agent.dhcp.agent [req-8c771a9d-67be-4f76-9c1a-c7c08c4371a6 b53a8059553341a5b41bd165e9ac03c3 09ee7dd1bff648b099a8fca98f49738c - - -] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=, binding:profile=, binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=normal, created_at=2018-12-25T10:26:50Z, description=, device_id=edf174a4-50d2-4b4c-bb54-cae74ccdb874, device_owner=, extra_dhcp_opts=[], fixed_ips=[{u'subnet_id': u'b23b3221-0df2-41c7-bec5-02f0cd700f52', u'ip_address': u'10.0.1.22'}, {u'subnet_id': u'51bcd214-3431-467b-8261-a90a59aff0bc', u'ip_address': u'2001::f816:3eff:fedc:8600'}], id=8328ebfe-c2ef-41b8-a8e6-35ee853aaccd, mac_address=fa:16:3e:dc:86:00, name=, network_id=ca9e10da-65c4-4deb-bab5-5481fae07f7a, port_security_enabled=True, project_id=09ee7dd1bff648b099a8fca98f49738c, revision_number=1, security_groups=[u'c4a46802-8aa5-4e21-84cc-3ff96a079582'], status=DOWN, tags=[], tenant_id=09ee7dd1bff648b099a8fca98f49738c, updated_at=2018-12-25T10:26:50Z

2018-12-25 10:26:53.901 124733 INFO neutron.agent.dhcp.agent [req-f408bfcf-5598-4025-884d-a8eff34cf028 78342ec326264a47ac8b68c3f598c561 8305ac01758543b2999465cda67389ce - - -] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=compute-1.localdomain, binding:profile=, binding:vif_details=, binding:vif_type=unbound, binding:vnic_type=normal, created_at=2018-12-25T10:26:50Z, description=, device_id=edf174a4-50d2-4b4c-bb54-cae74ccdb874, device_owner=compute:nova, extra_dhcp_opts=[], fixed_ips=[{u'subnet_id': u'b23b3221-0df2-41c7-bec5-02f0cd700f52', u'ip_address': u'10.0.1.22'}, {u'subnet_id': u'51bcd214-3431-467b-8261-a90a59aff0bc', u'ip_address': u'2001::f816:3eff:fedc:8600'}], id=8328ebfe-c2ef-41b8-a8e6-35ee853aaccd, mac_address=fa:16:3e:dc:86:00, name=, network_id=ca9e10da-65c4-4deb-bab5-5481fae07f7a, port_security_enabled=True, project_id=09ee7dd1bff648b099a8fca98f49738c, revision_number=2, security_groups=[u'c4a46802-8aa5-4e21-84cc-3ff96a079582'], status=DOWN, tags=[], tenant_id=09ee7dd1bff648b099a8fca98f49738c, updated_at=2018-12-25T10:26:53Z

2018-12-25 10:26:54.588 124733 INFO neutron.agent.dhcp.agent [req-f408bfcf-5598-4025-884d-a8eff34cf028 78342ec326264a47ac8b68c3f598c561 8305ac01758543b2999465cda67389ce - - -] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[], binding:host_id=compute-1.localdomain, binding:profile=, binding:vif_details=, binding:vif_type=ovs, binding:vnic_type=normal, created_at=2018-12-25T10:26:50Z, description=, device_id=edf174a4-50d2-4b4c-bb54-cae74ccdb874, device_owner=compute:nova, extra_dhcp_opts=[], fixed_ips=[{u'subnet_id': u'b23b3221-0df2-41c7-bec5-02f0cd700f52', u'ip_address': u'10.0.1.22'}, {u'subnet_id': u'51bcd214-3431-467b-8261-a90a59aff0bc', u'ip_address': u'2001::f816:3eff:fedc:8600'}], id=8328ebfe-c2ef-41b8-a8e6-35ee853aaccd, mac_address=fa:16:3e:dc:86:00, name=, network_id=ca9e10da-65c4-4deb-bab5-5481fae07f7a, port_security_enabled=True, project_id=09ee7dd1bff648b099a8fca98f49738c, revision_number=3, security_groups=[u'c4a46802-8aa5-4e21-84cc-3ff96a079582'], status=DOWN, tags=[], tenant_id=09ee7dd1bff648b099a8fca98f49738c, updated_at=2018-12-25T10:26:54Z


*** controller-1: /var/log/containers/nova/nova-api.log ***

2018-12-25 10:27:19.067 18 INFO nova.api.openstack.compute.server_external_events [req-740ed978-14b3-4fd3-8ef4-8477b0470079 623912a696184caaba0436c24394c51b 8305ac01758543b2999465cda67389ce - default default] Creating event network-changed:8328ebfe-c2ef-41b8-a8e6-35ee853aaccd for instance edf174a4-50d2-4b4c-bb54-cae74ccdb874 on compute-1.localdomain
2018-12-25 10:27:19.074 18 INFO nova.api.openstack.requestlog [req-740ed978-14b3-4fd3-8ef4-8477b0470079 623912a696184caaba0436c24394c51b 8305ac01758543b2999465cda67389ce - default default] 172.17.1.10 "POST /v2.1/os-server-external-events" status: 200 len: 179 microversion: 2.1 time: 0.417015


*** controller-1: /var/log/containers/opendaylight/karaf/logs/karaf.log ***

2018-12-25T10:26:51,096 | DEBUG | org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.ports.rev150712.ports.attributes.ports.Port_AsyncClusteredDataTreeChangeListenerBase-DataTreeChangeHandler-0 | org.opendaylight.netvirt.ipv6service.NeutronPortChangeListener | Add port notification handler is invoked for port Port{getAllowedAddressPairs=[], getDeviceId=edf174a4-50d2-4b4c-bb54-cae74ccdb874, getDeviceOwner=, getFixedIps=[FixedIps{getIpAddress=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.1.22]], getSubnetId=Uuid [_value=b23b3221-0df2-41c7-bec5-02f0cd700f52], augmentations={}}, FixedIps{getIpAddress=IpAddress [_ipv6Address=Ipv6Address [_value=2001::f816:3eff:fedc:8600]], getSubnetId=Uuid [_value=51bcd214-3431-467b-8261-a90a59aff0bc], augmentations={}}], getMacAddress=MacAddress [_value=fa:16:3e:dc:86:00], getName=, getNetworkId=Uuid [_value=ca9e10da-65c4-4deb-bab5-5481fae07f7a], getProjectId=09ee7dd1bff648b099a8fca98f49738c, getSecurityGroups=[Uuid [_value=c4a46802-8aa5-4e21-84cc-3ff96a079582]], getStatus=ACTIVE, getTenantId=Uuid [_value=09ee7dd1-bff6-48b0-99a8-fca98f49738c], getUuid=Uuid [_value=8328ebfe-c2ef-41b8-a8e6-35ee853aaccd], isAdminStateUp=true, augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.portsecurity.rev150712.PortSecurityExtension=PortSecurityExtension{isPortSecurityEnabled=true}, interface org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.binding.rev150712.PortBindingExtension=PortBindingExtension{getHostId=, getProfile={}, getVifDetails=[], getVifType=unbound, getVnicType=normal}}}

2018-12-25T10:26:51,097 | DEBUG | org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.ports.rev150712.ports.attributes.ports.Port_AsyncClusteredDataTreeChangeListenerBase-DataTreeChangeHandler-0 | org.opendaylight.netvirt.ipv6service.IfMgr                   | addHostIntf portId Uuid [_value=8328ebfe-c2ef-41b8-a8e6-35ee853aaccd], snetId Uuid [_value=51bcd214-3431-467b-8261-a90a59aff0bc], networkId Uuid [_value=ca9e10da-65c4-4deb-bab5-5481fae07f7a], ip IpAddress [_ipv6Address=Ipv6Address [_value=2001::f816:3eff:fedc:8600]], mac fa:16:3e:dc:86:00

2018-12-25T10:26:51,097 | INFO  | org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.ports.rev150712.ports.attributes.ports.Port_AsyncClusteredDataTreeChangeListenerBase-DataTreeChangeHandler-0 | org.opendaylight.netvirt.ipv6service.NeutronPortChangeListener | IPv6: addInterfaceInfo is invoked for a host interface Port{getAllowedAddressPairs=[], getDeviceId=edf174a4-50d2-4b4c-bb54-cae74ccdb874, getDeviceOwner=, getFixedIps=[FixedIps{getIpAddress=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.1.22]], getSubnetId=Uuid [_value=b23b3221-0df2-41c7-bec5-02f0cd700f52], augmentations={}}, FixedIps{getIpAddress=IpAddress [_ipv6Address=Ipv6Address [_value=2001::f816:3eff:fedc:8600]], getSubnetId=Uuid [_value=51bcd214-3431-467b-8261-a90a59aff0bc], augmentations={}}], getMacAddress=MacAddress [_value=fa:16:3e:dc:86:00], getName=, getNetworkId=Uuid [_value=ca9e10da-65c4-4deb-bab5-5481fae07f7a], getProjectId=09ee7dd1bff648b099a8fca98f49738c, getSecurityGroups=[Uuid [_value=c4a46802-8aa5-4e21-84cc-3ff96a079582]], getStatus=ACTIVE, getTenantId=Uuid [_value=09ee7dd1-bff6-48b0-99a8-fca98f49738c], getUuid=Uuid [_value=8328ebfe-c2ef-41b8-a8e6-35ee853aaccd], isAdminStateUp=true, augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.portsecurity.rev150712.PortSecurityExtension=PortSecurityExtension{isPortSecurityEnabled=true}, interface org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.binding.rev150712.PortBindingExtension=PortBindingExtension{getHostId=, getProfile={}, getVifDetails=[], getVifType=unbound, getVnicType=normal}}}, fixedIp: FixedIps{getIpAddress=IpAddress [_ipv6Address=Ipv6Address [_value=2001::f816:3eff:fedc:8600]], getSubnetId=Uuid [_value=51bcd214-3431-467b-8261-a90a59aff0bc], augmentations={}}



*** compute-1: /var/log/containers/nova/nova-compute.log ***

2018-12-25 10:26:51.331 1 DEBUG nova.network.neutronv2.api [req-d31c2371-a937-44b2-ba11-dc5aa2a2e06c b53a8059553341a5b41bd165e9ac03c3 09ee7dd1bff648b099a8fca98f49738c - default default] [instance: edf174a4-50d2-4b4c-bb54-cae74ccdb874] Successfully created port: 8328ebfe-c2ef-41b8-a8e6-35ee853aaccd _create_port_minimal /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:515
2018-12-25 10:26:54.099 1 DEBUG oslo_concurrency.processutils [req-d31c2371-a937-44b2-ba11-dc5aa2a2e06c b53a8059553341a5b41bd165e9ac03c3 09ee7dd1bff648b099a8fca98f49738c - default default] Running cmd (subprocess): /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/_base/926e18916c154174132819611627f34b08caa109.part --force-share execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372

2018-12-25 10:26:54.621 1 DEBUG nova.network.neutronv2.api [req-d31c2371-a937-44b2-ba11-dc5aa2a2e06c b53a8059553341a5b41bd165e9ac03c3 09ee7dd1bff648b099a8fca98f49738c - default default] [instance: edf174a4-50d2-4b4c-bb54-cae74ccdb874] Successfully updated port: 8328ebfe-c2ef-41b8-a8e6-35ee853aaccd _update_port /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:553

2018-12-25 10:26:55.337 1 DEBUG nova.network.neutronv2.api [req-d31c2371-a937-44b2-ba11-dc5aa2a2e06c b53a8059553341a5b41bd165e9ac03c3 09ee7dd1bff648b099a8fca98f49738c - default default] [instance: edf174a4-50d2-4b4c-bb54-cae74ccdb874] Instance cache missing network info. _get_preexisting_port_ids /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:2703

2018-12-25 10:26:55.986 1 DEBUG nova.compute.manager [req-12628adc-fbba-494c-8bd4-04717326d4e5 623912a696184caaba0436c24394c51b 8305ac01758543b2999465cda67389ce - default default] [instance: edf174a4-50d2-4b4c-bb54-cae74ccdb874] Received event network-changed-8328ebfe-c2ef-41b8-a8e6-35ee853aaccd external_instance_event /usr/lib/python2.7/site-packages/nova/compute/manager.py:8100



Expected results:
VM floating ip should be reachable.

Additional info:

Comment 1 Noam Manos 2018-12-25 14:14:33 UTC
SOS reports:
http://rhos-release.virt.bos.redhat.com/log/bz1662042

Comment 2 Aswin Suryanarayanan 2018-12-26 19:17:40 UTC
The controller fails to ARP the mac of the node(PNF) from which we are pinging the  FIP. In the log I could see the below error.

2018-12-26T11:52:27,385 | DEBUG | pool-44-thread-1 | org.opendaylight.netvirt.vpnmanager.SubnetRoutePacketInHandler | SUBNETROUTE: handlePacketToExternalNetwork: Can't find external router with id 4f53e5d4-4132-44fb-8465-bbf12c4fd682

Here 4f53e5d4-4132-44fb-8465-bbf12c4fd682 is the external network Id while it should be the router Id.

Comment 4 Noam Manos 2019-01-02 19:26:19 UTC
Created attachment 1518022 [details]
Bug was reproduced on a new OSP14 env - full output + log traces attached.

(overcloud) [stack@undercloud-0 ~]$ openstack server list --all
+--------------------------------------+-------------------+--------+-------------------------------------------------------------+----------+---------------+
| ID                                   | Name              | Status | Networks                                                    | Image    | Flavor        |
+--------------------------------------+-------------------+--------+-------------------------------------------------------------+----------+---------------+
| 1fc18c53-d815-4eec-ac0e-0714cf7ba524 | cirros35_vm1_net1 | ACTIVE | net_ipv64_1=10.0.1.6, 2001::f816:3eff:fefc:914a, 10.0.0.211 | cirros35 | cirros_flavor |
+--------------------------------------+-------------------+--------+-------------------------------------------------------------+----------+---------------+

(overcloud) [stack@undercloud-0 ~]$ ping 10.0.0.211
PING 10.0.0.211 (10.0.0.211) 56(84) bytes of data.
^C
--- 10.0.0.211 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6001ms

Comment 8 Aswin Suryanarayanan 2019-01-04 19:01:54 UTC
The issue is due to the dual stack network associated to the vm. When the we try to ARP for the PNF  (undercloud node from where we ping the FIP from), we need the router id to process the ARP which  is returned from a list .  If an Ipv6 network is associated to  the router this list will have both router id and external network id. The code now expects only one element in the list , so when there are more than one it is uses the first entry. If the first entry happens to be the router id the usecase works else it fails. If we use only Ipv4 network we should not be hitting the issue.

Comment 9 Noam Manos 2019-01-06 16:07:49 UTC
I can confirm this by running my script without creating IPv6 network and subnet:

(overcloud) [stack@undercloud-0 ~]$ openstack server list --all
+--------------------------------------+-------------------+--------+-----------------------------------+----------+---------------+
| ID                                   | Name              | Status | Networks                          | Image    | Flavor        |
+--------------------------------------+-------------------+--------+-----------------------------------+----------+---------------+
| cc52f18e-29db-49bf-9dfd-090bc7536185 | cirros35_vm1_net1 | ACTIVE | multi_net_1=10.0.1.21, 10.0.0.216 | cirros35 | cirros_flavor |
+--------------------------------------+-------------------+--------+-----------------------------------+----------+---------------+

(overcloud) [stack@undercloud-0 ~]$ ping -c 5 10.0.0.216
PING 10.0.0.216 (10.0.0.216) 56(84) bytes of data.
64 bytes from 10.0.0.216: icmp_seq=1 ttl=64 time=2.59 ms
64 bytes from 10.0.0.216: icmp_seq=2 ttl=64 time=0.744 ms
64 bytes from 10.0.0.216: icmp_seq=3 ttl=64 time=0.785 ms
64 bytes from 10.0.0.216: icmp_seq=4 ttl=64 time=0.532 ms
64 bytes from 10.0.0.216: icmp_seq=5 ttl=64 time=0.661 ms

--- 10.0.0.216 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.532/1.064/2.599/0.772 ms

(overcloud) [stack@undercloud-0 ~]$ openstack server show cc52f18e-29db-49bf-9dfd-090bc7536185
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | compute-1.redhat.local                                   |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.redhat.local                                   |
| OS-EXT-SRV-ATTR:instance_name       | instance-0000000a                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2019-01-06T15:52:52.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | multi_net_1=10.0.1.21, 10.0.0.216                        |
| config_drive                        |                                                          |
| created                             | 2019-01-06T15:52:36Z                                     |
| flavor                              | cirros_flavor (15627900-382b-4533-9d9b-2939f4900a0f)     |
| hostId                              | 7fa349bcac6bf07664b798622a8b92fac404ed160e315bc4f059a3d0 |
| id                                  | cc52f18e-29db-49bf-9dfd-090bc7536185                     |
| image                               | cirros35 (5c16482d-a8e9-43ce-aee4-9267f12d2142)          |
| key_name                            | tester-key                                               |
| name                                | cirros35_vm1_net1                                        |
| progress                            | 0                                                        |
| project_id                          | eb870ce9506f4586b67da3fd11721b54                         |
| properties                          |                                                          |
| security_groups                     | name='sec_group'                                         |
| status                              | ACTIVE                                                   |
| updated                             | 2019-01-06T15:52:52Z                                     |
| user_id                             | 6d92d68d65784c769849397a41ef4b0f                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 8b7e3cbd-72cf-49b4-a215-cc1376d4790b | 10.0.0.216          | 10.0.1.21        | 1edd9bba-b509-4799-955c-bff61f1d427b | a41b4698-8645-420a-841d-d38b1e4d39f7 | eb870ce9506f4586b67da3fd11721b54 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+

(overcloud) [stack@undercloud-0 ~]$ openstack floating ip show 10.0.0.216
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2019-01-06T15:52:11Z                 |
| description         |                                      |
| dns_domain          | None                                 |
| dns_name            | None                                 |
| fixed_ip_address    | 10.0.1.21                            |
| floating_ip_address | 10.0.0.216                           |
| floating_network_id | a41b4698-8645-420a-841d-d38b1e4d39f7 |
| id                  | 8b7e3cbd-72cf-49b4-a215-cc1376d4790b |
| name                | 10.0.0.216                           |
| port_details        | None                                 |
| port_id             | 1edd9bba-b509-4799-955c-bff61f1d427b |
| project_id          | eb870ce9506f4586b67da3fd11721b54     |
| qos_policy_id       | None                                 |
| revision_number     | 2                                    |
| router_id           | 2c173245-902b-44c5-9b3e-f88f850ad7be |
| status              | ACTIVE                               |
| subnet_id           | None                                 |
| tags                | []                                   |
| updated_at          | 2019-01-06T15:53:08Z                 |
+---------------------+--------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ openstack port show 1edd9bba-b509-4799-955c-bff61f1d427b
+-----------------------+--------------------------------------------------------------------------+
| Field                 | Value                                                                    |
+-----------------------+--------------------------------------------------------------------------+
| admin_state_up        | UP                                                                       |
| allowed_address_pairs |                                                                          |
| binding_host_id       | compute-1.redhat.local                                                   |
| binding_profile       |                                                                          |
| binding_vif_details   |                                                                          |
| binding_vif_type      | ovs                                                                      |
| binding_vnic_type     | normal                                                                   |
| created_at            | 2019-01-06T15:52:42Z                                                     |
| data_plane_status     | None                                                                     |
| description           |                                                                          |
| device_id             | cc52f18e-29db-49bf-9dfd-090bc7536185                                     |
| device_owner          | compute:nova                                                             |
| dns_assignment        | None                                                                     |
| dns_domain            | None                                                                     |
| dns_name              | None                                                                     |
| extra_dhcp_opts       |                                                                          |
| fixed_ips             | ip_address='10.0.1.21', subnet_id='144a4e77-ca72-42f8-9393-dc0a1b36b41c' |
| id                    | 1edd9bba-b509-4799-955c-bff61f1d427b                                     |
| mac_address           | fa:16:3e:a6:26:47                                                        |
| name                  | cirros35_vm1_net1_10.0.0.216                                             |
| network_id            | 7d789d55-4c4e-4357-b86d-f2918f71ef54                                     |
| port_security_enabled | True                                                                     |
| project_id            | eb870ce9506f4586b67da3fd11721b54                                         |
| qos_policy_id         | None                                                                     |
| revision_number       | 5                                                                        |
| security_group_ids    | 51eae504-38e7-472d-8791-223b24f10f0d                                     |
| status                | ACTIVE                                                                   |
| tags                  |                                                                          |
| trunk_details         | None                                                                     |
| updated_at            | 2019-01-06T15:53:33Z                                                     |
+-----------------------+--------------------------------------------------------------------------+

Comment 13 Franck Baudin 2019-03-06 16:14:37 UTC
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality

Comment 14 Franck Baudin 2019-03-06 16:17:09 UTC
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality


Note You need to log in before you can comment on or make changes to this bug.