Bug 1497652

Summary: [IPv6] ping from VM to router's IPv6 (2003::1) doesn't work on 1of2 VMs in tempest test
Product: Red Hat OpenStack Reporter: Waldemar Znoinski <wznoinsk>
Component: opendaylightAssignee: lpeer <lpeer>
Status: CLOSED DUPLICATE QA Contact: Toni Freger <tfreger>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 12.0 (Pike)CC: mkolesni, nyechiel
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
N/A
Last Closed: 2017-10-02 12:25:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Waldemar Znoinski 2017-10-02 11:14:13 UTC 
Description of problem:
problem spotted when running tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless when trying to ping gateway's IPv6 address '2003::1'

at https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_v6.py#L198


with tempest error:
2017-09-29 12:31:14,067 5616 DEBUG    [tempest.lib.common.utils.linux.remote_client] Remote command: set -eu -o pipefail; PATH=$PATH:/sbin; ping6 -c1 -w1 -s56 2003::1
2017-09-29 12:31:14,067 5616 INFO     [tempest.lib.common.ssh] Creating ssh connection to '10.0.0.211:22' as 'cirros' with public key authentication
2017-09-29 12:31:14,069 5616 INFO     [paramiko.transport] Connected (version 2.0, client dropbear_2012.55)
2017-09-29 12:31:14,171 5616 INFO     [paramiko.transport] Authentication (publickey) successful!
2017-09-29 12:31:14,201 5616 INFO     [tempest.lib.common.ssh] ssh connection to cirros.0.211 successfully created
2017-09-29 12:31:15,209 5616 ERROR    [tempest.lib.common.utils.linux.remote_client] (TestGettingAddress:test_dualnet_multi_prefix_dhcpv6_stateless) Initializing SSH connection to 10.0.0.211 failed. Error: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; ping6 -c1 -w1 -s56 2003::1', exit status: 1, stderr:

stdout:
PING 2003::1 (2003::1): 56 data bytes

--- 2003::1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

2017-09-29 12:31:15,210 5616 WARNING  [tempest.scenario.manager] Failed to ping IP: 2003::1 via a ssh connection from: 10.0.0.211.

Version-Release number of selected component (if applicable):



How reproducible:
always

Steps to Reproduce:
1. stack RHOSP 12 with 1 controller and 2 compute nodes, opendaylight-6.2.0-0.1.20170921snap729.el7.noarch
2. run tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless
3.

Actual results:
failed tempest test because of failed ping to the neutron gateway's IPv6


Expected results:
ping and tempest test successful



Additional info:
VM affected is on compute-0
tempest-network-smoke--50134449=2003::f816:3eff:fef8:2f0d, 2003::1:f816:3eff:fef8:2f0d; tempest-network-smoke--517531218=10.100.0.13, 10.0.0.214


VM cirros shell:
$ ip -o a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue \    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
1: lo    inet 127.0.0.1/8 scope host lo
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000\    link/ether fa:16:3e:93:36:e7 brd ff:ff:ff:ff:ff:ff
2: eth0    inet 10.100.0.13/28 brd 10.100.0.15 scope global eth0
2: eth0    inet6 fe80::f816:3eff:fe93:36e7/64 scope link \       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000\    link/ether fa:16:3e:f8:2f:0d brd ff:ff:ff:ff:ff:ff
3: eth1    inet6 2003::1:f816:3eff:fef8:2f0d/64 scope global dynamic \       valid_lft 2591943sec preferred_lft 604743sec
3: eth1    inet6 2003::f816:3eff:fef8:2f0d/64 scope global dynamic \       valid_lft 2591943sec preferred_lft 604743sec
3: eth1    inet6 fe80::f816:3eff:fef8:2f0d/64 scope link \       valid_lft forever preferred_lft forever
$ ping -c 3 2003::1
PING 2003::1 (2003::1): 56 data bytes

--- 2003::1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss


ping from VM that fails the test to the VM that succeeds the test works:
# ping 2003::f816:3eff:fe2e:3f8f
PING 2003::f816:3eff:fe2e:3f8f (2003::f816:3eff:fe2e:3f8f): 56 data bytes
64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=0 ttl=64 time=3.876 ms
64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=1 ttl=64 time=1.799 ms
64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=2 ttl=64 time=1.229 ms
64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=3 ttl=64 time=1.674 ms



ofctl flows on compute-0:
[root@compute-0 ~]# ovs-ofctl -O OpenFlow13 dump-flows br-int | grep 2003::1
 cookie=0x8000003, duration=87202.502s, table=21, n_packets=0, n_bytes=0, priority=138,icmp6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1,icmp_type=128,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:9a:5c:a0->eth_src,move:NXM_NX_IPV6_SRC[]->NXM_NX_IPV6_DST[],set_field:2003::1->ipv6_src,set_field:129->icmpv6_type,load:0->NXM_OF_IN_PORT[],resubmit(,21)
 cookie=0x8000003, duration=87202.344s, table=21, n_packets=0, n_bytes=0, priority=138,ipv6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1:f816:3eff:fe2e:3f8f actions=set_field:0x5->tun_id,set_field:fa:16:3e:2e:3f:8f->eth_dst,load:0x6700->NXM_NX_REG6[],resubmit(,220)
 cookie=0x4000000, duration=87202.504s, table=45, n_packets=53, n_bytes=4558, priority=50,icmp6,metadata=0x1394000000/0xffff000000,icmp_type=135,icmp_code=0,nd_target=2003::1 actions=CONTROLLER:65535
 cookie=0x6900000, duration=87202.503s, table=211, n_packets=498, n_bytes=58764, priority=61010,ipv6,dl_src=fa:16:3e:f8:2f:0d,ipv6_src=2003::1:f816:3eff:fef8:2f0d actions=ct(table=212,zone=5012)
 cookie=0x6900000, duration=87202.503s, table=241, n_packets=8, n_bytes=944, priority=61010,ipv6,dl_dst=fa:16:3e:f8:2f:0d,ipv6_dst=2003::1:f816:3eff:fef8:2f0d actions=ct(table=242,zone=5012)



VM !!!not!!! affected (ping to 2003::1 working - spawned by the same tempest test run) on compute-1:
tempest-network-smoke--50134449=2003::f816:3eff:fe2e:3f8f, 2003::1:f816:3eff:fe2e:3f8f; tempest-network-smoke--517531218=10.100.0.12, 10.0.0.
221 


VM's shell:
$ ip -o a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue \    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
1: lo    inet 127.0.0.1/8 scope host lo
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000\    link/ether fa:16:3e:9a:eb:45 brd ff:ff:ff:ff:ff:ff
2: eth0    inet 10.100.0.12/28 brd 10.100.0.15 scope global eth0
2: eth0    inet6 fe80::f816:3eff:fe9a:eb45/64 scope link \       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000\    link/ether fa:16:3e:2e:3f:8f brd ff:ff:ff:ff:ff:ff
3: eth1    inet6 2003::1:f816:3eff:fe2e:3f8f/64 scope global dynamic \       valid_lft 2591978sec preferred_lft 604778sec
3: eth1    inet6 2003::f816:3eff:fe2e:3f8f/64 scope global dynamic \       valid_lft 2591978sec preferred_lft 604778sec
3: eth1    inet6 fe80::f816:3eff:fe2e:3f8f/64 scope link \       valid_lft forever preferred_lft forever
$ ping6 -c 3 2003::1
PING 2003::1 (2003::1): 56 data bytes
64 bytes from 2003::1: seq=0 ttl=64 time=0.811 ms
64 bytes from 2003::1: seq=1 ttl=64 time=0.463 ms
64 bytes from 2003::1: seq=2 ttl=64 time=0.487 ms

--- 2003::1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.463/0.587/0.811 ms
$ 


[root@compute-1 ~]# ovs-ofctl -O OpenFlow13 dump-flows br-int | grep --color -i 2003::1
 cookie=0x8000003, duration=166074.674s, table=21, n_packets=2924, n_bytes=345032, priority=138,icmp6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1,icmp_type=128,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:9a:5c:a0->eth_src,move:NXM_NX_IPV6_SRC[]->NXM_NX_IPV6_DST[],set_field:2003::1->ipv6_src,set_field:129->icmpv6_type,load:0->NXM_OF_IN_PORT[],resubmit(,21)
 cookie=0x8000003, duration=166074.674s, table=21, n_packets=0, n_bytes=0, priority=138,ipv6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1:f816:3eff:fe2e:3f8f actions=group:150018
 cookie=0x4000000, duration=166074.845s, table=45, n_packets=65, n_bytes=5590, priority=50,icmp6,metadata=0x1394000000/0xffff000000,icmp_type=135,icmp_code=0,nd_target=2003::1 actions=CONTROLLER:65535
 cookie=0x6900000, duration=166074.674s, table=211, n_packets=8, n_bytes=944, priority=61010,ipv6,dl_src=fa:16:3e:2e:3f:8f,ipv6_src=2003::1:f816:3eff:fe2e:3f8f actions=ct(table=212,zone=5012)
 cookie=0x6900000, duration=166074.675s, table=241, n_packets=8, n_bytes=944, priority=61010,ipv6,dl_dst=fa:16:3e:2e:3f:8f,ipv6_dst=2003::1:f816:3eff:fe2e:3f8f actions=ct(table=242,zone=5012)



Openstack neutron (sub)net info:
(overcloud) [stack@undercloud-0 ~]$ openstack network show tempest-network-smoke--50134449
+---------------------------+----------------------------------------------------------------------------+
| Field                     | Value                                                                      |
+---------------------------+----------------------------------------------------------------------------+
| admin_state_up            | UP                                                                         |
| availability_zone_hints   |                                                                            |
| availability_zones        | nova                                                                       |
| created_at                | 2017-09-29T16:37:41Z                                                       |
| description               |                                                                            |
| dns_domain                | None                                                                       |
| id                        | 2b271624-81f9-41f7-bbac-2c2afcfc9014                                       |
| ipv4_address_scope        | None                                                                       |
| ipv6_address_scope        | None                                                                       |
| is_default                | None                                                                       |
| is_vlan_transparent       | None                                                                       |
| mtu                       | 1450                                                                       |
| name                      | tempest-network-smoke--50134449                                            |
| port_security_enabled     | True                                                                       |
| project_id                | 1b1a12006def4322b96531fd5017433f                                           |
| provider:network_type     | vxlan                                                                      |
| provider:physical_network | None                                                                       |
| provider:segmentation_id  | 5                                                                          |
| qos_policy_id             | None                                                                       |
| revision_number           | 4                                                                          |
| router:external           | Internal                                                                   |
| segments                  | None                                                                       |
| shared                    | False                                                                      |
| status                    | ACTIVE                                                                     |
| subnets                   | 10440451-b29f-4aaa-bcae-8b4e2a7b3f0f, 2a58a2b3-43cf-4a41-bd91-b8d2154227cd |
| tags                      |                                                                            |
| updated_at                | 2017-09-29T16:37:52Z                                                       |
+---------------------------+----------------------------------------------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ openstack subnet show 10440451-b29f-4aaa-bcae-8b4e2a7b3f0f
+-------------------------+--------------------------------------+
| Field                   | Value                                |
+-------------------------+--------------------------------------+
| allocation_pools        | 2003::2-2003::ffff:ffff:ffff:ffff    |
| cidr                    | 2003::/64                            |
| created_at              | 2017-09-29T16:37:48Z                 |
| description             |                                      |
| dns_nameservers         |                                      |
| enable_dhcp             | True                                 |
| gateway_ip              | 2003::1                              |
| host_routes             |                                      |
| id                      | 10440451-b29f-4aaa-bcae-8b4e2a7b3f0f |
| ip_version              | 6                                    |
| ipv6_address_mode       | dhcpv6-stateless                     |
| ipv6_ra_mode            | dhcpv6-stateless                     |
| name                    | tempest-sub6-2139066337              |
| network_id              | 2b271624-81f9-41f7-bbac-2c2afcfc9014 |
| project_id              | 1b1a12006def4322b96531fd5017433f     |
| revision_number         | 0                                    |
| segment_id              | None                                 |
| service_types           |                                      |
| subnetpool_id           | None                                 |
| tags                    |                                      |
| updated_at              | 2017-09-29T16:37:48Z                 |
| use_default_subnet_pool | None                                 |
+-------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack subnet show 2a58a2b3-43cf-4a41-bd91-b8d2154227cd
+-------------------------+-------------------------------------------+
| Field                   | Value                                     |
+-------------------------+-------------------------------------------+
| allocation_pools        | 2003:0:0:1::2-2003::1:ffff:ffff:ffff:ffff |
| cidr                    | 2003:0:0:1::/64                           |
| created_at              | 2017-09-29T16:37:52Z                      |
| description             |                                           |
| dns_nameservers         |                                           |
| enable_dhcp             | True                                      |
| gateway_ip              | 2003:0:0:1::1                             |
| host_routes             |                                           |
| id                      | 2a58a2b3-43cf-4a41-bd91-b8d2154227cd      |
| ip_version              | 6                                         |
| ipv6_address_mode       | dhcpv6-stateless                          |
| ipv6_ra_mode            | dhcpv6-stateless                          |
| name                    | tempest-sub6-1043157343                   |
| network_id              | 2b271624-81f9-41f7-bbac-2c2afcfc9014      |
| project_id              | 1b1a12006def4322b96531fd5017433f          |
| revision_number         | 0                                         |
| segment_id              | None                                      |
| service_types           |                                           |
| subnetpool_id           | None                                      |
| tags                    |                                           |
| updated_at              | 2017-09-29T16:37:52Z                      |
| use_default_subnet_pool | None                                      |
+-------------------------+-------------------------------------------+
Comment 1 Waldemar Znoinski 2017-10-02 12:25:23 UTC 

*** This bug has been marked as a duplicate of bug 1450894 ***