Description of problem: problem spotted when running tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless when trying to ping gateway's IPv6 address '2003::1' at https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_v6.py#L198 with tempest error: 2017-09-29 12:31:14,067 5616 DEBUG [tempest.lib.common.utils.linux.remote_client] Remote command: set -eu -o pipefail; PATH=$PATH:/sbin; ping6 -c1 -w1 -s56 2003::1 2017-09-29 12:31:14,067 5616 INFO [tempest.lib.common.ssh] Creating ssh connection to '10.0.0.211:22' as 'cirros' with public key authentication 2017-09-29 12:31:14,069 5616 INFO [paramiko.transport] Connected (version 2.0, client dropbear_2012.55) 2017-09-29 12:31:14,171 5616 INFO [paramiko.transport] Authentication (publickey) successful! 2017-09-29 12:31:14,201 5616 INFO [tempest.lib.common.ssh] ssh connection to cirros.0.211 successfully created 2017-09-29 12:31:15,209 5616 ERROR [tempest.lib.common.utils.linux.remote_client] (TestGettingAddress:test_dualnet_multi_prefix_dhcpv6_stateless) Initializing SSH connection to 10.0.0.211 failed. Error: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; ping6 -c1 -w1 -s56 2003::1', exit status: 1, stderr: stdout: PING 2003::1 (2003::1): 56 data bytes --- 2003::1 ping statistics --- 1 packets transmitted, 0 packets received, 100% packet loss 2017-09-29 12:31:15,210 5616 WARNING [tempest.scenario.manager] Failed to ping IP: 2003::1 via a ssh connection from: 10.0.0.211. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. stack RHOSP 12 with 1 controller and 2 compute nodes, opendaylight-6.2.0-0.1.20170921snap729.el7.noarch 2. run tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless 3. Actual results: failed tempest test because of failed ping to the neutron gateway's IPv6 Expected results: ping and tempest test successful Additional info: VM affected is on compute-0 tempest-network-smoke--50134449=2003::f816:3eff:fef8:2f0d, 2003::1:f816:3eff:fef8:2f0d; tempest-network-smoke--517531218=10.100.0.13, 10.0.0.214 VM cirros shell: $ ip -o a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue \ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 1: lo inet 127.0.0.1/8 scope host lo 1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000\ link/ether fa:16:3e:93:36:e7 brd ff:ff:ff:ff:ff:ff 2: eth0 inet 10.100.0.13/28 brd 10.100.0.15 scope global eth0 2: eth0 inet6 fe80::f816:3eff:fe93:36e7/64 scope link \ valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000\ link/ether fa:16:3e:f8:2f:0d brd ff:ff:ff:ff:ff:ff 3: eth1 inet6 2003::1:f816:3eff:fef8:2f0d/64 scope global dynamic \ valid_lft 2591943sec preferred_lft 604743sec 3: eth1 inet6 2003::f816:3eff:fef8:2f0d/64 scope global dynamic \ valid_lft 2591943sec preferred_lft 604743sec 3: eth1 inet6 fe80::f816:3eff:fef8:2f0d/64 scope link \ valid_lft forever preferred_lft forever $ ping -c 3 2003::1 PING 2003::1 (2003::1): 56 data bytes --- 2003::1 ping statistics --- 3 packets transmitted, 0 packets received, 100% packet loss ping from VM that fails the test to the VM that succeeds the test works: # ping 2003::f816:3eff:fe2e:3f8f PING 2003::f816:3eff:fe2e:3f8f (2003::f816:3eff:fe2e:3f8f): 56 data bytes 64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=0 ttl=64 time=3.876 ms 64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=1 ttl=64 time=1.799 ms 64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=2 ttl=64 time=1.229 ms 64 bytes from 2003::f816:3eff:fe2e:3f8f: seq=3 ttl=64 time=1.674 ms ofctl flows on compute-0: [root@compute-0 ~]# ovs-ofctl -O OpenFlow13 dump-flows br-int | grep 2003::1 cookie=0x8000003, duration=87202.502s, table=21, n_packets=0, n_bytes=0, priority=138,icmp6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1,icmp_type=128,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:9a:5c:a0->eth_src,move:NXM_NX_IPV6_SRC[]->NXM_NX_IPV6_DST[],set_field:2003::1->ipv6_src,set_field:129->icmpv6_type,load:0->NXM_OF_IN_PORT[],resubmit(,21) cookie=0x8000003, duration=87202.344s, table=21, n_packets=0, n_bytes=0, priority=138,ipv6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1:f816:3eff:fe2e:3f8f actions=set_field:0x5->tun_id,set_field:fa:16:3e:2e:3f:8f->eth_dst,load:0x6700->NXM_NX_REG6[],resubmit(,220) cookie=0x4000000, duration=87202.504s, table=45, n_packets=53, n_bytes=4558, priority=50,icmp6,metadata=0x1394000000/0xffff000000,icmp_type=135,icmp_code=0,nd_target=2003::1 actions=CONTROLLER:65535 cookie=0x6900000, duration=87202.503s, table=211, n_packets=498, n_bytes=58764, priority=61010,ipv6,dl_src=fa:16:3e:f8:2f:0d,ipv6_src=2003::1:f816:3eff:fef8:2f0d actions=ct(table=212,zone=5012) cookie=0x6900000, duration=87202.503s, table=241, n_packets=8, n_bytes=944, priority=61010,ipv6,dl_dst=fa:16:3e:f8:2f:0d,ipv6_dst=2003::1:f816:3eff:fef8:2f0d actions=ct(table=242,zone=5012) VM !!!not!!! affected (ping to 2003::1 working - spawned by the same tempest test run) on compute-1: tempest-network-smoke--50134449=2003::f816:3eff:fe2e:3f8f, 2003::1:f816:3eff:fe2e:3f8f; tempest-network-smoke--517531218=10.100.0.12, 10.0.0. 221 VM's shell: $ ip -o a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue \ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 1: lo inet 127.0.0.1/8 scope host lo 1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000\ link/ether fa:16:3e:9a:eb:45 brd ff:ff:ff:ff:ff:ff 2: eth0 inet 10.100.0.12/28 brd 10.100.0.15 scope global eth0 2: eth0 inet6 fe80::f816:3eff:fe9a:eb45/64 scope link \ valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000\ link/ether fa:16:3e:2e:3f:8f brd ff:ff:ff:ff:ff:ff 3: eth1 inet6 2003::1:f816:3eff:fe2e:3f8f/64 scope global dynamic \ valid_lft 2591978sec preferred_lft 604778sec 3: eth1 inet6 2003::f816:3eff:fe2e:3f8f/64 scope global dynamic \ valid_lft 2591978sec preferred_lft 604778sec 3: eth1 inet6 fe80::f816:3eff:fe2e:3f8f/64 scope link \ valid_lft forever preferred_lft forever $ ping6 -c 3 2003::1 PING 2003::1 (2003::1): 56 data bytes 64 bytes from 2003::1: seq=0 ttl=64 time=0.811 ms 64 bytes from 2003::1: seq=1 ttl=64 time=0.463 ms 64 bytes from 2003::1: seq=2 ttl=64 time=0.487 ms --- 2003::1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.463/0.587/0.811 ms $ [root@compute-1 ~]# ovs-ofctl -O OpenFlow13 dump-flows br-int | grep --color -i 2003::1 cookie=0x8000003, duration=166074.674s, table=21, n_packets=2924, n_bytes=345032, priority=138,icmp6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1,icmp_type=128,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:9a:5c:a0->eth_src,move:NXM_NX_IPV6_SRC[]->NXM_NX_IPV6_DST[],set_field:2003::1->ipv6_src,set_field:129->icmpv6_type,load:0->NXM_OF_IN_PORT[],resubmit(,21) cookie=0x8000003, duration=166074.674s, table=21, n_packets=0, n_bytes=0, priority=138,ipv6,metadata=0x30f48/0xfffffe,ipv6_dst=2003::1:f816:3eff:fe2e:3f8f actions=group:150018 cookie=0x4000000, duration=166074.845s, table=45, n_packets=65, n_bytes=5590, priority=50,icmp6,metadata=0x1394000000/0xffff000000,icmp_type=135,icmp_code=0,nd_target=2003::1 actions=CONTROLLER:65535 cookie=0x6900000, duration=166074.674s, table=211, n_packets=8, n_bytes=944, priority=61010,ipv6,dl_src=fa:16:3e:2e:3f:8f,ipv6_src=2003::1:f816:3eff:fe2e:3f8f actions=ct(table=212,zone=5012) cookie=0x6900000, duration=166074.675s, table=241, n_packets=8, n_bytes=944, priority=61010,ipv6,dl_dst=fa:16:3e:2e:3f:8f,ipv6_dst=2003::1:f816:3eff:fe2e:3f8f actions=ct(table=242,zone=5012) Openstack neutron (sub)net info: (overcloud) [stack@undercloud-0 ~]$ openstack network show tempest-network-smoke--50134449 +---------------------------+----------------------------------------------------------------------------+ | Field | Value | +---------------------------+----------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2017-09-29T16:37:41Z | | description | | | dns_domain | None | | id | 2b271624-81f9-41f7-bbac-2c2afcfc9014 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1450 | | name | tempest-network-smoke--50134449 | | port_security_enabled | True | | project_id | 1b1a12006def4322b96531fd5017433f | | provider:network_type | vxlan | | provider:physical_network | None | | provider:segmentation_id | 5 | | qos_policy_id | None | | revision_number | 4 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 10440451-b29f-4aaa-bcae-8b4e2a7b3f0f, 2a58a2b3-43cf-4a41-bd91-b8d2154227cd | | tags | | | updated_at | 2017-09-29T16:37:52Z | +---------------------------+----------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack subnet show 10440451-b29f-4aaa-bcae-8b4e2a7b3f0f +-------------------------+--------------------------------------+ | Field | Value | +-------------------------+--------------------------------------+ | allocation_pools | 2003::2-2003::ffff:ffff:ffff:ffff | | cidr | 2003::/64 | | created_at | 2017-09-29T16:37:48Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 2003::1 | | host_routes | | | id | 10440451-b29f-4aaa-bcae-8b4e2a7b3f0f | | ip_version | 6 | | ipv6_address_mode | dhcpv6-stateless | | ipv6_ra_mode | dhcpv6-stateless | | name | tempest-sub6-2139066337 | | network_id | 2b271624-81f9-41f7-bbac-2c2afcfc9014 | | project_id | 1b1a12006def4322b96531fd5017433f | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2017-09-29T16:37:48Z | | use_default_subnet_pool | None | +-------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack subnet show 2a58a2b3-43cf-4a41-bd91-b8d2154227cd +-------------------------+-------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------+ | allocation_pools | 2003:0:0:1::2-2003::1:ffff:ffff:ffff:ffff | | cidr | 2003:0:0:1::/64 | | created_at | 2017-09-29T16:37:52Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 2003:0:0:1::1 | | host_routes | | | id | 2a58a2b3-43cf-4a41-bd91-b8d2154227cd | | ip_version | 6 | | ipv6_address_mode | dhcpv6-stateless | | ipv6_ra_mode | dhcpv6-stateless | | name | tempest-sub6-1043157343 | | network_id | 2b271624-81f9-41f7-bbac-2c2afcfc9014 | | project_id | 1b1a12006def4322b96531fd5017433f | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2017-09-29T16:37:52Z | | use_default_subnet_pool | None | +-------------------------+-------------------------------------------+
*** This bug has been marked as a duplicate of bug 1450894 ***