Bug 1848462

Summary:	All tempest.scenario.test_network_v6.TestGettingAddress tests failed with DVR
Product:	Red Hat OpenStack	Reporter:	Alex Katz <akatz>
Component:	openstack-neutron	Assignee:	Slawek Kaplonski <skaplons>
Status:	CLOSED ERRATA	QA Contact:	Eran Kuris <ekuris>
Severity:	medium	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	amcleod, amuller, bcafarel, chrisw, ekuris, gregraka, igallagh, jamsmith, njohnston, scohen, skaplons, spower, tfreger
Target Milestone:	z2	Keywords:	AutomationBlocker, Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	Currently, on ML2/OVS and distributed virtual router (DVR) configurations, Open vSwitch (OVS) routes ICMPv6 traffic incorrectly, causing network outages on tenant networks. At this time, there is no workaround for this issue. If you have clouds that rely heavily on IPv6 and might experience issues caused by blocked ICMP traffic, such as pings, do not update to Red Hat OpenStack Platform 16.1 until this issue is fixed.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-28 15:38:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alex Katz 2020-06-18 12:11:48 UTC

Description of problem:
All the tempest.scenario.test_network_v6.TestGettingAddress tests fail from time to time in DVR environment.

If try to pause the test before it will start its connectivity checks and run continuous ping (VM->router) manually it will succeed only in ~5 minutes. But the issue can be simply avoided with waiting for ~30 seconds before initiating ping from VM. There is no connectivity issue between two VMs but only between VM and router.

Tcpdump on compute node (not router's namespace) shows that there are 3 ICMP requests (packet appears on 3 different interfaces) and only 1 ICMP reply. ICMP replies can be also seen on another compute node when the issue is happening and disappeared once it is fixed.


How reproducible:
Run tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os test

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Slawek Kaplonski 2020-07-03 15:03:51 UTC

After some investigation I think that the problem is that packets from qr- port instead of going to tapXXX port (VM) through rule:

 cookie=0xaa7dfc8bfab123a6, duration=91.307s, table=60, n_packets=164, n_bytes=17336, idle_age=13, priority=20,dl_vlan=1,dl_dst=fa:16:3e:e8:20:07 actions=strip_vlan,output:250

are going through:

 cookie=0xaa7dfc8bfab123a6, duration=290.239s, table=60, n_packets=2807, n_bytes=492469, idle_age=0, priority=3 actions=NORMAL

And are sending to the br-tun (I don't know why).

I will continue this investigation next week.

Comment 12 Slawek Kaplonski 2020-08-10 10:41:41 UTC

This issue should be fixed with openvswitch2.13-2.13.0-49.el8fdp

Comment 13 Bernard Cafarelli 2020-09-16 09:56:56 UTC

Tests are passing in CI with this version indeed

Comment 14 Toni Freger 2020-10-14 12:29:25 UTC

Hi Eran, sounds like it should be ON_QA, can you please verify it? Thanks

Comment 15 Eran Kuris 2020-10-14 14:09:52 UTC

(In reply to Toni Freger from comment #14)
> Hi Eran, sounds like it should be ON_QA, can you please verify it? Thanks

will be verified soon

Comment 16 Eran Kuris 2020-10-15 11:00:07 UTC

fix verified 
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/neutron/job/DFG-network-neutron-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs-dvr/17/

Comment 21 errata-xmlrpc 2020-10-28 15:38:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4284