1994624 – [4.8.z backport] On an IPv6 single stack cluster traffic between master nodes is sent via default gw instead of local subnet

Bug 1994624 - [4.8.z backport] On an IPv6 single stack cluster traffic between master nodes is sent via default gw instead of local subnet

Summary: [4.8.z backport] On an IPv6 single stack cluster traffic between master nodes...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	4.8.z
Assignee:	Jaime Caamaño Ruiz
QA Contact:	Ross Brattain
Docs Contact:
URL:
Whiteboard:
Depends On:	1980135
Blocks:
TreeView+	depends on / blocked

Reported:	2021-08-17 14:57 UTC by Jaime Caamaño Ruiz
Modified:	2021-10-12 06:01 UTC (History)
CC List:	1 user (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	* Previously, when using IPv6 DHCP, node interface addresses might be leased with a `/128` prefix. Consequently, OVN-Kubernetes uses the same prefix to infer the node's network and routes any other address traffic, including traffic to other cluster nodes, through the gateway. With this update, OVN-Kubernetes inspects the node's routing table and checks for the wider routing entry for the node's interface address and uses that prefix to infer the node's network. As a result, traffic to other cluster nodes is no longer routed through the gateway. (link:https://bugzilla.redhat.com/show_bug.cgi?id=1994624[BZ#1994624])
Clone Of:	1980135
Environment:
Last Closed:	2021-10-12 06:01:19 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift ovn-kubernetes pull 661	0	None	None	None	2021-08-31 08:49:38 UTC
Red Hat Product Errata	RHBA-2021:3682	0	None	None	None	2021-10-12 06:01:56 UTC

Description Jaime Caamaño Ruiz 2021-08-17 14:57:18 UTC

This bug was initially created as a copy of Bug #1980135

I am copying this bug because: 



Description of problem:

On an IPv6 single stack cluster traffic between master nodes is sent via the network's router. Since all master nodes are using the same subnet communication between the master nodes should not reach the network router. As a result this can impact the cluster functionality in the eventuality one of the master nodes cannot reach the router. 

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-07-04-112043

How reproducible:
100%

Steps to Reproduce:

1. Deploy a 3 x masters cluster with IPv6 single stack networking via baremetal IPI flow

[kni ~]$ oc get nodes -o wide

NAME                                                  STATUS   ROLES           AGE     VERSION           INTERNAL-IP         EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                CONTAINER-RUNTIME
openshift-master-0.kni-qe-0.lab.eng.rdu2.redhat.com   Ready    master,worker   2d11h   v1.21.1+f36aa36   2620:52:0:11c::20   <none>        Red Hat Enterprise Linux CoreOS 48.84.202107040900-0 (Ootpa)   4.18.0-305.7.1.el8_4.x86_64   cri-o://1.21.1-12.rhaos4.8.git30ca719.el8

openshift-master-1.kni-qe-0.lab.eng.rdu2.redhat.com   Ready    master,worker   2d11h   v1.21.1+f36aa36   2620:52:0:11c::21   <none>        Red Hat Enterprise Linux CoreOS 48.84.202107040900-0 (Ootpa)   4.18.0-305.7.1.el8_4.x86_64   cri-o://1.21.1-12.rhaos4.8.git30ca719.el8

openshift-master-2.kni-qe-0.lab.eng.rdu2.redhat.com   Ready    master,worker   2d11h   v1.21.1+f36aa36   2620:52:0:11c::22   <none>        Red Hat Enterprise Linux CoreOS 48.84.202107040900-0 (Ootpa)   4.18.0-305.7.1.el8_4.x86_64   cri-o://1.21.1-12.rhaos4.8.git30ca719.el8


3. Run a packet capture on the router(external to the cluster)

[root kni]$ tcpdump -i baremetal not tcp port 22 -ennn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on baremetal, link-type EN10MB (Ethernet), capture size 262144 bytes
16:53:02.641786 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 1782: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [P.], seq 350251707:350253403, ack 61527403, win 258, options [nop,nop,TS val 2888958758 ecr 3796529115], length 1696
16:53:02.641792 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 1782: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [P.], seq 0:1696, ack 1, win 258, options [nop,nop,TS val 2888958758 ecr 3796529115], length 1696
16:53:02.642231 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [.], ack 611, win 300, options [nop,nop,TS val 2888958758 ecr 3796529115], length 0
16:53:02.642237 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [.], ack 611, win 300, options [nop,nop,TS val 2888958758 ecr 3796529115], length 0
16:53:02.642309 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 110: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [P.], seq 1696:1720, ack 636, win 300, options [nop,nop,TS val 2888958758 ecr 3796529116], length 24
16:53:02.642314 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 110: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [P.], seq 1696:1720, ack 636, win 300, options [nop,nop,TS val 2888958758 ecr 3796529116], length 24
16:53:02.642345 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [F.], seq 1720, ack 636, win 300, options [nop,nop,TS val 2888958758 ecr 3796529116], length 0
16:53:02.642349 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [F.], seq 1720, ack 636, win 300, options [nop,nop,TS val 2888958758 ecr 3796529116], length 0
16:53:02.642351 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [R.], seq 1721, ack 636, win 300, options [nop,nop,TS val 2888958758 ecr 3796529116], length 0
16:53:02.642355 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.39343 > 2620:52:0:11c::21.6443: Flags [R.], seq 1721, ack 636, win 300, options [nop,nop,TS val 2888958758 ecr 3796529116], length 0
16:53:02.642373 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 125: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [P.], seq 2060208521:2060208560, ack 2866995814, win 9734, options [nop,nop,TS val 4109381108 ecr 3796529035], length 39
16:53:02.642407 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 125: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [P.], seq 0:39, ack 1, win 9734, options [nop,nop,TS val 4109381108 ecr 3796529035], length 39
16:53:02.644072 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 1782: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [P.], seq 3467523297:3467524993, ack 3528094138, win 258, options [nop,nop,TS val 3477795710 ecr 1716754515], length 1696
16:53:02.644081 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 1782: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [P.], seq 0:1696, ack 1, win 258, options [nop,nop,TS val 3477795710 ecr 1716754515], length 1696
16:53:02.644399 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 67, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 0
16:53:02.644416 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 67, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 0
16:53:02.644470 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 9166, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 0
16:53:02.644478 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 9166, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 0
16:53:02.644502 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [.], ack 611, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 0
16:53:02.644509 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [.], ack 611, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 0
16:53:02.644527 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 9197, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 0
16:53:02.644534 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 9197, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 0
16:53:02.644536 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 121: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [P.], seq 39:74, ack 9197, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 35
16:53:02.644542 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 121: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [P.], seq 39:74, ack 9197, win 9734, options [nop,nop,TS val 4109381110 ecr 3796529118], length 35
16:53:02.644585 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 110: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [P.], seq 1696:1720, ack 636, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 24
16:53:02.644592 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 110: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [P.], seq 1696:1720, ack 636, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 24
16:53:02.644599 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [F.], seq 1720, ack 636, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 0
16:53:02.644605 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [F.], seq 1720, ack 636, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 0
16:53:02.644621 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [R.], seq 1721, ack 636, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 0
16:53:02.644627 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33728 > 2620:52:0:11c::20.6443: Flags [R.], seq 1721, ack 636, win 300, options [nop,nop,TS val 3477795710 ecr 1716754516], length 0
16:53:02.645036 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 125: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [P.], seq 74:113, ack 9197, win 9734, options [nop,nop,TS val 4109381111 ecr 3796529118], length 39
16:53:02.645043 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 125: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [P.], seq 74:113, ack 9197, win 9734, options [nop,nop,TS val 4109381111 ecr 3796529118], length 39
16:53:02.645765 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 94: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [S], seq 303809659, win 26800, options [mss 1340,sackOK,TS val 3477795712 ecr 0,nop,wscale 7], length 0
16:53:02.645788 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 94: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [S], seq 303809659, win 26800, options [mss 1340,sackOK,TS val 3477795712 ecr 0,nop,wscale 7], length 0
16:53:02.645870 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [.], ack 1931895112, win 210, options [nop,nop,TS val 3477795712 ecr 1716754518], length 0
16:53:02.645879 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [.], ack 1, win 210, options [nop,nop,TS val 3477795712 ecr 1716754518], length 0
16:53:02.646061 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 628: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [P.], seq 0:542, ack 1, win 210, options [nop,nop,TS val 3477795712 ecr 1716754518], length 542
16:53:02.646068 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 628: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [P.], seq 0:542, ack 1, win 210, options [nop,nop,TS val 3477795712 ecr 1716754518], length 542
16:53:02.646251 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 9462, win 9734, options [nop,nop,TS val 4109381112 ecr 3796529120], length 0
16:53:02.646259 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7e:1a, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.37722 > 2620:52:0:11c::21.6443: Flags [.], ack 9462, win 9734, options [nop,nop,TS val 4109381112 ecr 3796529120], length 0
16:53:02.646326 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [.], ack 410, win 218, options [nop,nop,TS val 3477795712 ecr 1716754518], length 0
16:53:02.646333 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [.], ack 410, win 218, options [nop,nop,TS val 3477795712 ecr 1716754518], length 0
16:53:02.646573 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 1752: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [P.], seq 542:2208, ack 410, win 218, options [nop,nop,TS val 3477795712 ecr 1716754518], length 1666
16:53:02.646593 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 1752: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [P.], seq 542:2208, ack 410, win 218, options [nop,nop,TS val 3477795712 ecr 1716754518], length 1666
16:53:02.647058 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 110: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [P.], seq 2208:2232, ack 877, win 227, options [nop,nop,TS val 3477795713 ecr 1716754519], length 24
16:53:02.647078 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 110: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [P.], seq 2208:2232, ack 877, win 227, options [nop,nop,TS val 3477795713 ecr 1716754519], length 24
16:53:02.647082 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [F.], seq 2232, ack 877, win 227, options [nop,nop,TS val 3477795713 ecr 1716754519], length 0
16:53:02.647102 0c:42:a1:ee:86:72 > 0c:42:a1:ee:7f:12, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [F.], seq 2232, ack 877, win 227, options [nop,nop,TS val 3477795713 ecr 1716754519], length 0
16:53:02.647110 0c:42:a1:ee:7e:12 > 0c:42:a1:ee:86:72, ethertype IPv6 (0x86dd), length 86: 2620:52:0:11c::22.33738 > 2620:52:0:11c::20.6443: Flags [R.], seq 2233, ack 877, win 227, options [nop,nop,TS val 3477795713 ecr 1716754519], length

Router interface:

[root kni]$ ip a s dev baremetal
374: baremetal: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:42:a1:ee:86:72 brd ff:ff:ff:ff:ff:ff
    inet 10.1.28.1/24 brd 10.1.28.255 scope global noprefixroute baremetal
       valid_lft forever preferred_lft forever
    inet6 2620:52:0:11c::1/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::4315:47f1:33d8:3815/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever


master-0 interface and routing table:

[core@openshift-master-0 ~]$ ip a s dev br-ex
133: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 0c:42:a1:ee:7f:12 brd ff:ff:ff:ff:ff:ff
    inet6 2620:52:0:11c::20/128 scope global dynamic noprefixroute 
       valid_lft 3350sec preferred_lft 3350sec
    inet6 fe80::4600:7018:f0ae:c7f5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[core@openshift-master-0 ~]$ ip -6 r 
::1 dev lo proto kernel metric 256 pref medium
2620:52:0:11c::20 dev br-ex proto kernel metric 100 pref medium
2620:52:0:11c::/64 dev br-ex proto ra metric 100 pref medium
fd01:0:0:1::/64 dev ovn-k8s-mp0 proto kernel metric 256 pref medium
fd01::/48 via fd01:0:0:1::1 dev ovn-k8s-mp0 metric 1024 pref medium
fd02::/112 via fe80::4315:47f1:33d8:3815 dev br-ex metric 1024 mtu 1400 pref medium
fe80::/64 dev br-ex proto kernel metric 100 pref medium
fe80::/64 dev genev_sys_6081 proto kernel metric 256 pref medium
default via fe80::4315:47f1:33d8:3815 dev br-ex proto ra metric 100 pref medium

master-1 interface and routing table:

[core@openshift-master-1 ~]$ ip a s dev br-ex
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 0c:42:a1:ee:7e:1a brd ff:ff:ff:ff:ff:ff
    inet6 2620:52:0:11c::21/128 scope global dynamic noprefixroute 
       valid_lft 3083sec preferred_lft 3083sec
    inet6 fe80::922b:8255:e760:33a1/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[core@openshift-master-1 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
2620:52:0:11c::11 dev br-ex proto kernel metric 256 pref medium
2620:52:0:11c::21 dev br-ex proto kernel metric 100 pref medium
2620:52:0:11c::/64 dev br-ex proto ra metric 100 pref medium
fd01:0:0:3::/64 dev ovn-k8s-mp0 proto kernel metric 256 pref medium
fd01::/48 via fd01:0:0:3::1 dev ovn-k8s-mp0 metric 1024 pref medium
fd02::/112 via fe80::4315:47f1:33d8:3815 dev br-ex metric 1024 mtu 1400 pref medium
fe80::/64 dev br-ex proto kernel metric 100 pref medium
default via fe80::4315:47f1:33d8:3815 dev br-ex proto ra metric 100 pref medium

master-2 interface and routing table:

[core@openshift-master-2 ~]$ ip a s dev br-ex
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 0c:42:a1:ee:7e:12 brd ff:ff:ff:ff:ff:ff
    inet6 2620:52:0:11c::11/128 scope global nodad deprecated noprefixroute 
       valid_lft forever preferred_lft 0sec
    inet6 2620:52:0:11c::10/128 scope global nodad deprecated noprefixroute 
       valid_lft forever preferred_lft 0sec
    inet6 2620:52:0:11c::22/128 scope global dynamic noprefixroute 
       valid_lft 2364sec preferred_lft 2364sec
    inet6 fe80::3def:8ee0:1d96:d1e7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[core@openshift-master-2 ~]$ ip -6 r
::1 dev lo proto kernel metric 256 pref medium
2620:52:0:11c::10 dev br-ex proto kernel metric 256 pref medium
2620:52:0:11c::11 dev br-ex proto kernel metric 256 pref medium
2620:52:0:11c::22 dev br-ex proto kernel metric 100 pref medium
2620:52:0:11c::/64 dev br-ex proto ra metric 100 pref medium
fd01:0:0:2::/64 dev ovn-k8s-mp0 proto kernel metric 256 pref medium
fd01::/48 via fd01:0:0:2::1 dev ovn-k8s-mp0 metric 1024 pref medium
fd02::/112 via fe80::4315:47f1:33d8:3815 dev br-ex metric 1024 mtu 1400 pref medium
fe80::/64 dev br-ex proto kernel metric 100 pref medium
default via fe80::4315:47f1:33d8:3815 dev br-ex proto ra metric 100 pref medium


Actual results:

There is a lot of traffic between the master nodes reaching the network router which is external to the cluster.

Expected results:

Traffic between the master nodes should stay local to the subnet and not reach the router.

Additional info:

I tried removing on one of the nodes the interface from br-ex bridge and assigned the address to the interface outside the bridge and the router stopped receiving packets with source of this node and destination of other masters. The problem resumed after the ovnkube-node pod running on this node recovers so I suspect this may be a problem with OVN flows.

Comment 6 Ross Brattain 2021-10-06 19:40:56 UTC

Verified on 4.8.0-0.nightly-2021-10-06-061456

ovn-nbctl --no-leader-only --columns=_uuid,enabled,external_ids,ipv6_prefix,ipv6_ra_configs,mac,name,,networks --format=table find Logical_Router_Port

_uuid                                enabled external_ids              ipv6_prefix ipv6_ra_configs mac                 name                              networks
------------------------------------ ------- ------------------------- ----------- --------------- ------------------- --------------------------------- -------------------------
371ebbc8-d199-483d-aedc-587b30190461 []      {gateway-physical-ip=yes} []          {}              "52:54:00:9a:a7:53" rtoe-GR_master-0-0-o48e1-0.qe.lab ["fd2e:6f44:5dd8::71/64"]
b9836daa-75aa-4293-97d7-a931905344f7 []      {gateway-physical-ip=yes} []          {}              "52:54:00:92:66:6f" rtoe-GR_master-0-1-o48e1-0.qe.lab ["fd2e:6f44:5dd8::89/64"]
c527de0b-27f3-4bbe-813b-b7fb60edf37d []      {gateway-physical-ip=yes} []          {}              "52:54:00:7e:5c:e6" rtoe-GR_master-0-2-o48e1-0.qe.lab ["fd2e:6f44:5dd8::70/64"]


# ip l sh baremetal-0
260: baremetal-0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:16:e9:1a brd ff:ff:ff:ff:ff:ff

# tcpdump -i baremetal-0 -ennn tcp port 6443 and ether host 52:54:00:16:e9:1a
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on baremetal-0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured

Comment 8 errata-xmlrpc 2021-10-12 06:01:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.14 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3682

Note You need to log in before you can comment on or make changes to this bug.