2080255 – EgressIP broken on AWS with OpenShiftSDN / latest nightly build

Bug 2080255 - EgressIP broken on AWS with OpenShiftSDN / latest nightly build

Summary: EgressIP broken on AWS with OpenShiftSDN / latest nightly build

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	4.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.11.0
Assignee:	Andreas Karis
QA Contact:	huirwang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-04-29 11:18 UTC by Andreas Karis
Modified:	2022-08-10 11:10 UTC (History)
CC List:	1 user (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-08-10 11:09:50 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-network-operator pull 1409	0	None	open	Bug 2080255: SDN: Re-add list/watch/get permissions for nodes needed for EgressIP	2022-04-29 11:55:52 UTC
Red Hat Product Errata	RHSA-2022:5069	0	None	None	None	2022-08-10 11:10:26 UTC

Description Andreas Karis 2022-04-29 11:18:13 UTC

The EgressIp feature is currently broken for OpenShift SDN on latest nightly with AWS.
Traffic is dropped as soon as EgressIPs are configured.

~~~
[akaris@linux origin (egressip-tests-option3)]$ oc get netnamespace e2e-test-egressip-8pctt
NAME                      NETID    EGRESS IPS
e2e-test-egressip-8pctt   206988   ["10.0.128.5"]
[akaris@linux origin (egressip-tests-option3)]$ oc get hostsubnet
NAME                           HOST                           HOST IP        SUBNET          EGRESS CIDRS   EGRESS IPS
ip-10-0-130-173.ec2.internal   ip-10-0-130-173.ec2.internal   10.0.130.173   10.128.0.0/23                  
ip-10-0-153-252.ec2.internal   ip-10-0-153-252.ec2.internal   10.0.153.252   10.130.0.0/23                  
ip-10-0-154-45.ec2.internal    ip-10-0-154-45.ec2.internal    10.0.154.45    10.128.2.0/23                  
ip-10-0-182-38.ec2.internal    ip-10-0-182-38.ec2.internal    10.0.182.38    10.129.2.0/23                  ["10.0.128.5"]
ip-10-0-206-242.ec2.internal   ip-10-0-206-242.ec2.internal   10.0.206.242   10.129.0.0/23                  
ip-10-0-207-131.ec2.internal   ip-10-0-207-131.ec2.internal   10.0.207.131   10.131.0.0/23                  
(reverse-i-search)`gr': oc get netnamespace e2e-test-e^Cessip-8pctt
[akaris@linux origin (egressip-tests-option3)]$ oc get pods -A -o wide | grep -i egressip
e2e-test-egressip-8pctt                            e2e-test-egressip-8pctt-0-deployment-5595cf889-z4bmr          1/1     Running     0              44m     10.131.0.27    ip-10-0-207-131.ec2.internal   <none>           <none>
e2e-test-egressip-jj67v                            e2e-test-egressip-jj67v-packet-sniffer-nnktf                  1/1     Running     0              44m     10.0.182.38    ip-10-0-182-38.ec2.internal    <none>           <none>
e2e-test-egressip-jj67v                            e2e-test-egressip-jj67v-packet-sniffer-pkkjd                  1/1     Running     0              44m     10.0.207.131   ip-10-0-207-131.ec2.internal   <none>           <none>
[akaris@linux origin (egressip-tests-option3)]$ 
~~~

~~~
sh-4.4# chroot /host ovs-appctl ofproto/trace br0 udp,in_port=28,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,udp_src=10000,udp_dst=8001
Flow: udp,in_port=28,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=10000,tp_dst=8001

bridge("br0")
-------------
 0. ct_state=-trk,ip, priority 1000
    ct(table=0)
    drop
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 0.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.

Final flow: unchanged
Megaflow: recirc_id=0,ct_state=-trk,eth,ip,in_port=28,nw_frag=no
Datapath actions: ct,recirc(0x8128)

===============================================================================
recirc(0x8128) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================

Flow: recirc_id=0x8128,ct_state=new|trk,eth,udp,in_port=28,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=10000,tp_dst=8001

bridge("br0")
-------------
    thaw
        Resuming from table 0
 0. ip, priority 100
    goto_table:20
20. ip,in_port=28,nw_src=10.131.0.27, priority 100
    load:0x3288c->NXM_NX_REG0[]
    goto_table:27
27. reg0=0x3288c, priority 50
    goto_table:30
30. priority 0
    goto_table:31
31. ip, priority 0
    goto_table:99
99. priority 0
    goto_table:100
100. priority 0
    goto_table:101
101. reg0=0x3288c, priority 100
    drop

Final flow: recirc_id=0x8128,ct_state=new|trk,eth,udp,reg0=0x3288c,in_port=28,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=10000,tp_dst=8001
Megaflow: recirc_id=0x8128,ct_state=-rpl+trk,eth,udp,in_port=28,nw_src=10.131.0.27,nw_dst=8.0.0.0/7,nw_frag=no,tp_dst=0x1800/0xf800
Datapath actions: drop
~~~

When I remove the EgressIP assignment, then traffic is not dropped:
~~~
[akaris@linux origin (egressip-tests-option3)]$ oc patch netnamespace e2e-test-egressip-8pctt --type=merge -p   '{"egressIPs": []}'
netnamespace.network.openshift.io/e2e-test-egressip-8pctt patched
[akaris@linux origin (egressip-tests-option3)]$ 
~~~

~~~
sh-4.4# chroot /host ovs-appctl ofproto/trace br0 udp,in_port=28,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,udp_src=10000,udp_dst=8001
Flow: udp,in_port=28,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=10000,tp_dst=8001

bridge("br0")
-------------
 0. ct_state=-trk,ip, priority 1000
    ct(table=0)
    drop
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 0.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.

Final flow: unchanged
Megaflow: recirc_id=0,ct_state=-trk,eth,ip,in_port=28,nw_frag=no
Datapath actions: ct,recirc(0x822d)

===============================================================================
recirc(0x822d) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================

Flow: recirc_id=0x822d,ct_state=new|trk,eth,udp,in_port=28,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=10000,tp_dst=8001

bridge("br0")
-------------
    thaw
        Resuming from table 0
 0. ip, priority 100
    goto_table:20
20. ip,in_port=28,nw_src=10.131.0.27, priority 100
    load:0x3288c->NXM_NX_REG0[]
    goto_table:27
27. reg0=0x3288c, priority 50
    goto_table:30
30. priority 0
    goto_table:31
31. ip, priority 0
    goto_table:99
99. priority 0
    goto_table:100
100. priority 0
    goto_table:101
101. priority 0
    output:2

Final flow: recirc_id=0x822d,ct_state=new|trk,eth,udp,reg0=0x3288c,in_port=28,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:1b,dl_dst=62:33:1f:99:75:f2,nw_src=10.131.0.27,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=10000,tp_dst=8001
Megaflow: recirc_id=0x822d,ct_state=-rpl+trk,eth,udp,in_port=28,nw_src=10.131.0.27,nw_dst=8.0.0.0/7,nw_frag=no,tp_dst=0x1800/0xf800
Datapath actions: 3
~~~

Flows with EgressIP in place (with the drop):
~~~
sh-4.4# chroot /host ovs-ofctl -O OpenFlow13 dump-flows br0
 cookie=0x0, duration=6510.384s, table=0, n_packets=1012960, n_bytes=856261696, priority=1000,ct_state=-trk,ip actions=ct(table=0)
 cookie=0x0, duration=6510.384s, table=0, n_packets=75624, n_bytes=7880748, priority=400,ip,in_port=tun0,nw_src=10.131.0.1 actions=goto_table:30
 cookie=0x0, duration=6510.384s, table=0, n_packets=8033, n_bytes=1285552, priority=300,ip,in_port=tun0,nw_src=10.131.0.0/23,nw_dst=10.128.0.0/14 actions=goto_table:25
 cookie=0x0, duration=6510.384s, table=0, n_packets=0, n_bytes=0, priority=250,ip,in_port=tun0,nw_dst=224.0.0.0/4 actions=drop
 cookie=0x0, duration=6510.384s, table=0, n_packets=15304, n_bytes=642768, priority=200,arp,in_port=vxlan0,arp_spa=10.128.0.0/14,arp_tpa=10.131.0.0/23 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=6510.384s, table=0, n_packets=228815, n_bytes=169891025, priority=200,ip,in_port=vxlan0,nw_src=10.128.0.0/14 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=6510.384s, table=0, n_packets=0, n_bytes=0, priority=200,ip,in_port=vxlan0,nw_dst=10.128.0.0/14 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=6510.384s, table=0, n_packets=2354, n_bytes=98868, priority=200,arp,in_port=tun0,arp_spa=10.131.0.1,arp_tpa=10.128.0.0/14 actions=goto_table:30
 cookie=0x0, duration=6510.384s, table=0, n_packets=408477, n_bytes=1169582748, priority=200,ip,in_port=tun0 actions=goto_table:30
 cookie=0x0, duration=6510.384s, table=0, n_packets=0, n_bytes=0, priority=150,in_port=vxlan0 actions=drop
 cookie=0x0, duration=6510.384s, table=0, n_packets=13, n_bytes=998, priority=150,in_port=tun0 actions=drop
 cookie=0x0, duration=6510.384s, table=0, n_packets=18526, n_bytes=778092, priority=100,arp actions=goto_table:20
 cookie=0x0, duration=6510.384s, table=0, n_packets=784134, n_bytes=686369745, priority=100,ip actions=goto_table:20
 cookie=0x0, duration=6510.384s, table=0, n_packets=320, n_bytes=24328, priority=0 actions=drop
 cookie=0xbe7d4688, duration=6509.928s, table=10, n_packets=66445, n_bytes=82224987, priority=100,tun_src=10.0.130.173 actions=goto_table:30
 cookie=0x6290ecdb, duration=6509.847s, table=10, n_packets=32157, n_bytes=24915161, priority=100,tun_src=10.0.153.252 actions=goto_table:30
 cookie=0xcce4c564, duration=6509.754s, table=10, n_packets=19247, n_bytes=19822909, priority=100,tun_src=10.0.206.242 actions=goto_table:30
 cookie=0x8b508bcf, duration=6509.673s, table=10, n_packets=102449, n_bytes=35258572, priority=100,tun_src=10.0.154.45 actions=goto_table:30
 cookie=0xf380ca05, duration=6476.692s, table=10, n_packets=23821, n_bytes=8312164, priority=100,tun_src=10.0.182.38 actions=goto_table:30
 cookie=0x0, duration=6510.384s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6510.384s, table=20, n_packets=0, n_bytes=0, priority=300,udp,tp_dst=4789 actions=drop
 cookie=0x0, duration=6502.154s, table=20, n_packets=271, n_bytes=11382, priority=100,arp,in_port=vethbede2132,arp_spa=10.131.0.2,arp_sha=00:00:0a:83:00:02/00:00:ff:ff:ff:ff actions=load:0x76bbd9->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6493.517s, table=20, n_packets=1298, n_bytes=54516, priority=100,arp,in_port=veth73d7981c,arp_spa=10.131.0.3,arp_sha=00:00:0a:83:00:03/00:00:ff:ff:ff:ff actions=load:0x7b7e67->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6493.300s, table=20, n_packets=556, n_bytes=23352, priority=100,arp,in_port=vethf9aef496,arp_spa=10.131.0.5,arp_sha=00:00:0a:83:00:05/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6492.314s, table=20, n_packets=2213, n_bytes=92946, priority=100,arp,in_port=vethdd68c7e0,arp_spa=10.131.0.6,arp_sha=00:00:0a:83:00:06/00:00:ff:ff:ff:ff actions=load:0x8071a4->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6491.921s, table=20, n_packets=437, n_bytes=18354, priority=100,arp,in_port=veth2c851942,arp_spa=10.131.0.7,arp_sha=00:00:0a:83:00:07/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6481.492s, table=20, n_packets=915, n_bytes=38430, priority=100,arp,in_port=vethc6770694,arp_spa=10.131.0.9,arp_sha=00:00:0a:83:00:09/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6478.445s, table=20, n_packets=500, n_bytes=21000, priority=100,arp,in_port=vethde8ac3b2,arp_spa=10.131.0.10,arp_sha=00:00:0a:83:00:0a/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6478.294s, table=20, n_packets=2364, n_bytes=99288, priority=100,arp,in_port=veth35d9d440,arp_spa=10.131.0.11,arp_sha=00:00:0a:83:00:0b/00:00:ff:ff:ff:ff actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6478.131s, table=20, n_packets=363, n_bytes=15246, priority=100,arp,in_port=vethbd4b53d8,arp_spa=10.131.0.12,arp_sha=00:00:0a:83:00:0c/00:00:ff:ff:ff:ff actions=load:0x3b0e07->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6477.049s, table=20, n_packets=330, n_bytes=13860, priority=100,arp,in_port=veth6ee0e6f4,arp_spa=10.131.0.13,arp_sha=00:00:0a:83:00:0d/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6460.208s, table=20, n_packets=186, n_bytes=7812, priority=100,arp,in_port=vethcb7df831,arp_spa=10.131.0.15,arp_sha=00:00:0a:83:00:0f/00:00:ff:ff:ff:ff actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6090.631s, table=20, n_packets=330, n_bytes=13860, priority=100,arp,in_port=veth81c7cce1,arp_spa=10.131.0.16,arp_sha=00:00:0a:83:00:10/00:00:ff:ff:ff:ff actions=load:0x60a2a9->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6063.791s, table=20, n_packets=7103, n_bytes=298326, priority=100,arp,in_port=vethde54fe44,arp_spa=10.131.0.17,arp_sha=00:00:0a:83:00:11/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6046.096s, table=20, n_packets=1128, n_bytes=47376, priority=100,arp,in_port=veth46528948,arp_spa=10.131.0.18,arp_sha=00:00:0a:83:00:12/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=3001.184s, table=20, n_packets=39, n_bytes=1638, priority=100,arp,in_port=vethd0fc8f3f,arp_spa=10.131.0.27,arp_sha=00:00:0a:83:00:1b/00:00:ff:ff:ff:ff actions=load:0x3288c->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6502.154s, table=20, n_packets=4345, n_bytes=1455591, priority=100,ip,in_port=vethbede2132,nw_src=10.131.0.2 actions=load:0x76bbd9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6493.517s, table=20, n_packets=93593, n_bytes=20938026, priority=100,ip,in_port=veth73d7981c,nw_src=10.131.0.3 actions=load:0x7b7e67->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6493.300s, table=20, n_packets=6255, n_bytes=2287630, priority=100,ip,in_port=vethf9aef496,nw_src=10.131.0.5 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6492.314s, table=20, n_packets=27247, n_bytes=5614985, priority=100,ip,in_port=vethdd68c7e0,nw_src=10.131.0.6 actions=load:0x8071a4->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6491.921s, table=20, n_packets=81055, n_bytes=533099674, priority=100,ip,in_port=veth2c851942,nw_src=10.131.0.7 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6481.492s, table=20, n_packets=40957, n_bytes=17092859, priority=100,ip,in_port=vethc6770694,nw_src=10.131.0.9 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6478.445s, table=20, n_packets=3343, n_bytes=2569230, priority=100,ip,in_port=vethde8ac3b2,nw_src=10.131.0.10 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6478.294s, table=20, n_packets=22014, n_bytes=7360977, priority=100,ip,in_port=veth35d9d440,nw_src=10.131.0.11 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6478.131s, table=20, n_packets=2656, n_bytes=200024, priority=100,ip,in_port=vethbd4b53d8,nw_src=10.131.0.12 actions=load:0x3b0e07->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6477.049s, table=20, n_packets=41976, n_bytes=16213258, priority=100,ip,in_port=veth6ee0e6f4,nw_src=10.131.0.13 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6460.208s, table=20, n_packets=3732, n_bytes=401911, priority=100,ip,in_port=vethcb7df831,nw_src=10.131.0.15 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6090.631s, table=20, n_packets=8674, n_bytes=2161862, priority=100,ip,in_port=veth81c7cce1,nw_src=10.131.0.16 actions=load:0x60a2a9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6063.791s, table=20, n_packets=396492, n_bytes=60603339, priority=100,ip,in_port=vethde54fe44,nw_src=10.131.0.17 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6046.096s, table=20, n_packets=21608, n_bytes=11108317, priority=100,ip,in_port=veth46528948,nw_src=10.131.0.18 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=3001.184s, table=20, n_packets=140, n_bytes=11374, priority=100,ip,in_port=vethd0fc8f3f,nw_src=10.131.0.27 actions=load:0x3288c->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6510.384s, table=20, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6502.154s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.2 actions=load:0x76bbd9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6493.517s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.3 actions=load:0x7b7e67->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6493.300s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.5 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6492.314s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.6 actions=load:0x8071a4->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6491.921s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.7 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6481.492s, table=25, n_packets=3226, n_bytes=467261, priority=100,ip,nw_src=10.131.0.9 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6478.445s, table=25, n_packets=773, n_bytes=318198, priority=100,ip,nw_src=10.131.0.10 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6478.294s, table=25, n_packets=1992, n_bytes=225650, priority=100,ip,nw_src=10.131.0.11 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6478.131s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.12 actions=load:0x3b0e07->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6477.049s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.13 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6460.208s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.15 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6090.631s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.16 actions=load:0x60a2a9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6063.791s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.17 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6046.096s, table=25, n_packets=1684, n_bytes=233303, priority=100,ip,nw_src=10.131.0.18 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=3001.184s, table=25, n_packets=80, n_bytes=6912, priority=100,ip,nw_src=10.131.0.27 actions=load:0x3288c->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6510.384s, table=25, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6510.292s, table=27, n_packets=91584, n_bytes=12725824, priority=300,ip,nw_dst=172.30.0.0/16 actions=goto_table:30
 cookie=0x0, duration=6510.292s, table=27, n_packets=219221, n_bytes=618051517, priority=200,ct_state=+rpl,ip actions=goto_table:30
 cookie=0x0, duration=6502.106s, table=27, n_packets=0, n_bytes=0, priority=50,reg0=0x76bbd9 actions=goto_table:30
 cookie=0x0, duration=6493.457s, table=27, n_packets=76698, n_bytes=13020315, priority=50,reg0=0x7b7e67 actions=goto_table:30
 cookie=0x0, duration=6492.316s, table=27, n_packets=391326, n_bytes=42359042, priority=50,reg0=0xab6c55 actions=goto_table:30
 cookie=0x0, duration=6491.295s, table=27, n_packets=981, n_bytes=148168, priority=50,reg0=0x8071a4 actions=goto_table:30
 cookie=0x0, duration=6478.251s, table=27, n_packets=12062, n_bytes=1321522, priority=50,reg0=0x5a3f8f actions=goto_table:30
 cookie=0x0, duration=6478.078s, table=27, n_packets=0, n_bytes=0, priority=50,reg0=0x3b0e07 actions=goto_table:30
 cookie=0x0, duration=6090.591s, table=27, n_packets=0, n_bytes=0, priority=50,reg0=0x60a2a9 actions=goto_table:30
 cookie=0x0, duration=3001.138s, table=27, n_packets=140, n_bytes=11374, priority=50,reg0=0x3288c actions=goto_table:30
 cookie=0x0, duration=6510.385s, table=27, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6510.293s, table=30, n_packets=392430, n_bytes=51345378, priority=100,ct_state=-rpl,ip,nw_src=10.128.0.0/14,nw_dst=10.128.0.0/14 actions=ct(commit,table=31)
 cookie=0x0, duration=6510.385s, table=30, n_packets=1148805, n_bytes=1985181362, priority=0 actions=goto_table:31
 cookie=0x0, duration=6510.385s, table=31, n_packets=2345, n_bytes=98490, priority=300,arp,arp_tpa=10.131.0.1 actions=output:tun0
 cookie=0x0, duration=6510.385s, table=31, n_packets=65817, n_bytes=19313573, priority=300,ip,nw_dst=10.131.0.1 actions=output:tun0
 cookie=0x0, duration=6510.385s, table=31, n_packets=608218, n_bytes=1618336018, priority=250,ct_state=+rpl,ip,nw_dst=10.131.0.0/23 actions=ct(table=70,nat)
 cookie=0x0, duration=6510.385s, table=31, n_packets=18871, n_bytes=792582, priority=200,arp,arp_tpa=10.131.0.0/23 actions=goto_table:40
 cookie=0x0, duration=6510.385s, table=31, n_packets=245340, n_bytes=30038455, priority=200,ip,nw_dst=10.131.0.0/23 actions=goto_table:70
 cookie=0x0, duration=6510.385s, table=31, n_packets=14968, n_bytes=628656, priority=100,arp,arp_tpa=10.128.0.0/14 actions=goto_table:50
 cookie=0x0, duration=6510.385s, table=31, n_packets=233456, n_bytes=326465557, priority=100,ip,nw_dst=10.128.0.0/14 actions=goto_table:90
 cookie=0x0, duration=6510.385s, table=31, n_packets=91584, n_bytes=12725824, priority=100,ip,nw_dst=172.30.0.0/16 actions=goto_table:60
 cookie=0x0, duration=6510.385s, table=31, n_packets=0, n_bytes=0, priority=50,ip,in_port=vxlan0,nw_dst=224.0.0.0/4 actions=goto_table:120
 cookie=0x0, duration=6510.385s, table=31, n_packets=0, n_bytes=0, priority=25,ip,nw_dst=224.0.0.0/4 actions=goto_table:110
 cookie=0x0, duration=6510.385s, table=31, n_packets=260637, n_bytes=28127691, priority=0,ip actions=goto_table:99
 cookie=0x0, duration=6510.385s, table=31, n_packets=0, n_bytes=0, priority=0,arp actions=drop
 cookie=0x0, duration=6502.155s, table=40, n_packets=271, n_bytes=11382, priority=100,arp,arp_tpa=10.131.0.2 actions=output:vethbede2132
 cookie=0x0, duration=6493.518s, table=40, n_packets=1298, n_bytes=54516, priority=100,arp,arp_tpa=10.131.0.3 actions=output:veth73d7981c
 cookie=0x0, duration=6493.301s, table=40, n_packets=556, n_bytes=23352, priority=100,arp,arp_tpa=10.131.0.5 actions=output:vethf9aef496
 cookie=0x0, duration=6492.315s, table=40, n_packets=2213, n_bytes=92946, priority=100,arp,arp_tpa=10.131.0.6 actions=output:vethdd68c7e0
 cookie=0x0, duration=6491.922s, table=40, n_packets=434, n_bytes=18228, priority=100,arp,arp_tpa=10.131.0.7 actions=output:veth2c851942
 cookie=0x0, duration=6481.493s, table=40, n_packets=915, n_bytes=38430, priority=100,arp,arp_tpa=10.131.0.9 actions=output:vethc6770694
 cookie=0x0, duration=6478.446s, table=40, n_packets=500, n_bytes=21000, priority=100,arp,arp_tpa=10.131.0.10 actions=output:vethde8ac3b2
 cookie=0x0, duration=6478.295s, table=40, n_packets=2361, n_bytes=99162, priority=100,arp,arp_tpa=10.131.0.11 actions=output:veth35d9d440
 cookie=0x0, duration=6478.132s, table=40, n_packets=363, n_bytes=15246, priority=100,arp,arp_tpa=10.131.0.12 actions=output:vethbd4b53d8
 cookie=0x0, duration=6477.050s, table=40, n_packets=330, n_bytes=13860, priority=100,arp,arp_tpa=10.131.0.13 actions=output:veth6ee0e6f4
 cookie=0x0, duration=6460.209s, table=40, n_packets=186, n_bytes=7812, priority=100,arp,arp_tpa=10.131.0.15 actions=output:vethcb7df831
 cookie=0x0, duration=6090.632s, table=40, n_packets=330, n_bytes=13860, priority=100,arp,arp_tpa=10.131.0.16 actions=output:veth81c7cce1
 cookie=0x0, duration=6063.792s, table=40, n_packets=7103, n_bytes=298326, priority=100,arp,arp_tpa=10.131.0.17 actions=output:vethde54fe44
 cookie=0x0, duration=6046.097s, table=40, n_packets=1128, n_bytes=47376, priority=100,arp,arp_tpa=10.131.0.18 actions=output:veth46528948
 cookie=0x0, duration=3001.185s, table=40, n_packets=39, n_bytes=1638, priority=100,arp,arp_tpa=10.131.0.27 actions=output:vethd0fc8f3f
 cookie=0x0, duration=6510.385s, table=40, n_packets=351, n_bytes=14742, priority=0 actions=drop
 cookie=0xbe7d4688, duration=6509.929s, table=50, n_packets=5540, n_bytes=232680, priority=100,arp,arp_tpa=10.128.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.130.173->tun_dst,output:vxlan0
 cookie=0x6290ecdb, duration=6509.848s, table=50, n_packets=2476, n_bytes=103992, priority=100,arp,arp_tpa=10.130.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.153.252->tun_dst,output:vxlan0
 cookie=0xcce4c564, duration=6509.755s, table=50, n_packets=1960, n_bytes=82320, priority=100,arp,arp_tpa=10.129.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.206.242->tun_dst,output:vxlan0
 cookie=0x8b508bcf, duration=6509.674s, table=50, n_packets=2989, n_bytes=125538, priority=100,arp,arp_tpa=10.128.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.154.45->tun_dst,output:vxlan0
 cookie=0xf380ca05, duration=6476.693s, table=50, n_packets=2003, n_bytes=84126, priority=100,arp,arp_tpa=10.129.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.182.38->tun_dst,output:vxlan0
 cookie=0x0, duration=6510.385s, table=50, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6510.385s, table=60, n_packets=91584, n_bytes=12725824, priority=200 actions=output:tun0
 cookie=0x0, duration=6510.385s, table=60, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6502.155s, table=70, n_packets=5541, n_bytes=4368159, priority=100,ip,nw_dst=10.131.0.2 actions=load:0x76bbd9->NXM_NX_REG1[],load:0x3->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6493.518s, table=70, n_packets=70693, n_bytes=40131894, priority=100,ip,nw_dst=10.131.0.3 actions=load:0x7b7e67->NXM_NX_REG1[],load:0x4->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6493.301s, table=70, n_packets=6434, n_bytes=1122140, priority=100,ip,nw_dst=10.131.0.5 actions=load:0xab6c55->NXM_NX_REG1[],load:0x6->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6492.315s, table=70, n_packets=28000, n_bytes=4378640, priority=100,ip,nw_dst=10.131.0.6 actions=load:0x8071a4->NXM_NX_REG1[],load:0x7->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6491.922s, table=70, n_packets=78436, n_bytes=53779059, priority=100,ip,nw_dst=10.131.0.7 actions=load:0xab6c55->NXM_NX_REG1[],load:0x8->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6481.493s, table=70, n_packets=37476, n_bytes=6711455, priority=100,ip,nw_dst=10.131.0.9 actions=load:0xab6c55->NXM_NX_REG1[],load:0xa->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6478.446s, table=70, n_packets=3692, n_bytes=830895, priority=100,ip,nw_dst=10.131.0.10 actions=load:0xab6c55->NXM_NX_REG1[],load:0xb->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6478.295s, table=70, n_packets=18263, n_bytes=9780012, priority=100,ip,nw_dst=10.131.0.11 actions=load:0x5a3f8f->NXM_NX_REG1[],load:0xc->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6478.132s, table=70, n_packets=5312, n_bytes=383456, priority=100,ip,nw_dst=10.131.0.12 actions=load:0x3b0e07->NXM_NX_REG1[],load:0xd->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6477.050s, table=70, n_packets=51241, n_bytes=18887146, priority=100,ip,nw_dst=10.131.0.13 actions=load:0xab6c55->NXM_NX_REG1[],load:0xe->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6460.209s, table=70, n_packets=3979, n_bytes=361945, priority=100,ip,nw_dst=10.131.0.15 actions=load:0x5a3f8f->NXM_NX_REG1[],load:0x10->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6090.632s, table=70, n_packets=13346, n_bytes=1035411, priority=100,ip,nw_dst=10.131.0.16 actions=load:0x60a2a9->NXM_NX_REG1[],load:0x11->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6063.792s, table=70, n_packets=474031, n_bytes=1410651090, priority=100,ip,nw_dst=10.131.0.17 actions=load:0xab6c55->NXM_NX_REG1[],load:0x12->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6046.097s, table=70, n_packets=23416, n_bytes=4579684, priority=100,ip,nw_dst=10.131.0.18 actions=load:0xab6c55->NXM_NX_REG1[],load:0x13->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=3001.185s, table=70, n_packets=89, n_bytes=16100, priority=100,ip,nw_dst=10.131.0.27 actions=load:0x3288c->NXM_NX_REG1[],load:0x1c->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6510.385s, table=70, n_packets=46, n_bytes=3404, priority=0 actions=drop
 cookie=0x0, duration=6510.385s, table=80, n_packets=75432, n_bytes=7862210, priority=300,ip,nw_src=10.131.0.1 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6510.293s, table=80, n_packets=608233, n_bytes=1618340466, priority=200,ct_state=+rpl,ip actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6502.107s, table=80, n_packets=3614, n_bytes=301194, priority=50,reg1=0x76bbd9 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6493.458s, table=80, n_packets=8258, n_bytes=1678914, priority=50,reg1=0x7b7e67 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6492.317s, table=80, n_packets=133837, n_bytes=17878992, priority=50,reg1=0xab6c55 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6491.296s, table=80, n_packets=10457, n_bytes=1274071, priority=50,reg1=0x8071a4 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6478.251s, table=80, n_packets=3027, n_bytes=272892, priority=50,reg1=0x5a3f8f actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6478.078s, table=80, n_packets=5312, n_bytes=383456, priority=50,reg1=0x3b0e07 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6090.591s, table=80, n_packets=4868, n_bytes=340760, priority=50,reg1=0x60a2a9 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=3001.138s, table=80, n_packets=0, n_bytes=0, priority=50,reg1=0x3288c actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6510.385s, table=80, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0xbe7d4688, duration=6509.929s, table=90, n_packets=60239, n_bytes=8975481, priority=100,ip,nw_dst=10.128.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.130.173->tun_dst,output:vxlan0
 cookie=0x6290ecdb, duration=6509.848s, table=90, n_packets=33216, n_bytes=5446001, priority=100,ip,nw_dst=10.130.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.153.252->tun_dst,output:vxlan0
 cookie=0xcce4c564, duration=6509.755s, table=90, n_packets=19005, n_bytes=2751174, priority=100,ip,nw_dst=10.129.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.206.242->tun_dst,output:vxlan0
 cookie=0x8b508bcf, duration=6509.674s, table=90, n_packets=95543, n_bytes=305917175, priority=100,ip,nw_dst=10.128.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.154.45->tun_dst,output:vxlan0
 cookie=0xf380ca05, duration=6476.693s, table=90, n_packets=25453, n_bytes=3375726, priority=100,ip,nw_dst=10.129.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.182.38->tun_dst,output:vxlan0
 cookie=0x0, duration=6510.385s, table=90, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6510.385s, table=99, n_packets=0, n_bytes=0, priority=200,tcp,nw_dst=10.0.207.131,tp_dst=53 actions=output:tun0
 cookie=0x0, duration=6510.385s, table=99, n_packets=0, n_bytes=0, priority=200,udp,nw_dst=10.0.207.131,tp_dst=53 actions=output:tun0
 cookie=0x0, duration=6510.385s, table=99, n_packets=260637, n_bytes=28127691, priority=0 actions=goto_table:100
 cookie=0x0, duration=6510.385s, table=100, n_packets=260637, n_bytes=28127691, priority=0 actions=goto_table:101
 cookie=0x0, duration=6510.385s, table=101, n_packets=4753, n_bytes=2771430, priority=150,ct_state=+rpl actions=output:tun0
 cookie=0x0, duration=53.372s, table=101, n_packets=0, n_bytes=0, priority=100,reg0=0x3288c actions=drop
 cookie=0x0, duration=6510.385s, table=101, n_packets=255812, n_bytes=25350885, priority=0 actions=output:tun0
 cookie=0x0, duration=6510.385s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6476.667s, table=111, n_packets=0, n_bytes=0, priority=100 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.130.173->tun_dst,output:vxlan0,set_field:10.0.153.252->tun_dst,output:vxlan0,set_field:10.0.154.45->tun_dst,output:vxlan0,set_field:10.0.182.38->tun_dst,output:vxlan0,set_field:10.0.206.242->tun_dst,output:vxlan0,goto_table:120
 cookie=0x0, duration=6510.385s, table=120, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6510.146s, table=253, n_packets=0, n_bytes=0, actions=note:02.0d.00.00.00.00
sh-4.4# 
~~~

Flows without EgressIP in place (no drop):
~~~
sh-4.4# chroot /host ovs-ofctl -O OpenFlow13 dump-flows br0
 cookie=0x0, duration=6616.541s, table=0, n_packets=1029557, n_bytes=871695834, priority=1000,ct_state=-trk,ip actions=ct(table=0)
 cookie=0x0, duration=6616.541s, table=0, n_packets=76838, n_bytes=8007386, priority=400,ip,in_port=tun0,nw_src=10.131.0.1 actions=goto_table:30
 cookie=0x0, duration=6616.541s, table=0, n_packets=8162, n_bytes=1302788, priority=300,ip,in_port=tun0,nw_src=10.131.0.0/23,nw_dst=10.128.0.0/14 actions=goto_table:25
 cookie=0x0, duration=6616.541s, table=0, n_packets=0, n_bytes=0, priority=250,ip,in_port=tun0,nw_dst=224.0.0.0/4 actions=drop
 cookie=0x0, duration=6616.541s, table=0, n_packets=15556, n_bytes=653352, priority=200,arp,in_port=vxlan0,arp_spa=10.128.0.0/14,arp_tpa=10.131.0.0/23 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=6616.541s, table=0, n_packets=232520, n_bytes=172778921, priority=200,ip,in_port=vxlan0,nw_src=10.128.0.0/14 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=6616.541s, table=0, n_packets=0, n_bytes=0, priority=200,ip,in_port=vxlan0,nw_dst=10.128.0.0/14 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
 cookie=0x0, duration=6616.541s, table=0, n_packets=2380, n_bytes=99960, priority=200,arp,in_port=tun0,arp_spa=10.131.0.1,arp_tpa=10.128.0.0/14 actions=goto_table:30
 cookie=0x0, duration=6616.541s, table=0, n_packets=415061, n_bytes=1188512495, priority=200,ip,in_port=tun0 actions=goto_table:30
 cookie=0x0, duration=6616.541s, table=0, n_packets=0, n_bytes=0, priority=150,in_port=vxlan0 actions=drop
 cookie=0x0, duration=6616.541s, table=0, n_packets=13, n_bytes=998, priority=150,in_port=tun0 actions=drop
 cookie=0x0, duration=6616.541s, table=0, n_packets=18818, n_bytes=790356, priority=100,arp actions=goto_table:20
 cookie=0x0, duration=6616.541s, table=0, n_packets=797026, n_bytes=698917827, priority=100,ip actions=goto_table:20
 cookie=0x0, duration=6616.541s, table=0, n_packets=320, n_bytes=24328, priority=0 actions=drop
 cookie=0xbe7d4688, duration=6616.085s, table=10, n_packets=67537, n_bytes=83608266, priority=100,tun_src=10.0.130.173 actions=goto_table:30
 cookie=0x6290ecdb, duration=6616.004s, table=10, n_packets=32699, n_bytes=25348515, priority=100,tun_src=10.0.153.252 actions=goto_table:30
 cookie=0xcce4c564, duration=6615.911s, table=10, n_packets=19556, n_bytes=20180527, priority=100,tun_src=10.0.206.242 actions=goto_table:30
 cookie=0x8b508bcf, duration=6615.830s, table=10, n_packets=104062, n_bytes=35838676, priority=100,tun_src=10.0.154.45 actions=goto_table:30
 cookie=0xf380ca05, duration=6582.849s, table=10, n_packets=24222, n_bytes=8456289, priority=100,tun_src=10.0.182.38 actions=goto_table:30
 cookie=0x0, duration=6616.541s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6616.541s, table=20, n_packets=0, n_bytes=0, priority=300,udp,tp_dst=4789 actions=drop
 cookie=0x0, duration=6608.311s, table=20, n_packets=275, n_bytes=11550, priority=100,arp,in_port=vethbede2132,arp_spa=10.131.0.2,arp_sha=00:00:0a:83:00:02/00:00:ff:ff:ff:ff actions=load:0x76bbd9->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6599.674s, table=20, n_packets=1317, n_bytes=55314, priority=100,arp,in_port=veth73d7981c,arp_spa=10.131.0.3,arp_sha=00:00:0a:83:00:03/00:00:ff:ff:ff:ff actions=load:0x7b7e67->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6599.457s, table=20, n_packets=563, n_bytes=23646, priority=100,arp,in_port=vethf9aef496,arp_spa=10.131.0.5,arp_sha=00:00:0a:83:00:05/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6598.471s, table=20, n_packets=2240, n_bytes=94080, priority=100,arp,in_port=vethdd68c7e0,arp_spa=10.131.0.6,arp_sha=00:00:0a:83:00:06/00:00:ff:ff:ff:ff actions=load:0x8071a4->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6598.078s, table=20, n_packets=442, n_bytes=18564, priority=100,arp,in_port=veth2c851942,arp_spa=10.131.0.7,arp_sha=00:00:0a:83:00:07/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6587.649s, table=20, n_packets=932, n_bytes=39144, priority=100,arp,in_port=vethc6770694,arp_spa=10.131.0.9,arp_sha=00:00:0a:83:00:09/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6584.602s, table=20, n_packets=506, n_bytes=21252, priority=100,arp,in_port=vethde8ac3b2,arp_spa=10.131.0.10,arp_sha=00:00:0a:83:00:0a/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6584.451s, table=20, n_packets=2387, n_bytes=100254, priority=100,arp,in_port=veth35d9d440,arp_spa=10.131.0.11,arp_sha=00:00:0a:83:00:0b/00:00:ff:ff:ff:ff actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6584.288s, table=20, n_packets=367, n_bytes=15414, priority=100,arp,in_port=vethbd4b53d8,arp_spa=10.131.0.12,arp_sha=00:00:0a:83:00:0c/00:00:ff:ff:ff:ff actions=load:0x3b0e07->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6583.206s, table=20, n_packets=338, n_bytes=14196, priority=100,arp,in_port=veth6ee0e6f4,arp_spa=10.131.0.13,arp_sha=00:00:0a:83:00:0d/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6566.365s, table=20, n_packets=188, n_bytes=7896, priority=100,arp,in_port=vethcb7df831,arp_spa=10.131.0.15,arp_sha=00:00:0a:83:00:0f/00:00:ff:ff:ff:ff actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6196.788s, table=20, n_packets=334, n_bytes=14028, priority=100,arp,in_port=veth81c7cce1,arp_spa=10.131.0.16,arp_sha=00:00:0a:83:00:10/00:00:ff:ff:ff:ff actions=load:0x60a2a9->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6169.948s, table=20, n_packets=7249, n_bytes=304458, priority=100,arp,in_port=vethde54fe44,arp_spa=10.131.0.17,arp_sha=00:00:0a:83:00:11/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6152.253s, table=20, n_packets=1148, n_bytes=48216, priority=100,arp,in_port=veth46528948,arp_spa=10.131.0.18,arp_sha=00:00:0a:83:00:12/00:00:ff:ff:ff:ff actions=load:0xab6c55->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=3107.341s, table=20, n_packets=39, n_bytes=1638, priority=100,arp,in_port=vethd0fc8f3f,arp_spa=10.131.0.27,arp_sha=00:00:0a:83:00:1b/00:00:ff:ff:ff:ff actions=load:0x3288c->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=6608.311s, table=20, n_packets=4415, n_bytes=1481077, priority=100,ip,in_port=vethbede2132,nw_src=10.131.0.2 actions=load:0x76bbd9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6599.674s, table=20, n_packets=95059, n_bytes=21261893, priority=100,ip,in_port=veth73d7981c,nw_src=10.131.0.3 actions=load:0x7b7e67->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6599.457s, table=20, n_packets=6371, n_bytes=2335363, priority=100,ip,in_port=vethf9aef496,nw_src=10.131.0.5 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6598.471s, table=20, n_packets=27644, n_bytes=5703909, priority=100,ip,in_port=vethdd68c7e0,nw_src=10.131.0.6 actions=load:0x8071a4->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6598.078s, table=20, n_packets=82411, n_bytes=543189101, priority=100,ip,in_port=veth2c851942,nw_src=10.131.0.7 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6587.649s, table=20, n_packets=41629, n_bytes=17367386, priority=100,ip,in_port=vethc6770694,nw_src=10.131.0.9 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6584.602s, table=20, n_packets=3381, n_bytes=2600941, priority=100,ip,in_port=vethde8ac3b2,nw_src=10.131.0.10 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6584.451s, table=20, n_packets=22403, n_bytes=7473840, priority=100,ip,in_port=veth35d9d440,nw_src=10.131.0.11 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6584.288s, table=20, n_packets=2698, n_bytes=203132, priority=100,ip,in_port=vethbd4b53d8,nw_src=10.131.0.12 actions=load:0x3b0e07->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6583.206s, table=20, n_packets=42644, n_bytes=16483826, priority=100,ip,in_port=veth6ee0e6f4,nw_src=10.131.0.13 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6566.365s, table=20, n_packets=3792, n_bytes=408323, priority=100,ip,in_port=vethcb7df831,nw_src=10.131.0.15 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6196.788s, table=20, n_packets=8818, n_bytes=2197518, priority=100,ip,in_port=veth81c7cce1,nw_src=10.131.0.16 actions=load:0x60a2a9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6169.948s, table=20, n_packets=403590, n_bytes=61647590, priority=100,ip,in_port=vethde54fe44,nw_src=10.131.0.17 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6152.253s, table=20, n_packets=21984, n_bytes=11301866, priority=100,ip,in_port=veth46528948,nw_src=10.131.0.18 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=3107.341s, table=20, n_packets=140, n_bytes=11374, priority=100,ip,in_port=vethd0fc8f3f,nw_src=10.131.0.27 actions=load:0x3288c->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6616.541s, table=20, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6608.311s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.2 actions=load:0x76bbd9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6599.674s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.3 actions=load:0x7b7e67->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6599.457s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.5 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6598.471s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.6 actions=load:0x8071a4->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6598.078s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.7 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6587.649s, table=25, n_packets=3286, n_bytes=475949, priority=100,ip,nw_src=10.131.0.9 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6584.602s, table=25, n_packets=773, n_bytes=318198, priority=100,ip,nw_src=10.131.0.10 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6584.451s, table=25, n_packets=2033, n_bytes=230278, priority=100,ip,nw_src=10.131.0.11 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6584.288s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.12 actions=load:0x3b0e07->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6583.206s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.13 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6566.365s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.15 actions=load:0x5a3f8f->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6196.788s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.16 actions=load:0x60a2a9->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6169.948s, table=25, n_packets=0, n_bytes=0, priority=100,ip,nw_src=10.131.0.17 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6152.253s, table=25, n_packets=1712, n_bytes=237223, priority=100,ip,nw_src=10.131.0.18 actions=load:0xab6c55->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=3107.341s, table=25, n_packets=80, n_bytes=6912, priority=100,ip,nw_src=10.131.0.27 actions=load:0x3288c->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=6616.541s, table=25, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6616.449s, table=27, n_packets=92867, n_bytes=12903346, priority=300,ip,nw_dst=172.30.0.0/16 actions=goto_table:30
 cookie=0x0, duration=6616.449s, table=27, n_packets=222839, n_bytes=629503095, priority=200,ct_state=+rpl,ip actions=goto_table:30
 cookie=0x0, duration=6608.263s, table=27, n_packets=0, n_bytes=0, priority=50,reg0=0x76bbd9 actions=goto_table:30
 cookie=0x0, duration=6599.614s, table=27, n_packets=77944, n_bytes=13233159, priority=50,reg0=0x7b7e67 actions=goto_table:30
 cookie=0x0, duration=6598.473s, table=27, n_packets=397945, n_bytes=43053700, priority=50,reg0=0xab6c55 actions=goto_table:30
 cookie=0x0, duration=6597.452s, table=27, n_packets=993, n_bytes=150120, priority=50,reg0=0x8071a4 actions=goto_table:30
 cookie=0x0, duration=6584.407s, table=27, n_packets=12305, n_bytes=1348286, priority=50,reg0=0x5a3f8f actions=goto_table:30
 cookie=0x0, duration=6584.234s, table=27, n_packets=0, n_bytes=0, priority=50,reg0=0x3b0e07 actions=goto_table:30
 cookie=0x0, duration=6196.747s, table=27, n_packets=0, n_bytes=0, priority=50,reg0=0x60a2a9 actions=goto_table:30
 cookie=0x0, duration=3107.294s, table=27, n_packets=140, n_bytes=11374, priority=50,reg0=0x3288c actions=goto_table:30
 cookie=0x0, duration=6616.541s, table=27, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6616.449s, table=30, n_packets=398893, n_bytes=52182880, priority=100,ct_state=-rpl,ip,nw_src=10.128.0.0/14,nw_dst=10.128.0.0/14 actions=ct(commit,table=31)
 cookie=0x0, duration=6616.541s, table=30, n_packets=1167436, n_bytes=2018877399, priority=0 actions=goto_table:31
 cookie=0x0, duration=6616.541s, table=31, n_packets=2371, n_bytes=99582, priority=300,arp,arp_tpa=10.131.0.1 actions=output:tun0
 cookie=0x0, duration=6616.541s, table=31, n_packets=66871, n_bytes=19625044, priority=300,ip,nw_dst=10.131.0.1 actions=output:tun0
 cookie=0x0, duration=6616.541s, table=31, n_packets=618301, n_bytes=1645454115, priority=250,ct_state=+rpl,ip,nw_dst=10.131.0.0/23 actions=ct(table=70,nat)
 cookie=0x0, duration=6616.541s, table=31, n_packets=19163, n_bytes=804846, priority=200,arp,arp_tpa=10.131.0.0/23 actions=goto_table:40
 cookie=0x0, duration=6616.541s, table=31, n_packets=249310, n_bytes=30518526, priority=200,ip,nw_dst=10.131.0.0/23 actions=goto_table:70
 cookie=0x0, duration=6616.541s, table=31, n_packets=15220, n_bytes=639240, priority=100,arp,arp_tpa=10.128.0.0/14 actions=goto_table:50
 cookie=0x0, duration=6616.541s, table=31, n_packets=237282, n_bytes=332456114, priority=100,ip,nw_dst=10.128.0.0/14 actions=goto_table:90
 cookie=0x0, duration=6616.541s, table=31, n_packets=92867, n_bytes=12903346, priority=100,ip,nw_dst=172.30.0.0/16 actions=goto_table:60
 cookie=0x0, duration=6616.541s, table=31, n_packets=0, n_bytes=0, priority=50,ip,in_port=vxlan0,nw_dst=224.0.0.0/4 actions=goto_table:120
 cookie=0x0, duration=6616.541s, table=31, n_packets=0, n_bytes=0, priority=25,ip,nw_dst=224.0.0.0/4 actions=goto_table:110
 cookie=0x0, duration=6616.541s, table=31, n_packets=264944, n_bytes=28559466, priority=0,ip actions=goto_table:99
 cookie=0x0, duration=6616.541s, table=31, n_packets=0, n_bytes=0, priority=0,arp actions=drop
 cookie=0x0, duration=6608.311s, table=40, n_packets=275, n_bytes=11550, priority=100,arp,arp_tpa=10.131.0.2 actions=output:vethbede2132
 cookie=0x0, duration=6599.674s, table=40, n_packets=1317, n_bytes=55314, priority=100,arp,arp_tpa=10.131.0.3 actions=output:veth73d7981c
 cookie=0x0, duration=6599.457s, table=40, n_packets=563, n_bytes=23646, priority=100,arp,arp_tpa=10.131.0.5 actions=output:vethf9aef496
 cookie=0x0, duration=6598.471s, table=40, n_packets=2240, n_bytes=94080, priority=100,arp,arp_tpa=10.131.0.6 actions=output:vethdd68c7e0
 cookie=0x0, duration=6598.078s, table=40, n_packets=439, n_bytes=18438, priority=100,arp,arp_tpa=10.131.0.7 actions=output:veth2c851942
 cookie=0x0, duration=6587.649s, table=40, n_packets=932, n_bytes=39144, priority=100,arp,arp_tpa=10.131.0.9 actions=output:vethc6770694
 cookie=0x0, duration=6584.602s, table=40, n_packets=506, n_bytes=21252, priority=100,arp,arp_tpa=10.131.0.10 actions=output:vethde8ac3b2
 cookie=0x0, duration=6584.451s, table=40, n_packets=2384, n_bytes=100128, priority=100,arp,arp_tpa=10.131.0.11 actions=output:veth35d9d440
 cookie=0x0, duration=6584.288s, table=40, n_packets=367, n_bytes=15414, priority=100,arp,arp_tpa=10.131.0.12 actions=output:vethbd4b53d8
 cookie=0x0, duration=6583.206s, table=40, n_packets=338, n_bytes=14196, priority=100,arp,arp_tpa=10.131.0.13 actions=output:veth6ee0e6f4
 cookie=0x0, duration=6566.365s, table=40, n_packets=188, n_bytes=7896, priority=100,arp,arp_tpa=10.131.0.15 actions=output:vethcb7df831
 cookie=0x0, duration=6196.788s, table=40, n_packets=334, n_bytes=14028, priority=100,arp,arp_tpa=10.131.0.16 actions=output:veth81c7cce1
 cookie=0x0, duration=6169.948s, table=40, n_packets=7249, n_bytes=304458, priority=100,arp,arp_tpa=10.131.0.17 actions=output:vethde54fe44
 cookie=0x0, duration=6152.253s, table=40, n_packets=1148, n_bytes=48216, priority=100,arp,arp_tpa=10.131.0.18 actions=output:veth46528948
 cookie=0x0, duration=3107.341s, table=40, n_packets=39, n_bytes=1638, priority=100,arp,arp_tpa=10.131.0.27 actions=output:vethd0fc8f3f
 cookie=0x0, duration=6616.541s, table=40, n_packets=351, n_bytes=14742, priority=0 actions=drop
 cookie=0xbe7d4688, duration=6616.085s, table=50, n_packets=5640, n_bytes=236880, priority=100,arp,arp_tpa=10.128.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.130.173->tun_dst,output:vxlan0
 cookie=0x6290ecdb, duration=6616.004s, table=50, n_packets=2522, n_bytes=105924, priority=100,arp,arp_tpa=10.130.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.153.252->tun_dst,output:vxlan0
 cookie=0xcce4c564, duration=6615.911s, table=50, n_packets=1987, n_bytes=83454, priority=100,arp,arp_tpa=10.129.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.206.242->tun_dst,output:vxlan0
 cookie=0x8b508bcf, duration=6615.830s, table=50, n_packets=3035, n_bytes=127470, priority=100,arp,arp_tpa=10.128.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.154.45->tun_dst,output:vxlan0
 cookie=0xf380ca05, duration=6582.849s, table=50, n_packets=2036, n_bytes=85512, priority=100,arp,arp_tpa=10.129.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.182.38->tun_dst,output:vxlan0
 cookie=0x0, duration=6616.541s, table=50, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6616.541s, table=60, n_packets=92867, n_bytes=12903346, priority=200 actions=output:tun0
 cookie=0x0, duration=6616.541s, table=60, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6608.311s, table=70, n_packets=5632, n_bytes=4383129, priority=100,ip,nw_dst=10.131.0.2 actions=load:0x76bbd9->NXM_NX_REG1[],load:0x3->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6599.674s, table=70, n_packets=71779, n_bytes=40701178, priority=100,ip,nw_dst=10.131.0.3 actions=load:0x7b7e67->NXM_NX_REG1[],load:0x4->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6599.457s, table=70, n_packets=6557, n_bytes=1148759, priority=100,ip,nw_dst=10.131.0.5 actions=load:0xab6c55->NXM_NX_REG1[],load:0x6->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6598.471s, table=70, n_packets=28411, n_bytes=4429275, priority=100,ip,nw_dst=10.131.0.6 actions=load:0x8071a4->NXM_NX_REG1[],load:0x7->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6598.078s, table=70, n_packets=79676, n_bytes=54451326, priority=100,ip,nw_dst=10.131.0.7 actions=load:0xab6c55->NXM_NX_REG1[],load:0x8->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6587.649s, table=70, n_packets=38090, n_bytes=6821097, priority=100,ip,nw_dst=10.131.0.9 actions=load:0xab6c55->NXM_NX_REG1[],load:0xa->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6584.602s, table=70, n_packets=3738, n_bytes=834275, priority=100,ip,nw_dst=10.131.0.10 actions=load:0xab6c55->NXM_NX_REG1[],load:0xb->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6584.451s, table=70, n_packets=18578, n_bytes=9941634, priority=100,ip,nw_dst=10.131.0.11 actions=load:0x5a3f8f->NXM_NX_REG1[],load:0xc->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6584.288s, table=70, n_packets=5396, n_bytes=389336, priority=100,ip,nw_dst=10.131.0.12 actions=load:0x3b0e07->NXM_NX_REG1[],load:0xd->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6583.206s, table=70, n_packets=52055, n_bytes=19071170, priority=100,ip,nw_dst=10.131.0.13 actions=load:0xab6c55->NXM_NX_REG1[],load:0xe->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6566.365s, table=70, n_packets=4042, n_bytes=367715, priority=100,ip,nw_dst=10.131.0.15 actions=load:0x5a3f8f->NXM_NX_REG1[],load:0x10->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6196.788s, table=70, n_packets=13574, n_bytes=1053031, priority=100,ip,nw_dst=10.131.0.16 actions=load:0x60a2a9->NXM_NX_REG1[],load:0x11->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6169.948s, table=70, n_packets=482561, n_bytes=1436347051, priority=100,ip,nw_dst=10.131.0.17 actions=load:0xab6c55->NXM_NX_REG1[],load:0x12->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6152.253s, table=70, n_packets=23823, n_bytes=4656689, priority=100,ip,nw_dst=10.131.0.18 actions=load:0xab6c55->NXM_NX_REG1[],load:0x13->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=3107.341s, table=70, n_packets=89, n_bytes=16100, priority=100,ip,nw_dst=10.131.0.27 actions=load:0x3288c->NXM_NX_REG1[],load:0x1c->NXM_NX_REG2[],goto_table:80
 cookie=0x0, duration=6616.541s, table=70, n_packets=46, n_bytes=3404, priority=0 actions=drop
 cookie=0x0, duration=6616.541s, table=80, n_packets=76644, n_bytes=7988568, priority=300,ip,nw_src=10.131.0.1 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6616.449s, table=80, n_packets=618315, n_bytes=1645455074, priority=200,ct_state=+rpl,ip actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6608.263s, table=80, n_packets=3680, n_bytes=306430, priority=50,reg1=0x76bbd9 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6599.614s, table=80, n_packets=8346, n_bytes=1700062, priority=50,reg1=0x7b7e67 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6598.473s, table=80, n_packets=136077, n_bytes=18172600, priority=50,reg1=0xab6c55 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6597.452s, table=80, n_packets=10599, n_bytes=1291377, priority=50,reg1=0x8071a4 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6584.407s, table=80, n_packets=3077, n_bytes=277267, priority=50,reg1=0x5a3f8f actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6584.234s, table=80, n_packets=5396, n_bytes=389336, priority=50,reg1=0x3b0e07 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6196.747s, table=80, n_packets=4956, n_bytes=346920, priority=50,reg1=0x60a2a9 actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=3107.294s, table=80, n_packets=0, n_bytes=0, priority=50,reg1=0x3288c actions=output:NXM_NX_REG2[]
 cookie=0x0, duration=6616.541s, table=80, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0xbe7d4688, duration=6616.085s, table=90, n_packets=61207, n_bytes=9116063, priority=100,ip,nw_dst=10.128.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.130.173->tun_dst,output:vxlan0
 cookie=0x6290ecdb, duration=6616.004s, table=90, n_packets=33769, n_bytes=5537391, priority=100,ip,nw_dst=10.130.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.153.252->tun_dst,output:vxlan0
 cookie=0xcce4c564, duration=6615.911s, table=90, n_packets=19310, n_bytes=2794787, priority=100,ip,nw_dst=10.129.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.206.242->tun_dst,output:vxlan0
 cookie=0x8b508bcf, duration=6615.830s, table=90, n_packets=97109, n_bytes=311574955, priority=100,ip,nw_dst=10.128.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.154.45->tun_dst,output:vxlan0
 cookie=0xf380ca05, duration=6582.849s, table=90, n_packets=25887, n_bytes=3432918, priority=100,ip,nw_dst=10.129.2.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.182.38->tun_dst,output:vxlan0
 cookie=0x0, duration=6616.541s, table=90, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6616.541s, table=99, n_packets=0, n_bytes=0, priority=200,tcp,nw_dst=10.0.207.131,tp_dst=53 actions=output:tun0
 cookie=0x0, duration=6616.541s, table=99, n_packets=0, n_bytes=0, priority=200,udp,nw_dst=10.0.207.131,tp_dst=53 actions=output:tun0
 cookie=0x0, duration=6616.541s, table=99, n_packets=264944, n_bytes=28559466, priority=0 actions=goto_table:100
 cookie=0x0, duration=6616.541s, table=100, n_packets=264944, n_bytes=28559466, priority=0 actions=goto_table:101
 cookie=0x0, duration=6616.541s, table=101, n_packets=4792, n_bytes=2795132, priority=150,ct_state=+rpl actions=output:tun0
 cookie=0x0, duration=6616.541s, table=101, n_packets=260080, n_bytes=25758958, priority=0 actions=output:tun0
 cookie=0x0, duration=6616.541s, table=110, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6582.823s, table=111, n_packets=0, n_bytes=0, priority=100 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.0.130.173->tun_dst,output:vxlan0,set_field:10.0.153.252->tun_dst,output:vxlan0,set_field:10.0.154.45->tun_dst,output:vxlan0,set_field:10.0.182.38->tun_dst,output:vxlan0,set_field:10.0.206.242->tun_dst,output:vxlan0,goto_table:120
 cookie=0x0, duration=6616.541s, table=120, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x0, duration=6616.302s, table=253, n_packets=0, n_bytes=0, actions=note:02.0d.00.00.00.00
~~~

Comment 2 Andreas Karis 2022-04-29 11:39:49 UTC

Ouch ...

~~~
[akaris@linux must-gather.local.2579743636966703416]$ omg get pods -n openshift-sdn -o wide | grep ip-10-0-207-131.ec2.internal
sdn-4fc9d             2/2    Running  0         1h51m  10.0.207.131  ip-10-0-207-131.ec2.internal


[akaris@linux must-gather.local.2579743636966703416]$ omg logs -n openshift-sdn sdn-4fc9d -c sdn | tail
2022-04-29T11:13:54.845914139Z E0429 11:13:54.845902    1796 reflector.go:138] k8s.io/client-go.4/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:14:38.351143719Z W0429 11:14:38.351104    1796 reflector.go:324] k8s.io/client-go.4/tools/cache/reflector.go:167: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:14:38.351143719Z E0429 11:14:38.351130    1796 reflector.go:138] k8s.io/client-go.4/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:15:28.924555741Z W0429 11:15:28.924515    1796 reflector.go:324] k8s.io/client-go.4/tools/cache/reflector.go:167: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:15:28.924555741Z E0429 11:15:28.924544    1796 reflector.go:138] k8s.io/client-go.4/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:16:21.184065683Z W0429 11:16:21.184025    1796 reflector.go:324] k8s.io/client-go.4/tools/cache/reflector.go:167: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:16:21.184065683Z E0429 11:16:21.184053    1796 reflector.go:138] k8s.io/client-go.4/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:17:09.420543631Z W0429 11:17:09.420504    1796 reflector.go:324] k8s.io/client-go.4/tools/cache/reflector.go:167: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
2022-04-29T11:17:09.420543631Z E0429 11:17:09.420530    1796 reflector.go:138] k8s.io/client-go.4/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-sdn:sdn" cannot list resource "nodes" in API group "" at the cluster scope
~~~

Comment 3 Andreas Karis 2022-04-29 11:40:51 UTC

Caused by:
https://github.com/openshift/cluster-network-operator/pull/1350/commits/98ab16976a4f5d1524abcc738ed0121a36f82a0b

Comment 4 Andreas Karis 2022-04-29 11:42:30 UTC

~~~
[akaris@linux ~]$ oc get clusterrole openshift-sdn -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2022-04-29T09:19:35Z"
  name: openshift-sdn
  ownerReferences:
  - apiVersion: operator.openshift.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Network
    name: cluster
    uid: 89267bf8-012a-412f-a783-5c5156128652
  resourceVersion: "2956"
  uid: 0a639727-28cf-4196-9e8d-e47afcff1da4
rules:
- apiGroups:
  - network.openshift.io
  resources:
  - clusternetworks
  - egressnetworkpolicies
  - hostsubnets
  - netnamespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  - endpoints
  - services
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
  - update
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
~~~

Comment 9 errata-xmlrpc 2022-08-10 11:09:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069

Note You need to log in before you can comment on or make changes to this bug.