Bug 1917875

Summary: High number of flows on large Scale OpenShift Environments with number of services/pods
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Sai Sindhur Malleni <smalleni>
Component: ovn2.13Assignee: Dumitru Ceara <dceara>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: high Docs Contact:
Priority: high    
Version: FDP 20.HCC: cfontain, ctrautma, dblack, dcbw, dceara, fbaudin, fhallal, fiezzi, fleitner, i.maximets, jishi, mark.d.gray, nusiddiq, ralongi, rkhan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-15 14:36:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sai Sindhur Malleni 2021-01-19 14:55:33 UTC 
Description of problem:
We installed a a 500 node OpenShift Cluster and at an idle state with just the cluster operators and realted pods running there are 1.4 million OpenFlow rules per worker node.

On running a test that creates 4277 "app" pods,4722 services backed by 4722 postgres pods, we see 8 million flows per worker node at the end of teh test. Reduced from 12 million with the fix from https://bugzilla.redhat.com/show_bug.cgi?id=1903224 

Version-Release number of selected component (if applicable):
sh-4.4# rpm -qa | grep ovn
ovn2.13-20.12.0-1prerel.el8fdn.x86_64
ovn2.13-host-20.12.0-1prerel.el8fdn.x86_64
ovn2.13-central-20.12.0-1prerel.el8fdn.x86_64
ovn2.13-vtep-20.12.0-1prerel.el8fdn.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Deploy a large OCP cluster with OVN
2. Deploy a kubelet-density-heavy workload that creates pod and, services
3. Check number of flows per worker node

Actual results:
There are 8 million OpenFlow rules per worker node, we typically expect lower. Note that even idle cluster has 1.4 millon flows.

Expected results:
Lower OpenFlow flows

Additional info:
Freshly installed cluster
[root@worker113-fc640 core]# ovs-ofctl dump-aggregate br-int
NXST_AGGREGATE reply (xid=0x4): packet_count=344368172 byte_count=62583087031 flow_count=1408875


After running workload that creates pods/services
[root@worker015-fc640 core]# ovs-ofctl dump-aggregate br-int
NXST_AGGREGATE reply (xid=0x4): packet_count=219461024 byte_count=36304624630 flow_count=8422408
Comment 5 Dan Williams 2021-02-05 15:35:15 UTC 
Dumitru has developed a fix based on the OVS learn action that significantly reduces the number of LB hairpin flows (which are the majority of flows in this case). He expects to submit that patch upstream next week.
Comment 6 Dumitru Ceara 2021-02-05 22:31:54 UTC 
Patch to reduce number of LB hairpin flows posted for review: http://patchwork.ozlabs.org/project/ovn/list/?series=228263
Comment 11 Jianlin Shi 2021-02-18 02:00:56 UTC 
tested with following script:

systemctl start openvswitch
systemctl start ovn-northd

ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642

ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:1.1.175.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=1.1.175.25
systemctl start ovn-controller

NODES=50                                                                                            
  VIPS=20                                                                                             
  BACKENDS=10                                                                                         
  ovn-nbctl lr-add rtr                                                                                
  for ((i = 1; i <= $NODES; i++)); do                                                                 
      ovn-nbctl \
          -- ls-add ls$i \
          -- lsp-add ls$i vm$i \
          -- lsp-add ls$i ls$i-rtr \
          -- lsp-set-type ls$i-rtr router \
          -- lsp-set-options ls$i-rtr router-port=rtr-ls$i \
          -- lrp-add rtr rtr-ls$i 00:00:00:00:01:00 42.42.42.$i/24
  done                                                                                                
                                                                                                      
  for ((i = 1; i <= $VIPS; i++)); do                                                                  
      lb=lb$i                                                                                                                                                                                               
      vip=10.10.10.$i:1                                                                               
      bip=20.20.20.1:2                                                                                
      for ((j = 2; j <= $BACKENDS; j++)); do                                                          
          bip="$bip,20.20.20.$j:2"                                                                    
      done                                                                                            
      ovn-nbctl lb-add $lb $vip $bip
  done                                                                                                
                                                                                                      
  for ((i = 1; i <= $NODES; i++)); do                                                                 
      for ((j = 1; j <= $VIPS; j++)); do                                                              
          ovn-nbctl ls-lb-add ls$i lb$j                                                               
      done                                                                                            
  done                                                                                                
                                                                                                      
  ovs-vsctl add-port br-int vm1 \
      -- set interface vm1 type=internal \
      -- set interface vm1 external-ids:iface-id=vm1
ovs-ofctl dump-aggregate br-int table=69

result on 20.12.0-15:


[root@wsfd-advnetlab21 bz1917875]# ovs-ofctl dump-aggregate br-int table=69
NXST_AGGREGATE reply (xid=0x8): packet_count=0 byte_count=0 flow_count=10000

<=== 10k flows

[root@wsfd-advnetlab21 bz1917875]# rpm -qa | grep -E "openvswitch2.13|ovn2.13"
openvswitch2.13-2.13.0-82.el7fdp.x86_64
ovn2.13-20.12.0-15.el7fdp.x86_64
ovn2.13-central-20.12.0-15.el7fdp.x86_64
ovn2.13-host-20.12.0-15.el7fdp.x86_64

result on 20.12.0-17:

[root@wsfd-advnetlab21 bz1917875]# rpm -qa | grep -E "openvswitch2.13|ovn2.13"
openvswitch2.13-2.13.0-82.el7fdp.x86_64
ovn2.13-central-20.12.0-17.el7fdp.x86_64
ovn2.13-20.12.0-17.el7fdp.x86_64
ovn2.13-host-20.12.0-17.el7fdp.x86_64

[root@wsfd-advnetlab21 bz1917875]# ovs-ofctl dump-aggregate br-int table=69                                                                                                                                
NXST_AGGREGATE reply (xid=0x8): packet_count=0 byte_count=0 flow_count=0

[root@wsfd-advnetlab21 bz1917875]# ovs-ofctl dump-flows br-int table=68
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.1,nw_dst=20.20.20.1,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.2,nw_dst=20.20.20.2,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.3,nw_dst=20.20.20.3,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.4,nw_dst=20.20.20.4,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.5,nw_dst=20.20.20.5,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.6,nw_dst=20.20.20.6,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.7,nw_dst=20.20.20.7,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.8,nw_dst=20.20.20.8,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.9,nw_dst=20.20.20.9,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.10,nw_dst=20.20.20.10,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])

<=== learn action

Dumitru , Does the result show the fix of the issue?
Comment 13 Dumitru Ceara 2021-02-18 09:41:23 UTC 
(In reply to Jianlin Shi from comment #11)

[...]

> 
> result on 20.12.0-15:
> 
> 
> [root@wsfd-advnetlab21 bz1917875]# ovs-ofctl dump-aggregate br-int table=69
> NXST_AGGREGATE reply (xid=0x8): packet_count=0 byte_count=0 flow_count=10000
> 
> <=== 10k flows
> 
> [root@wsfd-advnetlab21 bz1917875]# rpm -qa | grep -E
> "openvswitch2.13|ovn2.13"
> openvswitch2.13-2.13.0-82.el7fdp.x86_64
> ovn2.13-20.12.0-15.el7fdp.x86_64
> ovn2.13-central-20.12.0-15.el7fdp.x86_64
> ovn2.13-host-20.12.0-15.el7fdp.x86_64
> 
> result on 20.12.0-17:
> 
> [root@wsfd-advnetlab21 bz1917875]# rpm -qa | grep -E
> "openvswitch2.13|ovn2.13"
> openvswitch2.13-2.13.0-82.el7fdp.x86_64
> ovn2.13-central-20.12.0-17.el7fdp.x86_64
> ovn2.13-20.12.0-17.el7fdp.x86_64
> ovn2.13-host-20.12.0-17.el7fdp.x86_64
> 
> [root@wsfd-advnetlab21 bz1917875]# ovs-ofctl dump-aggregate br-int table=69 
> 
> NXST_AGGREGATE reply (xid=0x8): packet_count=0 byte_count=0 flow_count=0
> 
> [root@wsfd-advnetlab21 bz1917875]# ovs-ofctl dump-flows br-int table=68
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.1,nw_dst=20.20.20.1,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.2,nw_dst=20.20.20.2,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.3,nw_dst=20.20.20.3,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.4,nw_dst=20.20.20.4,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.5,nw_dst=20.20.20.5,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.6,nw_dst=20.20.20.6,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.7,nw_dst=20.20.20.7,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.8,nw_dst=20.20.20.8,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.9,nw_dst=20.20.20.9,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
>  cookie=0xc0d5e49, duration=524.177s, table=68, n_packets=0, n_bytes=0,
> priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.10,nw_dst=20.20.20.10,
> tp_dst=2
> actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,
> cookie=0xc0d5e49,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.
> 10.10.20,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1-
> >NXM_NX_REG10[7])
> 
> <=== learn action
> 
> Dumitru , Does the result show the fix of the issue?

To me this looks OK, although it would be great if this could also be tested
in a large scale OpenShift deployment.

Regards,
Dumitru
Comment 14 Jianlin Shi 2021-02-19 03:25:24 UTC 
also verified on rhel8:

[root@dell-per740-12 bz1917875]# rpm -qa | grep -E "openvswitch2.13|ovn2.13"
openvswitch2.13-2.13.0-95.el8fdp.x86_64
ovn2.13-host-20.12.0-20.el8fdp.x86_64
ovn2.13-20.12.0-20.el8fdp.x86_64
ovn2.13-central-20.12.0-20.el8fdp.x86_64

[root@dell-per740-12 bz1917875]# ovs-ofctl dump-aggregate br-int table=69                                                                                                                                   NXST_AGGREGATE reply (xid=0x8): packet_count=0 byte_count=0 flow_count=0                                                                                                                                   
[root@dell-per740-12 bz1917875]# ovs-ofctl dump-flows br-int table=68                                                                                                                                        cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.10,nw_dst=20.20.20.10,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,de
lete_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                   cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.3,nw_dst=20.20.20.3,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,dele
te_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                     cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.6,nw_dst=20.20.20.6,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,dele
te_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                     cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.5,nw_dst=20.20.20.5,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,dele
te_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                     cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.4,nw_dst=20.20.20.4,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,dele
te_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                     cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.1,nw_dst=20.20.20.1,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,dele
te_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
 cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.2,nw_dst=20.20.20.2,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                   
 cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.8,nw_dst=20.20.20.8,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                   
 cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.9,nw_dst=20.20.20.9,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])                                   
 cookie=0xaf2c2f7e, duration=47.470s, table=68, n_packets=0, n_bytes=0, priority=100,ct_label=0x2/0x2,tcp,nw_src=20.20.20.7,nw_dst=20.20.20.7,tp_dst=2 actions=load:0x1->NXM_NX_REG10[7],learn(table=69,delete_learned,cookie=0xaf2c2f7e,OXM_OF_METADATA[],eth_type=0x800,NXM_OF_IP_SRC[],ip_dst=10.10.10.13,nw_proto=6,NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],load:0x1->NXM_NX_REG10[7])
Comment 16 errata-xmlrpc 2021-03-15 14:36:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0836