The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.
Bug 1953278 - Pod to external traffic is not offloaded with OVN Kubernetes: combination of CT/CT(nat) in the datapath
Summary: Pod to external traffic is not offloaded with OVN Kubernetes: combination of ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: OVN
Version: RHEL 8.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Numan Siddique
QA Contact: Jianlin Shi
URL:
Whiteboard:
: 1988189 (view as bug list)
Depends On:
Blocks: 1956740 1961063 1983894 2131355
TreeView+ depends on / blocked
 
Reported: 2021-04-25 07:10 UTC by Alaa Hleihel (NVIDIA Mellanox)
Modified: 2023-06-15 06:41 UTC (History)
11 users (show)

Fixed In Version: ovn21.09-21.09.0-8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1956740 (view as bug list)
Environment:
Last Closed: 2023-03-13 07:13:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
local conf.db (139.24 KB, text/plain)
2021-05-02 06:54 UTC, Adrian Chiris
no flags Details
ovn nb db (215.59 KB, text/plain)
2021-05-02 06:55 UTC, Adrian Chiris
no flags Details
ovn sb db (1.09 MB, text/plain)
2021-05-02 06:55 UTC, Adrian Chiris
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-1277 0 None None None 2021-08-09 02:45:32 UTC

Description Alaa Hleihel (NVIDIA Mellanox) 2021-04-25 07:10:55 UTC
Opening this BZ following a meeting between Adrian (NVIDIA) and Numan (Red Hat).


Description of problem:


Pod to external traffic is not offloaded with OVN Kubernetes upstream branch when using OVN 21.03 with V2 version applied
of the series: https://patchwork.ozlabs.org/project/ovn/list/?series=240351

we are still seeing issues with the datapath rules on ingress where there is a CT then a CT NAT actions on the same zone which breaks HW offload capability.

OS: CentOS 8.3
OVN Kubernetes version : master branch, commit: 5a1fdceb203259498c41d2058bebee95265b44e6
OVN: 21.03 branch commit: 36276f75d5b4705e341b7992c36bff2811a8ff95 with series https://patchwork.ozlabs.org/project/ovn/list/?series=240351 applied
OVS: 2.14.1

Repro steps:
On Kubernetes setup with the above components with SR-IOV and hw offload enabled (ovs has hw-offload enabled) run pod to external traffic.


Datapath rules: (problematic flows in bold)

recirc_id(0),in_port(enp3s0f0),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(dst=32768/0x8000), packets:122237, bytes:9466915, used:0.380s, actions:ct(zone=64000),recirc(0x3)                   
recirc_id(0),in_port(enp3s0f0),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(dst=8192/0xe000), packets:18, bytes:936, used:0.580s, actions:ct(zone=64000),recirc(0x3)                            
ct_state(-est+trk),recirc_id(0x3),in_port(enp3s0f0),eth(dst=b8:ce:f6:09:fa:38),eth_type(0x0800),ipv4(proto=6,frag=no), packets:1914, bytes:698451, used:0.580s, actions:brenp3s0f0            
ct_state(+est+rpl+trk),ct_label(0/0x1),recirc_id(0x3),in_port(enp3s0f0),eth(src=24:8a:07:8d:ae:3f,dst=b8:ce:f6:09:fa:38),eth_type(0x0800),ipv4(src=10.11.0.0/255.255.255.248,dst=10.11.0.11,proto=6,ttl=64,frag=no), packets:123315, bytes:6756575, used:0.000s, actions:ct_clear,ct(zone=16),recirc(0x2a)                                                                                  
recirc_id(0x2a),in_port(enp3s0f0),eth_type(0x0800),ipv4(dst=10.11.0.11,proto=6,frag=no),tcp(dst=32768/0x8000), packets:123315, bytes:6756575, used:0.000s, actions:ct(nat),recirc(0x2b)       
ct_state(+est+trk),recirc_id(0x2b),in_port(enp3s0f0),eth_type(0x0800),ipv4(dst=128.0.0.0/128.0.0.0,frag=no), packets:123314, bytes:6756523, used:0.000s, actions:ct(zone=16,nat),recirc(0x2c) 
ct_state(+est+rpl+trk),ct_label(0/0x1),recirc_id(0x2c),in_port(enp3s0f0),eth(src=24:8a:07:8d:ae:3f,dst=b8:ce:f6:09:fa:38),eth_type(0x0800),ipv4(src=10.11.0.0/255.255.255.248,dst=192.168.0.3,proto=6,ttl=64,frag=no), packets:123314, bytes:6756523, used:0.000s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:c0:a8:00:01,dst=0a:58:c0:a8:00:03)),set(ipv4(ttl=62)),ct(zone=22,nat),recirc(0x2d)                                                                                                                                                                                
ct_state(+est+rpl+trk),ct_label(0/0x1),recirc_id(0x2d),in_port(enp3s0f0),eth(dst=0a:58:c0:a8:00:03),eth_type(0x0800),ipv4(dst=192.168.0.3,frag=no), packets:123313, bytes:6756470, used:0.000s, actions:aa2875c8e472cbc                                                                                                                                                                     
ct_state(+est-rpl+trk),ct_label(0/0x1),recirc_id(0),in_port(brenp3s0f0),eth(src=b8:ce:f6:09:fa:38,dst=24:8a:07:8d:ae:3e),eth_type(0x0800),ipv4(dst=10.0.0.0/255.192.0.0,frag=no), packets:748, bytes:100384, used:0.950s, actions:enp3s0f0                                                                                                                                                  
ct_state(-new-est-rpl-trk),ct_label(0/0x1),recirc_id(0),in_port(brenp3s0f0),eth(src=b8:ce:f6:09:fa:38,dst=24:8a:07:8d:ae:3e),eth_type(0x0800),ipv4(dst=10.0.0.0/255.192.0.0,frag=no), packets:18, bytes:3238, used:0.580s, actions:enp3s0f0                                                                                                                                                 
ct_state(-new-est-trk),recirc_id(0),in_port(aa2875c8e472cbc),eth(src=0a:58:c0:a8:00:03,dst=0a:58:c0:a8:00:01),eth_type(0x0800),ipv4(src=192.168.0.3,dst=10.11.0.1,proto=6,frag=no),tcp(dst=5201), packets:4262300, bytes:6026886820, used:0.380s, actions:ct(zone=22,nat),recirc(0x2e)                                                                                                      
ct_state(+est-rpl+trk),ct_label(0/0x3),recirc_id(0x2e),in_port(aa2875c8e472cbc),eth(src=0a:58:c0:a8:00:03,dst=0a:58:c0:a8:00:01),eth_type(0x0800),ipv4(src=192.168.0.3,dst=10.11.0.0/255.255.255.248,proto=6,ttl=64,frag=no), packets:4262295, bytes:6026879750, used:0.381s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:02)),set(ipv4(ttl=63)),ct(zone=16,nat),recirc(0x2f)                                                                                                                                                                     
ct_state(+est-rpl+trk),ct_label(0/0x1),recirc_id(0x2f),in_port(aa2875c8e472cbc),eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:02),eth_type(0x0800),ipv4(src=0.0.0.0/128.0.0.0,dst=10.11.0.1,proto=6,ttl=63,frag=no), packets:4262293, bytes:6026876922, used:0.380s, actions:ct_clear,set(eth(src=b8:ce:f6:09:fa:38,dst=24:8a:07:8d:ae:3f)),set(ipv4(ttl=62)),ct(commit,zone=64000),enp3s0f0 
recirc_id(0),in_port(enp3s0f0),eth(src=00:02:c9:7b:7c:25,dst=01:80:c2:00:00:00),eth_type(0/0xffff), packets:412, bytes:24720, used:0.809s, actions:drop                                       
recirc_id(0x2c),in_port(enp3s0f0),ct_state(-new-est-rel-rpl+trk),ct_label(0/0x1),eth(),eth_type(0x86dd),ipv6(proto=58,hlimit=255,frag=no),icmpv6(type=133,code=0), packets:0, bytes:0, used:never, actions:drop





OVN Kubernetes uses host CT zone for traffic flowing through the Gateway router.
(i.e options:snat-ct-zone=0 is set for every OVN gateway router in OVN-Kubernetes)

Changing this to a different value nor removing NB_GLOBAL option use_ct_inv_match changes the outcome.

Comment 1 Marcelo Ricardo Leitner 2021-04-29 13:21:57 UTC
(In reply to Alaa Hleihel (NVIDIA Mellanox) from comment #0)
> Datapath rules: (problematic flows in bold)

The marking (bold) didn't get through. Can you please highlight them in some other way? Thanks.

Comment 2 Alaa Hleihel (NVIDIA Mellanox) 2021-04-29 14:45:23 UTC
(In reply to Marcelo Ricardo Leitner from comment #1)
> (In reply to Alaa Hleihel (NVIDIA Mellanox) from comment #0)
> > Datapath rules: (problematic flows in bold)
> 
> The marking (bold) didn't get through. Can you please highlight them in some
> other way? Thanks.

Sorry about that :)

This is the text that was in bold:

ct_state(+est+rpl+trk),ct_label(0/0x1),recirc_id(0x3),in_port(enp3s0f0),eth(src=24:8a:07:8d:ae:3f,dst=b8:ce:f6:09:fa:38),eth_type(0x0800),ipv4(src=10.11.0.0/255.255.255.248,dst=10.11.0.11,proto=6,ttl=64,frag=no), packets:123315, bytes:6756575, used:0.000s, actions:ct_clear,ct(zone=16),recirc(0x2a)                                                                                  
recirc_id(0x2a),in_port(enp3s0f0),eth_type(0x0800),ipv4(dst=10.11.0.11,proto=6,frag=no),tcp(dst=32768/0x8000), packets:123315, bytes:6756575, used:0.000s, actions:ct(nat),recirc(0x2b)       
ct_state(+est+trk),recirc_id(0x2b),in_port(enp3s0f0),eth_type(0x0800),ipv4(dst=128.0.0.0/128.0.0.0,frag=no), packets:123314, bytes:6756523, used:0.000s, actions:ct(zone=16,nat),recirc(0x2c)

Comment 3 Numan Siddique 2021-04-30 12:05:49 UTC
Hi Alaa,

I am not clear on your last statement - "Changing this to a different value nor removing NB_GLOBAL option use_ct_inv_match changes the outcome.".

Can you please provide more details ? Do you mean setting options:snat-ct-zone to a different value has no effect ? 
And removing the option use_ct_inv_match has no effect ? and you still these 2 extra datapath flows ?

 
Thanks

Comment 4 Numan Siddique 2021-04-30 12:08:37 UTC
Also is it possible for you to attach the OVN Northbound db and south bound db ?

Comment 5 Numan Siddique 2021-04-30 13:09:15 UTC
Hi Alaa,

Request to also attach the local ovsdb - conf.db on the host where the pod runs ?

If your setup has changed/destroyed, then can you please reproduce the issue again (sorry for the trouble)
and share the datapath flows, ovn nb db, ovn sb db
and the node conf.db.  This will help me in mapping the zone ids to the ovn resources.

Can you also please share the source ip of the traffic and destination ip of the traffic and the name of the pod ?

Thanks
Numan

Comment 6 Numan Siddique 2021-04-30 13:09:15 UTC
Hi Alaa,

Request to also attach the local ovsdb - conf.db on the host where the pod runs ?

If your setup has changed/destroyed, then can you please reproduce the issue again (sorry for the trouble)
and share the datapath flows, ovn nb db, ovn sb db
and the node conf.db.  This will help me in mapping the zone ids to the ovn resources.

Can you also please share the source ip of the traffic and destination ip of the traffic and the name of the pod ?

Thanks
Numan

Comment 7 Adrian Chiris 2021-05-02 06:52:20 UTC
Hi Numan,

> Can you please provide more details ? Do you mean setting options:snat-ct-zone to a different value has no effect ? 
And removing the option use_ct_inv_match has no effect ? and you still these 2 extra datapath flows ?

Yes, exactly.

Comment 8 Adrian Chiris 2021-05-02 06:54:34 UTC
Created attachment 1778426 [details]
local conf.db

Comment 9 Adrian Chiris 2021-05-02 06:55:10 UTC
Created attachment 1778427 [details]
ovn nb db

Comment 10 Adrian Chiris 2021-05-02 06:55:37 UTC
Created attachment 1778428 [details]
ovn sb db

Comment 11 Adrian Chiris 2021-05-02 07:04:26 UTC
Additional details on the setup:

Pod IP : 192.168.0.3
Pod MAC: 0a:58:c0:a8:00:03

Host where pod runs IP:10.11.0.11
Host where pod runs MAC: b8:ce:f6:09:fa:38

External IP:10.11.0.1
External MAC : 24:8a:07:8d:ae:3f

Comment 12 Numan Siddique 2021-05-03 19:47:44 UTC
Thanks a lot for providing the information.

Numan

Comment 13 Alaa Hleihel (NVIDIA Mellanox) 2021-05-04 09:48:13 UTC
Hi,

I am adding more info from Ariel.
There are two issues here.

Alaa
```

I wanted to provide some additional info on this bug following some investigation we did internally.

There are actually 2 issues behind this ticket the causing traffic not to be offloaded:
1. We still see combination of CT/CT(nat) in the datapath in the pod 2 external traffic.
2. We spotted some asymmetric paths for original and reply traffic causing CT actions not to be offloaded.

Below is the full analysis of the datapath (running iperf test between a pod and an external host). Please review and let us know what you think. This is quite urgent:


//------------------- ORIG DIRECTION ----------------------//


recirc_id(0),ct_state(-new-est-trk),in_port(cdb440f8c522946),eth(src=0a:58:c0:a8:00:08,dst=0a:58:c0:a8:00:01),eth_type(0x0800),ipv4(src=192.168.0.8,dst=10.11.0.1,proto=6,frag=no),tcp(dst=5201), packets:6625425, bytes:9368334929, used:0.630s, actions:ct(zone=20,nat),recirc(0x4)                                                                                                                                                                                                     

#############
CT ZONE 20
#############
recirc_id(0x4),ct_state(+new-est-rel-rpl+trk),ct_label(0/0x1),in_port(cdb440f8c522946),eth(src=0a:58:c0:a8:00:08,dst=0a:58:c0:a8:00:01),eth_type(0x0800),ipv4(src=192.168.0.8/255.255.255.248,dst=10.11.0.0/255.255.255.248,proto=6,ttl=64,frag=no), packets:1, bytes:60, used:20.090s, actions:ct(commit,zone=20,label=0/0x1),ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:03)),set(ipv4(ttl=63)),ct(zone=16,nat),recirc(0x5)
=====> Missing commit to CT(zone=16) ?

recirc_id(0x4),ct_state(+est-rel-rpl+trk),ct_label(0/0x3),in_port(cdb440f8c522946),eth(src=0a:58:c0:a8:00:08,dst=0a:58:c0:a8:00:01),eth_type(0x0800),ipv4(src=192.168.0.8/255.255.255.248,dst=10.11.0.0/255.255.255.248,proto=6,ttl=64,frag=no), packets:6625422, bytes:9368334714, used:0.630s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:03)),set(ipv4(ttl=63)),ct(zone=16,nat),recirc(0x5)                                                   

#############
CT ZONE 16
#############
recirc_id(0x5),ct_state(+new-est-rel-rpl+trk),ct_label(0/0x1),in_port(cdb440f8c522946),eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:03),eth_type(0x0800),ipv4(src=192.168.0.0/255.255.0.0,dst=10.11.0.1,proto=6,ttl=63,frag=no), packets:2, bytes:149, used:20.090s, actions:set(eth(src=b8:ce:f6:09:fa:38,dst=86:e9:f5:48:39:cc)),set(ipv4(ttl=62)),ct(commit,nat(src=10.11.0.11)),recirc(0x6)  
=====> `nat(src=10.11.0.11)` --> Going via CT(zone=0) only when state from zone=16 is new                                                                                  

recirc_id(0x5),ct_state(+est-rel-rpl+trk),ct_label(0/0x1),in_port(cdb440f8c522946),eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:03),eth_type(0x0800),ipv4(src=0.0.0.0/128.0.0.0,dst=10.11.0.1,proto=6,ttl=63,frag=no), packets:6637459, bytes:9385111725, used:0.630s, actions:ct_clear,set(eth(src=b8:ce:f6:09:fa:38,dst=86:e9:f5:48:39:cc)),set(ipv4(ttl=62)),ct(commit,zone=64000),enp3s0f0  
=====> Skips NAT at zone=0 if +est in previous zone=16                                                                              

#######################################
CT ZONE 0 (HAS ORIG SNAT CONFIGURED)
           ^^^^^^^^^^^^^^^^^^^^^^^^
#######################################
recirc_id(0x6),ct_state(+new-est-rel-rpl+trk),ct_label(0/0x1),in_port(cdb440f8c522946),eth(src=b8:ce:f6:09:fa:38,dst=86:e9:f5:48:39:cc),eth_type(0x0800),ipv4(dst=10.0.0.0/255.192.0.0,frag=no), packets:2, bytes:149, used:20.090s, actions:ct_clear,ct(commit,zone=64000),enp3s0f0        

recirc_id(0x6),ct_state(+est-rel-rpl+trk),ct_label(0/0x1),in_port(cdb440f8c522946),eth(src=b8:ce:f6:09:fa:38,dst=86:e9:f5:48:39:cc),eth_type(0x0800),ipv4(dst=10.0.0.0/255.192.0.0,frag=no), packets:0, bytes:0, used:20.540s, actions:ct_clear,ct(commit,zone=64000),enp3s0f0                                                                                                                                                                                                            



//------------------- REPLY DIRECTION ----------------------//

recirc_id(0),in_port(enp3s0f0),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(), packets:26, bytes:1427, used:0.080s, actions:ct(zone=64000),recirc(0x2)

#############
CT ZONE 64000
#############
recirc_id(0x2),ct_state(+est-rel+rpl+trk),ct_label(0/0x1),in_port(enp3s0f0),eth(src=86:e9:f5:48:39:cc,dst=b8:ce:f6:09:fa:38),eth_type(0x0800),ipv4(src=10.11.0.0/255.255.255.248,dst=10.11.0.11,proto=6,ttl=64,frag=no), packets:104040, bytes:7287124, used:0.620s, actions:ct_clear,ct(zone=16),recirc(0x7)  
=====> Here we have the ct/ct(nat). Zone=16 was define with nat but rule does ct(zone=16) without nat.

#############
CT ZONE 16   
#############       
recirc_id(0x7),in_port(enp3s0f0),eth_type(0x0800),ipv4(dst=10.11.0.11,proto=6,frag=no),tcp(dst=32768/0x8000), packets:107493, bytes:5594242, used:0.000s, actions:ct(nat),recirc(0x8) 
=====> Always going via zone=0 even though original direction doesn't


#######################################
CT ZONE 0 (SHOULD RESTORE REPLY DNAT)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
#######################################
recirc_id(0x8),ct_state(+est+trk),in_port(enp3s0f0),eth_type(0x0800),ipv4(dst=128.0.0.0/128.0.0.0,frag=no), packets:107489, bytes:5594032, used:0.000s, actions:ct(zone=16,nat),recirc(0x3)

recirc_id(0x8),ct_state(+new-est+trk),in_port(enp3s0f0),eth_type(0x0800),ipv4(dst=10.11.0.11,proto=6,frag=no),tcp(dst=32768/0x8000), packets:1, bytes:53, used:20.310s, actions:ct(zone=16,nat),recirc(0x3)


########################################################################################################
CT ZONE 16 (** Creates a "wrong" new tuple for SYN_RECV because we get here after zone 0 nat restore...)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
########################################################################################################

recirc_id(0x3),ct_state(+est-rel+rpl+trk),ct_label(0/0x1),in_port(enp3s0f0),eth(src=86:e9:f5:48:39:cc,dst=b8:ce:f6:09:fa:38),eth_type(0x0800),ipv4(src=10.11.0.0/255.255.255.248,dst=192.168.0.8,proto=6,ttl=64,frag=no), packets:119345, bytes:22358404, used:0.000s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:c0:a8:00:01,dst=0a:58:c0:a8:00:08)),set(ipv4(ttl=62)),ct(zone=20,nat),recirc(0x9)                                                                             
recirc_id(0x3),ct_state(+new-est-rel-rpl+trk),ct_label(0/0x1),in_port(enp3s0f0),eth(src=86:e9:f5:48:39:cc,dst=b8:ce:00:00:00:00/ff:ff:00:00:00:00),eth_type(0x0800),ipv4(dst=10.11.0.11,ttl=64,frag=no), packets:1, bytes:53, used:20.310s, actions:drop                                                                                                                                                                                                                                  
recirc_id(0x3),ct_state(-new-est-rel-rpl+trk),ct_label(0/0x1),in_port(enp3s0f0),eth(src=86:e9:f5:48:39:cc,dst=b8:ce:f6:09:fa:38),eth_type(0x0800),ipv4(src=10.11.0.0/255.255.255.248,dst=192.168.0.8,proto=6,ttl=64,frag=no), packets:0, bytes:0, used:20.540s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:c0:a8:00:01,dst=0a:58:c0:a8:00:08)),set(ipv4(ttl=62)),ct(zone=20,nat),recirc(0x9)                                                                                    

#############
CT ZONE 20
#############
recirc_id(0x9),ct_state(+est-rel+rpl+trk),ct_label(0/0x1),in_port(enp3s0f0),eth(dst=0a:58:c0:a8:00:08),eth_type(0x0800),ipv4(dst=192.168.0.8,frag=no), packets:107489, bytes:5594020, used:0.000s, actions:cdb440f8c522946



Below is analysis of the conntrack stages of the connection - 

root@l-cloudx6-03 ~ $ conntrack -E | grep -i 5201

####################################################
FIRST IPERF3 CONNECTION, PORT 33132(Management?)
####################################################

******  ORIG SYN UPDATES:
    [NEW] tcp      6 120 SYN_SENT src=192.168.0.8 dst=10.11.0.1 sport=33132 dport=5201 [UNREPLIED] src=10.11.0.1 dst=192.168.0.8 sport=5201 dport=33132 zone=20 
=====> **** HERE Missing src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33132 ....  src=192.168.0.8 dst=10.11.0.1 sport=33132 dport=5201  zone=16
    [NEW] tcp      6 120 SYN_SENT src=192.168.0.8 dst=10.11.0.1 sport=33132 dport=5201 [UNREPLIED] src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33132                        
=========> SNAT
    [NEW] tcp      6 120 SYN_SENT src=10.11.0.11 dst=10.11.0.1 sport=33132 dport=5201 [UNREPLIED] src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33132 zone=64000   
=========> TUPLE AFTER NAT

******  Reply SYN_ACK updates: 
    [NEW] tcp      6 300 src=192.168.0.8 dst=10.11.0.1 sport=33132 dport=5201 [UNREPLIED] src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33132 zone=16                          -  
=========> WRONG NEW REPLY TUPLE , should be reversed
   [UPDATE] tcp      6 300 src=192.168.0.8 dst=10.11.0.1 sport=33132 dport=5201 src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33132 zone=16
=========> -  What is update with no tcp state


####################################################
SECOND IPERF3 CONNECTION, PORT 33134 (Traffic?) 
####################################################

    [NEW] tcp      6 120 src=192.168.0.8 dst=10.11.0.1 sport=33134 dport=5201 [UNREPLIED] src=10.11.0.1 dst=192.168.0.8 sport=5201 dport=33134 zone=20
    [NEW] tcp      6 120 src=192.168.0.8 dst=10.11.0.1 sport=33134 dport=5201 [UNREPLIED] src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33134 zone=16
    [NEW] tcp      6 120 src=10.11.0.11 dst=10.11.0.1 sport=33134 dport=5201 [UNREPLIED] src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33134 zone=64000
 [UPDATE] tcp      6 60 SYN_RECV src=10.11.0.11 dst=10.11.0.1 sport=33134 dport=5201 src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33134 zone=64000
 [UPDATE] tcp      6 60 SYN_RECV src=192.168.0.8 dst=10.11.0.1 sport=33134 dport=5201 src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33134 zone=16
 [UPDATE] tcp      6 60 SYN_RECV src=192.168.0.8 dst=10.11.0.1 sport=33134 dport=5201 src=10.11.0.1 dst=192.168.0.8 sport=5201 dport=33134 zone=20
 [UPDATE] tcp      6 432000 ESTABLISHED src=192.168.0.8 dst=10.11.0.1 sport=33134 dport=5201 src=10.11.0.1 dst=192.168.0.8 sport=5201 dport=33134 [ASSURED] zone=20
 [UPDATE] tcp      6 432000 ESTABLISHED src=192.168.0.8 dst=10.11.0.1 sport=33134 dport=5201 src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33134 [ASSURED] zone=16
 [UPDATE] tcp      6 432000 ESTABLISHED src=10.11.0.11 dst=10.11.0.1 sport=33134 dport=5201 src=10.11.0.1 dst=10.11.0.11 sport=5201 dport=33134 [ASSURED] zone=64000

====> Missing zone 0 for second tuple?

Comment 14 Alaa Hleihel (NVIDIA Mellanox) 2021-05-04 10:41:49 UTC
(In reply to Alaa Hleihel (NVIDIA Mellanox) from comment #13)
> 
> There are actually 2 issues behind this ticket the causing traffic not to be
> offloaded:
> 1. We still see combination of CT/CT(nat) in the datapath in the pod 2
> external traffic.

Let's focus in this BZ on this issue.

> 2. We spotted some asymmetric paths for original and reply traffic causing
> CT actions not to be offloaded.
> 

And this one under BZ # 1956740

Comment 15 Numan Siddique 2021-05-11 12:06:13 UTC
We are seeing these extra ct(zone=16) and ct(zone=16, nat) because in the logical router pipeline stage - lr_in_ip_defrag we first send the pkt to conntrack
and then in lr_in_dnat we do ct(zone=x, nat) for established traffic.

Just like the way we optimized here [1] for logical switch pipeline, we need to do the same for router pipeline.

[1] - https://github.com/ovn-org/ovn/commit/0038579d192802fff03c3594e4f85dab4f7af2bd

I'll update the BZ when I submit the patch for review.

Thanks
Numan

Comment 16 Dumitru Ceara 2021-06-01 07:21:30 UTC
Hi Alaa,

Is there any specific reason why this BZ is opened as "private"?  Would
it be ok to make it public?  There are some external requests (e.g.,
from Girish @ Nvidia) to access the BZ contents.

Thanks,
Dumitru

Comment 17 Alaa Hleihel (NVIDIA Mellanox) 2021-06-01 07:49:04 UTC
Hi, Dumitru.
Can you make your comment public?

Comment 18 Dumitru Ceara 2021-06-01 07:56:53 UTC
(In reply to Alaa Hleihel (NVIDIA Mellanox) from comment #17)
> Hi, Dumitru.
> Can you make your comment public?

Oops, done :)

Comment 19 Alaa Hleihel (NVIDIA Mellanox) 2021-06-01 08:04:02 UTC
(In reply to Dumitru Ceara from comment #16)
> Hi Alaa,
> 
> Is there any specific reason why this BZ is opened as "private"?  Would
> it be ok to make it public?  There are some external requests (e.g.,
> from Girish @ Nvidia) to access the BZ contents.
> 
> Thanks,
> Dumitru

Not really, feel free to CC him or make the BZ public.

Regards,
Alaa

Comment 20 Dumitru Ceara 2021-06-01 08:08:27 UTC
(In reply to Alaa Hleihel (NVIDIA Mellanox) from comment #19)
[...]
> 
> Not really, feel free to CC him or make the BZ public.
> 
> Regards,
> Alaa

Done, thanks!

Comment 21 Dumitru Ceara 2021-06-02 07:35:56 UTC
Hi Alaa,

Seems that I need to clear the "mellanox" group on this BZ to really
make it public but for some reason I don't have the access rights to
do that.  Are you able to clear it?

Thanks,
Dumitru

Comment 22 Alaa Hleihel (NVIDIA Mellanox) 2021-06-02 07:39:01 UTC
(In reply to Dumitru Ceara from comment #21)
> Hi Alaa,
> 
> Seems that I need to clear the "mellanox" group on this BZ to really
> make it public but for some reason I don't have the access rights to
> do that.  Are you able to clear it?
> 
> Thanks,
> Dumitru

Yes, I'm removing it.

Regards,
Alaa

Comment 25 Numan Siddique 2021-07-16 15:13:59 UTC
Patches merged upstream and will be available d/s ovn21.09

Comment 26 zenghui.shi 2021-08-09 02:40:01 UTC
*** Bug 1988189 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.