Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2043187

Summary: Flow not offloaded to HW while testing Stateful ACL use case
Product: Red Hat Enterprise Linux Fast Datapath Reporter: arn
Component: OVNAssignee: Numan Siddique <nusiddiq>
Status: CLOSED CURRENTRELEASE QA Contact: Jianlin Shi <jishi>
Severity: high Docs Contact:
Priority: high    
Version: RHEL 8.0CC: arn, ctrautma, hakhande, jiji, jishi, lariel, mleitner, mmichels, nusiddiq, vchundur
Target Milestone: ---Keywords: TestOnly
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-11 18:12:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2028504    
Bug Blocks: 1984953, 2024599, 2131355    
Attachments:
Description Flags
OVN NB DB none

Description arn 2022-01-20 19:13:17 UTC 
Created attachment 1852301 [details]
OVN NB DB

Description of problem:

While testing the Sateful NAT with ACL use case all the Flows are not offloaded to hardware and hence we can see traffic on representator port. Without the ACL configs it works fine though. Both working and non working logs are shared below.



Network topology used for the test
compute2(Remote host) ------------------------------------------compute1(local host - DUT - Has mellanox nic)[ VM is created on this ]

Remote host IP and MAC -- 172.16.0.111/24 and  e4:43:4b:4d:f1:12    ---> iperf3 client running on this 
-
-
-
Local hostIp has VM and IP and MAC is 192.168.0.2 and 6a:62:68:b2:61:f6  ---> iperf3 server is running here.

We have a playbook which we have used to test the use case.
https://github.com/rh-nfv-int/hw-offload-playbooks/blob/main/testNATwithACLoffload.yml

Flow Dumps collected in OVS and connection tracking table dump logs are as below.
[root@rhos-nfv-09 ~]# cat dump_ct_table_20220113081253.txt
ipv4     2 tcp      6 src=172.16.0.111 dst=192.168.0.2 sport=6000 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=6000 [HW_OFFLOAD] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=4 use=3
ipv4     2 tcp      6 src=172.16.0.111 dst=172.16.0.7 sport=6000 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=6000 [HW_OFFLOAD] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=7 use=25784744

[root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m
ufid:38f5e66c-6364-47e3-950f-89f4b2aa2e8a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:12,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0)), packets:17813382, bytes:26969445560, used:0.890s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=7,nat),recirc(0x7)

ufid:a22c8e1f-c22c-47dd-9a95-5cd6e5b06bd9, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x7),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:12,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:18533989, bytes:26860935601, used:0.000s, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),ct(zone=4),recirc(0x9)

ufid:9302e0f5-f4b0-4590-b04e-df97d8b0ee87, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x9),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:ff:01,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=5201), packets:18533989, bytes:26860935601, used:0.000s, offloaded:yes, dp:tc, actions:enp4s0f0_0

ufid:6c16d779-4ca5-4fc5-8e42-4df97d0f8829, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0), packets:316476, bytes:20888874, used:0.890s, offloaded:yes, dp:tc, actions:ct(zone=4),recirc(0xa)

ufid:05f9a996-54e8-4c4e-bf45-3e6f825f0d24, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0xa),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:316473, bytes:20888689, used:0.890s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:12)),set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0xe)

ufid:491a7601-9eb0-4bad-ab5d-922a22dd43c9, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0xe),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:12),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:316472, bytes:22154503, used:0.890s, offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0

Without the ACL configs the working log is as below...
ipv4     2 tcp      6 src=172.16.0.111 dst=172.16.0.7 sport=6000 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=6000 [HW_OFFLOAD] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=6 use=3

[root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m
ufid:d3c2c365-1703-4286-8011-b557bf08a68d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:12,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0)), packets:23, bytes:21827, used:2.010s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=6,nat),recirc(0xc)

ufid:7bb1bab2-6098-4967-9854-ffc2425fb12c, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xc),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:12,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:22, bytes:21767, used:2.010s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0

ufid:3cc6a4db-9341-42de-903f-c05f507722bd, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xc),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=172.16.0.7,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:0, bytes:0, used:4.130s, offloaded:yes, dp:tc, actions:ct(commit,zone=6,nat(dst=192.168.0.2)),recirc(0xe)

ufid:12ef7d35-3ea0-45e6-acfd-8c95f736560b, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xe),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:12,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:0, bytes:0, used:4.130s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0

ufid:149d8ad0-b08c-4670-8c77-9fe7457514d4, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:7, bytes:452, used:2.010s, offloaded:yes, dp:tc, actions:set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:12)),set(ipv4(ttl=63)),ct(zone=6,nat),recirc(0xd)

ufid:40aa8509-4c6b-4418-b6af-b8212bc01988, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xd),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:12),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:7, bytes:472, used:2.010s, offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0


Version-Release number of selected component (if applicable):


How reproducible:
Ansible playbook which we have used can be used to reproduce the issue.
https://github.com/rh-nfv-int/hw-offload-playbooks/blob/main/testNATwithACLoffload.yml

While if we run the same use case without ACLs it works.
https://github.com/rh-nfv-int/hw-offload-playbooks/blob/main/testNAToffload.yml

Actual results:
All flows are not offloaded to hardware

Expected results:
All flows needs to be offloaded to hardware and traffic shouldn't not be observed on representator port.

Additional info:

[root@rhos-nfv-09 ~]#
[root@rhos-nfv-09 ~]# ovn-nb show
-bash: ovn-nb: command not found
[root@rhos-nfv-09 ~]# ovn-nbctl show
switch c8c9be48-e207-43e4-9968-8d01d129f401 (sw0)
    port sw0-port1
        addresses: ["6a:62:68:b2:61:f6 dynamic"]
    port lrp0-attachment
        type: router
        router-port: lrp0
switch 0f78b94c-80e7-4ce9-9a33-f4541e29b72d (public)
    port ln-public
        type: localnet
        tag: 201
        addresses: ["unknown"]
    port public-lr0
        type: router
        router-port: lr0-public
router cec9c233-f8ac-42ec-9e01-c8eda5e687be (lr0)
    port lrp0
        mac: "00:00:00:00:ff:01"
        networks: ["192.168.0.1/24"]
    port lr0-public
        mac: "0a:00:20:20:12:13"
        networks: ["172.16.0.1/24"]
        gateway chassis: [dummy]
    nat 277dddac-4e9a-46c8-9ed8-719a421f6f39
        external ip: "172.16.0.7"
        logical ip: "192.168.0.2"
        type: "dnat_and_snat"
    nat ad4534f3-62dc-4e45-ab61-89142abe750c
        external ip: "172.16.0.2"
        logical ip: "192.168.0.0/24"
        type: "snat"
[root@rhos-nfv-09 ~]#
[root@rhos-nfv-09 ~]#
[root@rhos-nfv-09 ~]# ovn-sbctl show
Chassis ""
    hostname: rhos-nfv-09.lab.eng.rdu2.redhat.com
    Encap geneve
        ip: "10.8.2.160"
        options: {csum="true"}
    Port_Binding sw0-port1
[root@rhos-nfv-09 ~]#
[root@rhos-nfv-09 ~]# ovs-vsctl show
800e5315-5ef8-40c3-bd48-a35f90979838
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port enp4s0f0_0
            Interface enp4s0f0_0
        Port patch-br-int-to-ln-public
            Interface patch-br-int-to-ln-public
                type: patch
                options: {peer=patch-ln-public-to-br-int}
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port enp4s0f0
            Interface enp4s0f0
        Port patch-ln-public-to-br-int
            Interface patch-ln-public-to-br-int
                type: patch
                options: {peer=patch-br-int-to-ln-public}
    ovs_version: "2.16.90"
[root@rhos-nfv-09 ~]#
Comment 1 arn 2022-01-20 19:14:34 UTC 
Created attachment 1852302 [details]
OVN SB DB
Comment 2 Haresh Khandelwal 2022-02-21 13:48:55 UTC 
we are seeing this issue in OSP 16.2.2 with below versions.

kernel: 4.18.0-305.34.2.el8_4.x86_64
ovs: openvswitch2.15-2.15.0-55.el8fdp.x86_64
ovn: ovn-2021-21.12.0-11.el8fdp.x86_64

Raising its priority to get before 16.2.3 (when osp qe cycle kicks in)

Thanks
Comment 3 Numan Siddique 2022-03-04 22:59:21 UTC 
I tested the same scenario in another machine which has ConnectX-5 Ex and I don't see the issue.

All the flows are offloading and tcpdump on the representor port doesn't show any packets.


eth0 is the representor (VM) port.

# ovs-appctl dpctl/dump-flows -m | grep "in_port(eth"
ufid:c2c4cf4e-14b1-4d71-85f0-7a788c088845, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(eth0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0), packets:10, bytes:646, used:3.160s, offloaded:yes, dp:tc, actions:ct(zone=3),recirc(0x4d)
ufid:43b2eeb1-c474-42bf-bf49-5a078cd1e125, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x4d),dp_hash(0/0),in_port(eth0),packet_type(ns=0/0,id=0/0),eth(src=06:5f:0e:8a:9b:6a,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.105,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:10, bytes:646, used:3.160s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=0a:00:20:20:12:14,dst=0c:42:a1:2a:81:c0)),set(ipv4(ttl=63)),ct(zone=1,nat),recirc(0x4e)
ufid:1d431860-9392-4a0e-9ce5-bd1461f158f6, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x4e),dp_hash(0/0),in_port(eth0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:14,dst=0c:42:a1:2a:81:c0),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:11, bytes:730, used:3.160s, offloaded:yes, dp:tc, actions:ct_clear,ens1f0np0


[root@wsfd-advnetlab085 ovn]# ovs-appctl dpctl/dump-flows -m | grep "in_port(ens"
ufid:251c9e4b-9a02-4b4a-b2fe-11250ca81d53, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens1f0np0),packet_type(ns=0/0,id=0/0),eth(src=0c:42:a1:2a:81:c0,dst=0a:00:20:20:12:14),eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:6, bytes:394, used:1.210s, offloaded:yes, dp:tc, actions:ct(zone=1,nat),recirc(0x51)
ufid:186aa595-0705-493a-878c-8ae51ceb5ee7, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x51),dp_hash(0/0),in_port(ens1f0np0),packet_type(ns=0/0,id=0/0),eth(src=0c:42:a1:2a:81:c0,dst=0a:00:20:20:12:14),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:6, bytes:394, used:1.210s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=06:5f:0e:8a:9b:6a)),set(ipv4(ttl=63)),ct(zone=3),recirc(0x52)
ufid:30469cb1-15e0-40e4-a239-e69d9d3220c4, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x52),dp_hash(0/0),in_port(ens1f0np0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:ff:01,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=6144/0xf800), packets:5, bytes:330, used:1.210s, offloaded:yes, dp:tc, actions:eth0

# uname -a
Linux wsfd-advnetlab085.lab3.eng.bos.redhat.com 4.18.0-305.10.2.el8_4.x86_64 #1 SMP Mon Jul 12 04:43:18 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

# lspci | grep Mella
3b:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
3b:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
3b:00.2 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
3b:00.3 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
3b:00.4 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
3b:00.5 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]

# ovs-vsctl show
ce2703f4-871d-47f7-b2c3-e4d93bf83173
    Bridge br-ex
        Port patch-ln-public-to-br-int
            Interface patch-ln-public-to-br-int
                type: patch
                options: {peer=patch-br-int-to-ln-public}
        Port br-ex
            Interface br-ex
                type: internal
        Port ens1f0np0
            Interface ens1f0np0
    Bridge br-int
        fail_mode: secure
        datapath_type: system
        Port br-int
            Interface br-int
                type: internal
        Port eth0
            Interface eth0
        Port patch-br-int-to-ln-public
            Interface patch-br-int-to-ln-public
                type: patch
                options: {peer=patch-ln-public-to-br-int}
    ovs_version: "2.16.1"



Looks to me its a driver/firmware issue and not an OVN (or OVS) issue.
Comment 4 Numan Siddique 2022-03-04 23:02:03 UTC 
I also tested with the kernel version 4.18.0-305.22.1.el8_4.x86_64 and the issue is not seen.


# uname -a
Linux wsfd-advnetlab085.lab3.eng.bos.redhat.com 4.18.0-305.22.1.el8_4.x86_64 #1 SMP Wed Sep 29 10:56:12 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
Comment 5 Marcelo Ricardo Leitner 2022-03-09 12:52:14 UTC 
(Ariel is on PTO on this week, btw)
Comment 6 Marcelo Ricardo Leitner 2022-03-10 13:59:26 UTC 
Hey folks, I confused this with the subport bz for a while.
Seems we are lacking some basic HWOL debugs here, like dmesg output and extack log msg for when the flow was rejected by the driver, like the flow from comment 0:

ufid:a22c8e1f-c22c-47dd-9a95-5cd6e5b06bd9, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x7),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:12,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:18533989, bytes:26860935601, used:0.000s, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),ct(zone=4),recirc(0x9)

Also, I don't see an indication on which NIC/fw it didn't work and the fw version for when it worked.
Comment 7 Ariel Levkovich 2022-03-12 00:41:55 UTC 
- U can try to add a similar rule manually with TC and get the error message in dmesg.

- With which device it worked running 4.18.0-305.22.1.el8_4.x86_64? CX5Ex or the same device where it failed with 4.18.0-305.34.2.el8_4.x86_64?

- Please provide FW/NIC details as Marcelo mentioned.
Comment 9 arn 2022-03-14 12:25:02 UTC 
(In reply to Marcelo Ricardo Leitner from comment #6)
> Hey folks, I confused this with the subport bz for a while.
> Seems we are lacking some basic HWOL debugs here, like dmesg output and
> extack log msg for when the flow was rejected by the driver, like the flow
> from comment 0:
> 
> ufid:a22c8e1f-c22c-47dd-9a95-5cd6e5b06bd9,
> skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/
> 0),ct_label(0/0x1),recirc_id(0x7),dp_hash(0/0),in_port(enp4s0f0),
> packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:12,dst=0a:00:20:20:12:13),
> eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,
> ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:18533989, bytes:26860935601,
> used:0.000s, dp:tc,
> actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),
> set(ipv4(ttl=63)),ct(zone=4),recirc(0x9)
> 
> Also, I don't see an indication on which NIC/fw it didn't work and the fw
> version for when it worked.

Hi Marcelo,

I have attached the logs of the OVS [please see the log.txt] . Initially I saw a log message saying 
2022-03-14T12:04:20.478Z|00001|tc(handler3)|WARN|Kernel flower acknowledgment does not match request!  Set dpif_netlink to dbg to see which rule caused this error.

So I have enabled the dpif_netlink DBG as well. 

Below is the error I got from dpif
2022-03-14T12:04:46.881Z|00006|dpif_netlink(handler6)|DBG|failed to offload flow: Operation not supported: enp4s0f0_0


From the extack logs I can see below ones
2022-03-14T12:04:37.952Z|00075|netlink_socket|ERR|received NAK error=2 - Filter with specified priority/protocol not found
2022-03-14T12:04:37.952Z|00076|netlink_socket|ERR|received NAK error=2 - Cannot find specified filter chain
2022-03-14T12:04:37.952Z|00077|netlink_socket|ERR|received NAK error=2 - Filter with specified priority/protocol not found
2022-03-14T12:04:37.952Z|00078|netlink_socket|ERR|received NAK error=2 - Cannot find specified filter chain

For full log pleas se attached log file [log.txt]

NIC and FW details where issue is observed is as below.
[root@rhos-nfv-09 ~]#  lspci | grep Mella | grep 04:00.0
04:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
[root@rhos-nfv-09 ~]# 

[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# ethtool -i enp4s0f0
driver: mlx5_core
version: 5.4-1.0.3
firmware-version: 16.31.1014 (DEL0000000015)
expansion-rom-version: 
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# uname -r
4.18.0-305.22.1.el8_4.x86_64
[root@rhos-nfv-09 ~]#

Thanks & Regards,
Abhiram R N
Comment 10 Ariel Levkovich 2022-03-18 19:48:53 UTC 
Is this with DMFS or SMFS?
Have we tried to manually add the rule via TC as was suggested during the meeting on Thursday?
Comment 11 Marcelo Ricardo Leitner 2022-03-18 20:02:15 UTC 
(In reply to lariel from comment #10)
> Is this with DMFS or SMFS?

OSP defaults to SMFS when supported by the driver, so I would suspect that. Abhiram, can you please confirm?

> Have we tried to manually add the rule via TC as was suggested during the
> meeting on Thursday?

Not yet. I got backlogged today and asked Abhiram for us to try it on Tue.
Comment 12 Ariel Levkovich 2022-03-18 20:34:00 UTC 
I just tried the following command:
"tc filter add dev enp8s0f0 ingress prio 1 chain 4 proto ip flower ip_flags nofrag ip_proto tcp ct_state +trk+est action ct clear action pedit ex munge eth src set 24:8a:07:9c:01:3f munge eth dst set 24:8a:07:9c:13:0f pipe action pedit ex munge ip ttl set 63 pipe action csum iph and tcp pipe action ct zone 3 pipe action goto chain 5"

Where enp8s0f0 is the uplink rep and it worked.

Device: CX6dx
FW: 22.32.2004
Kernel: 4.18.0-305.39.1.el8_4.x86_64 (I used the setup I worked on for the OCP 4.10 debug since I had it available already - not sure if it matters).

Ariel
Comment 13 arn 2022-03-22 14:33:38 UTC 
@
Comment 14 arn 2022-03-22 14:36:51 UTC 
(In reply to lariel from comment #12)
> I just tried the following command:
> "tc filter add dev enp8s0f0 ingress prio 1 chain 4 proto ip flower ip_flags
> nofrag ip_proto tcp ct_state +trk+est action ct clear action pedit ex munge
> eth src set 24:8a:07:9c:01:3f munge eth dst set 24:8a:07:9c:13:0f pipe
> action pedit ex munge ip ttl set 63 pipe action csum iph and tcp pipe action
> ct zone 3 pipe action goto chain 5"
> 
> Where enp8s0f0 is the uplink rep and it worked.
> 
> Device: CX6dx
> FW: 22.32.2004
> Kernel: 4.18.0-305.39.1.el8_4.x86_64 (I used the setup I worked on for the
> OCP 4.10 debug since I had it available already - not sure if it matters).
> 
> Ariel

Myself and Marcelo tried adding this tc filter and it was accepted and we didnt see any errors.

But like I mentioned earlier if we try using the OVN commands to set the ACLs there is something wrong.

@
Comment 15 arn 2022-03-22 14:41:54 UTC 
Hi Numan

Below are the specific commands I am using to set the ACLs.

            ovn-nbctl pg-add pg1 sw0-port1
            ovn-nbctl acl-add pg1 to-lport 1002 "outport == @pg1 && tcp && tcp.dst == 5201" allow-related


Below are the logs .. In the logs if you see below in the connection tracking table 2 are in HW_OFFLOAD and 2 are in ASSURED. And the ones which are in ASSURED are the ones with dport=5201 which from the ACL rule we have allowed actually. So, is it something to do with the OVS flows which is causing it not to offload?

Also if you can share the same connection tracking and OVS flow dumps from your setup maybe we can compare as well and see where is the mismatch.

[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# cat /proc/net/nf_conntrack | grep 172
ipv4     2 tcp      6 154 ESTABLISHED src=172.16.0.111 dst=192.168.0.2 sport=34954 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=34954 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=1 use=2
ipv4     2 tcp      6 src=172.16.0.111 dst=192.168.0.2 sport=6000 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=6000 [HW_OFFLOAD] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=1 use=3
ipv4     2 tcp      6 src=172.16.0.111 dst=172.16.0.7 sport=6000 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=6000 [HW_OFFLOAD] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=7 use=3
ipv4     2 tcp      6 431848 ESTABLISHED src=172.16.0.111 dst=172.16.0.7 sport=34954 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=34954 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=7 use=2
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m
ufid:9d1e9080-5648-4ff0-a0f0-cf7ca145c4d7, skb_priority(0/0),skb_mark(0/0),ct_state(0/0x21),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0)), packets:115080536, bytes:166165763902, used:0.000s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=6,nat),recirc(0x2f)
ufid:d85e3849-d9fe-4354-b9f0-f25408189f65, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0x2f),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=172.16.0.7,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:115080536, bytes:166165763902, used:0.000s, offloaded:yes, dp:tc, actions:ct(commit,zone=7,nat(dst=192.168.0.2)),recirc(0x30)
ufid:6558a688-9dcf-4e4a-99ae-f3a03455707a, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x30),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:115080581, bytes:166165828889, used:0.000s, dp:tc, actions:set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0x31)
ufid:ef1d93a1-f3d3-4659-a9d2-af3c72928b7c, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x31),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0), packets:115080581, bytes:166165828889, used:0.000s, offloaded:yes, dp:tc, actions:ct_clear,ct(zone=1),recirc(0x32)
ufid:3d6687ed-a862-4e9a-96b3-7aa62842d91c, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x32),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:ff:01,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=5201), packets:115080581, bytes:166165828889, used:0.000s, offloaded:yes, dp:tc, actions:enp4s0f0_0
ufid:50b9e2d7-5456-42d4-8072-4533df4bf3db, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0), packets:1691191, bytes:111621756, used:0.880s, offloaded:yes, dp:tc, actions:ct(zone=1),recirc(0x33)
ufid:d46e7050-a3e7-4aac-87b5-e0e9404e84ea, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x33),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:1691191, bytes:111621756, used:0.880s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)),set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0x34)
ufid:2922baf4-c2b5-444e-8434-5eb5f24b1ffe, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x34),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1691191, bytes:118386516, used:0.881s, offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 

Thanks & Regards,
Abhiram R N
Comment 16 Marcelo Ricardo Leitner 2022-03-22 15:07:38 UTC 
Two things. First, on the ovs flow that is not getting offloaded:

Checked dmesg with Abhiram now and it does log some errors, like:

[ 4920.725006] mlx5_core 0000:04:00.0: mlx5dr_actions_build_ste_arr:662:(pid 12340): Failed to handle checksum recalculation err -22
[ 4920.728625] mlx5_core 0000:04:00.0: dr_rule_create_rule:1287:(pid 12340): Failed creating rule
[ 4920.730493] mlx5_core 0000:04:00.0: mlx5_cmd_dr_create_fte:539:(pid 12340): Failed to create dr rule err(-22)
[ 4920.732909] mlx5_core 0000:04:00.0 enp4s0f0: Failed to offload ct flow, err -22

That's from:
        /* Due to a HW bug in some devices, modifying TTL on RX flows will
         * cause an incorrect checksum calculation. In this case we will
         * use a FW table to recalculate.
         */
        if (dmn->type == MLX5DR_DOMAIN_TYPE_FDB &&
            rx_rule && recalc_cs_required && dest_action) {
                ret = dr_action_handle_cs_recalc(dmn, dest_action, &attr.final_icm_addr);
                if (ret) {
                        mlx5dr_err(dmn,
                                   "Failed to handle checksum recalculation err %d\n",
                                   ret);
                        return ret;
                }
        }

Ariel, were you running with dmfs? Abhiram was with smfs.

2nd, we still don't know why 2 conntrack entries (the iperf3 control connection) in comment #15 were not offloaded.
Comment 17 Ariel Levkovich 2022-03-23 23:13:07 UTC 
I think the error above comes from SMFS trying to do the dec_ttl workaround for devices which have the csum bug (Such as CX5) but in order to set the wa properly it the DEC_TTL param needs to be set in mlxconfig - which I assume wasn't set in your case (Marcelo, I think you are familiar with this).


When u tried to add it manually did u make sure that the in_port is the uplink? This is to ensure the flow will be an rx flow which is where we have the limitation. A tx flow (in_port is vf rep) will not have a problem to offload such rule.

Ariel
Comment 18 arn 2022-03-31 11:55:29 UTC 
Hi Marcel/Ariel

I tried with dmfs mode today again. Its working fine in dmfs mode. So that seems like for smfs mode it needs to be resolved.

Before running the test set dmfs mode
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# devlink dev param set pci/0000:04:00.0 name flow_steering_mode value dmfs cmode runtime
[root@rhos-nfv-09 ~]# 

After running the test and traffic in progress.

[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m
ufid:d0e80564-c60e-4634-8063-d588aedfac1b, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=c8:fe:6a:f2:44:0b,dst=01:80:c2:00:00:0e),eth_type(0x88cc), packets:0, bytes:0, used:2.580s, offloaded:yes, dp:tc, actions:drop
ufid:7383c712-968b-496f-b4fe-cb6022c3fb8a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0)), packets:23963915, bytes:36278209339, used:0.540s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=6,nat),recirc(0x6)
ufid:525556c1-7c25-4341-9613-25557ef156e0, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x6),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:23963912, bytes:36278209138, used:0.540s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),ct(zone=4),recirc(0x8)
ufid:54a21929-a2a9-4ce0-994e-62c5e843aaff, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x8),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:ff:01,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=5201), packets:23963912, bytes:36278209138, used:0.540s, offloaded:yes, dp:tc, actions:enp4s0f0_0
ufid:a3f04835-3aa0-4558-bcbd-caddba6d19fb, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=0/0), packets:503324, bytes:33219617, used:0.540s, offloaded:yes, dp:tc, actions:ct(zone=4),recirc(0x9)
ufid:27fec6b6-6b4f-4474-b351-e36dc20c5113, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0x9),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=6,tos=0/0,ttl=64,frag=no),tcp(src=0/0,dst=0/0), packets:503322, bytes:33219498, used:0.540s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)),set(ipv4(ttl=63)),ct(zone=6,nat),recirc(0xd)
ufid:750d3641-4818-4591-8004-7ee484f4e0bc, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),recirc_id(0xd),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:503321, bytes:35232722, used:0.540s, offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# tcpdump -i enp4s0f0_0
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp4s0f0_0, link-type EN10MB (Ethernet), capture size 262144 bytes



07:26:10.719685 IP 172.16.0.111.38066 > 192.168.0.2.targus-getdata1: Flags [.], ack 4141882828, win 8, options [nop,nop,TS val 3476294748 ecr 1263419343], length 0
07:26:10.719757 IP 172.16.0.111.38066 > 192.168.0.2.targus-getdata1: Flags [.], ack 1, win 8, options [nop,nop,TS val 3476294748 ecr 1263419343,nop,nop,sack 1 {4294967293:4294967294}], length 0
07:26:38.498782 IP6 fe80::a57:6a48:9700:e266 > ff02::2: ICMP6, router solicitation, length 8


^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# cat /proc/net/nf_conntrack | grep 6000
ipv4     2 tcp      6 src=172.16.0.111 dst=192.168.0.2 sport=6000 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=6000 [HW_OFFLOAD] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=4 use=3
ipv4     2 tcp      6 src=172.16.0.111 dst=172.16.0.7 sport=6000 dport=5201 src=192.168.0.2 dst=172.16.0.111 sport=5201 dport=6000 [HW_OFFLOAD] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=6 use=3
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# ovn-nbctl show
switch fbe03f0c-70f9-4f80-8aee-e5f5d8bcbcad (public)
    port public-lr0
        type: router
        router-port: lr0-public
    port ln-public
        type: localnet
        tag: 201
        addresses: ["unknown"]
switch e5ae5f21-c159-4dbd-8e02-70fed320cd56 (sw0)
    port sw0-port2
        addresses: ["04:3f:72:d9:c0:30 10.10.10.2"]
    port lrp0-attachment
        type: router
        router-port: lrp0
    port sw0-port1
        addresses: ["6a:62:68:b2:61:f6 dynamic"]
router 49b35429-a597-4e5e-bf75-b867926fc8fc (lr0)
    port lr0-public
        mac: "0a:00:20:20:12:13"
        networks: ["172.16.0.1/24"]
        gateway chassis: [dummy]
    port lrp0
        mac: "00:00:00:00:ff:01"
        networks: ["192.168.0.1/24"]
    nat b708d646-946a-4988-97e6-fd7cb2e1c53b
        external ip: "172.16.0.7"
        logical ip: "192.168.0.2"
        type: "dnat_and_snat"
    nat c30d705e-5a96-4f82-a309-57fd5e645ada
        external ip: "172.16.0.2"
        logical ip: "192.168.0.0/24"
        type: "snat"
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# ovn-nbctl list acl
_uuid               : 4fbfe06e-5e69-4fff-b6f9-7099867a9707
action              : allow-related
direction           : to-lport
external_ids        : {}
label               : 0
log                 : false
match               : "outport == @pg1 && tcp && tcp.dst == 5201"
meter               : []
name                : []
options             : {}
priority            : 1002
severity            : []

_uuid               : 673b709b-d8c1-4973-8a5e-5cf253c43e05
action              : allow-related
direction           : to-lport
external_ids        : {}
label               : 0
log                 : false
match               : "outport == @pg1 && udp && udp.dst == 5201"
meter               : []
name                : []
options             : {}
priority            : 1002
severity            : []
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# 


Thanks & Regards,
Abhiram R N
Comment 19 Marcelo Ricardo Leitner 2022-03-31 12:17:57 UTC 
This bz then is to be fixed by bz2028504. As this one is on OVN component, lets mark it as TestOnly so it can be verified once the fix lands in 8.4.z. Thanks.
Comment 23 Marcelo Ricardo Leitner 2022-08-05 00:05:03 UTC 
How can we re-test and close this bz? The dependency is now closed.
Comment 24 arn 2022-08-09 05:22:41 UTC 
hi Marcelo,

We have a playbook for it.
https://github.com/rh-nfv-int/playbooks/blob/main/regression/ovs-hwol/testNATwithACLoffload.yml

Thanks & Regards,
Abhiram R N
Comment 25 Marcelo Ricardo Leitner 2022-10-04 18:13:19 UTC 
Nice! Not sure if you ran it recently.. does it work?
Comment 27 Marcelo Ricardo Leitner 2022-10-10 13:17:25 UTC 
Nice! Can we close this bz then? Do we need to identify a root cause or not? I think we won't need a z-stream request, because this is fixed in 8.4.z already per test above.
Comment 28 arn 2022-10-11 07:53:56 UTC 
Hi Marcelo/Jianlin,

Jianlin,

Thanks for verifying it. (Just for documenting sake. Hope you have used sw steering mode 'smfs'. Because in dmfs it used to work fine with ACLs)

Marcelo,
Since Jianlin has verified it we can close it.

Thanks & Regards,
Abhiram R N
Comment 29 Jianlin Shi 2022-10-11 08:22:08 UTC 
(In reply to arn from comment #28)
> Hi Marcelo/Jianlin,
> 
> Jianlin,
> 
> Thanks for verifying it. (Just for documenting sake. Hope you have used sw
> steering mode 'smfs'. Because in dmfs it used to work fine with ACLs)
> 
> Marcelo,
> Since Jianlin has verified it we can close it.
> 
> Thanks & Regards,
> Abhiram R N

Hi Abhiram,

yes, I used smfs to reproduce and verify the issue.
but it's better that you verify it with the latest 8.4.z kernel.
Comment 30 arn 2022-10-11 17:29:36 UTC 
Hi Jianlin,

I verified it with below 
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# uname -r
4.18.0-305.66.1.el8_4.x86_64
[root@rhos-nfv-09 ~]# 
[root@rhos-nfv-09 ~]# devlink dev param show pci/0000:04:00.0 name flow_steering_mode
pci/0000:04:00.0:
  name flow_steering_mode type driver-specific
    values:
      cmode runtime value smfs
[root@rhos-nfv-09 ~]# 

and the playbook -->https://github.com/rh-nfv-int/playbooks/blob/main/regression/ovs-hwol/testNATwithACLoffload.yml

Its working.

Please kindly close it.

Thanks for the Support

Regards,
Abhiram R N
Comment 31 Marcelo Ricardo Leitner 2022-10-11 18:12:30 UTC 
Super! Thanks a lot folks.