Bug 2054153
| Summary: | Flow not offloaded when local port mirroring enabled also unable to ping after a while | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | arn |
| Component: | openvswitch | Assignee: | Marcelo Ricardo Leitner <mleitner> |
| openvswitch sub component: | ovs-hw-offload | QA Contact: | ovs-qe |
| Status: | NEW --- | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | ctrautma, lariel, qding |
| Version: | RHEL 8.0 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2172625 | ||
Some more logs .. I tried the same scenario with tc policy as skip_hw. Here also same issue is seen. Initially it starts pinging after few successful ping it stops pinging and its unreachable from there on. Below are the logs and also flow dumps initially while ping was working and when it stopped pinging. [root@rhos-nfv-08 ~]# [root@rhos-nfv-08 ~]# ping 172.16.0.7 PING 172.16.0.7 (172.16.0.7) 56(84) bytes of data. 64 bytes from 172.16.0.7: icmp_seq=1 ttl=63 time=1.76 ms 64 bytes from 172.16.0.7: icmp_seq=2 ttl=63 time=0.580 ms 64 bytes from 172.16.0.7: icmp_seq=3 ttl=63 time=0.245 ms 64 bytes from 172.16.0.7: icmp_seq=4 ttl=63 time=0.235 ms 64 bytes from 172.16.0.7: icmp_seq=5 ttl=63 time=0.281 ms 64 bytes from 172.16.0.7: icmp_seq=6 ttl=63 time=0.248 ms 64 bytes from 172.16.0.7: icmp_seq=7 ttl=63 time=0.239 ms 64 bytes from 172.16.0.7: icmp_seq=8 ttl=63 time=0.239 ms 64 bytes from 172.16.0.7: icmp_seq=9 ttl=63 time=0.241 ms 64 bytes from 172.16.0.7: icmp_seq=10 ttl=63 time=0.234 ms 64 bytes from 172.16.0.7: icmp_seq=11 ttl=63 time=0.231 ms 64 bytes from 172.16.0.7: icmp_seq=12 ttl=63 time=0.229 ms 64 bytes from 172.16.0.7: icmp_seq=13 ttl=63 time=0.231 ms 64 bytes from 172.16.0.7: icmp_seq=14 ttl=63 time=0.244 ms 64 bytes from 172.16.0.7: icmp_seq=15 ttl=63 time=0.240 ms 64 bytes from 172.16.0.7: icmp_seq=16 ttl=63 time=0.230 ms 64 bytes from 172.16.0.7: icmp_seq=17 ttl=63 time=0.230 ms 64 bytes from 172.16.0.7: icmp_seq=18 ttl=63 time=0.243 ms 64 bytes from 172.16.0.7: icmp_seq=19 ttl=63 time=0.227 ms 64 bytes from 172.16.0.7: icmp_seq=20 ttl=63 time=0.232 ms 64 bytes from 172.16.0.7: icmp_seq=21 ttl=63 time=0.254 ms 64 bytes from 172.16.0.7: icmp_seq=22 ttl=63 time=0.218 ms 64 bytes from 172.16.0.7: icmp_seq=23 ttl=63 time=0.236 ms 64 bytes from 172.16.0.7: icmp_seq=24 ttl=63 time=0.233 ms 64 bytes from 172.16.0.7: icmp_seq=25 ttl=63 time=0.231 ms 64 bytes from 172.16.0.7: icmp_seq=26 ttl=63 time=0.237 ms 64 bytes from 172.16.0.7: icmp_seq=27 ttl=63 time=0.227 ms 64 bytes from 172.16.0.7: icmp_seq=28 ttl=63 time=0.230 ms 64 bytes from 172.16.0.7: icmp_seq=29 ttl=63 time=0.236 ms 64 bytes from 172.16.0.7: icmp_seq=30 ttl=63 time=0.230 ms 64 bytes from 172.16.0.7: icmp_seq=31 ttl=63 time=0.248 ms ^C --- 172.16.0.7 ping statistics --- 153 packets transmitted, 31 received, 79.7386% packet loss, time 811ms rtt min/avg/max/mdev = 0.218/0.297/1.756/0.273 ms [root@rhos-nfv-08 ~]# Flow dump during when ping initially is started [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:419ab9ed-c3ca-431d-a3ab-135eb73a3fc9, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0_0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=0/0,tos=0/0,ttl=64,frag=no), packets:26, bytes:2548, used:0.930s, dp:ovs, actions:enp4s0f0_2,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)),set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0xe) ufid:8859ba10-96a6-4ca9-bd45-037a80c1bc48, recirc_id(0xe),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0_0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:26, bytes:2548, used:0.930s, dp:ovs, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0 ufid:40c21c67-bc05-4564-a209-df6a04892ce4, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:26, bytes:2652, used:0.930s, dp:ovs, actions:pop_vlan,ct(zone=7,nat),recirc(0xc) ufid:56c50cc8-8a2a-4b5c-a818-5823fbdbc93c, recirc_id(0xc),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=0/0,tos=0/0,ttl=64,frag=no), packets:25, bytes:2450, used:0.930s, dp:ovs, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# After few seconds when ping is not working. From the flow actions though we see enp4s0f0_0 VM is not receiving any traffic and hence no response packets as well are coming. [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:40c21c67-bc05-4564-a209-df6a04892ce4, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:48, bytes:4896, used:0.487s, dp:ovs, actions:pop_vlan,ct(zone=7,nat),recirc(0xc) ufid:56c50cc8-8a2a-4b5c-a818-5823fbdbc93c, recirc_id(0xc),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=0/0,tos=0/0,ttl=64,frag=no), packets:47, bytes:4606, used:0.487s, dp:ovs, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:40c21c67-bc05-4564-a209-df6a04892ce4, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:69, bytes:7038, used:0.166s, dp:ovs, actions:pop_vlan,ct(zone=7,nat),recirc(0xc) ufid:56c50cc8-8a2a-4b5c-a818-5823fbdbc93c, recirc_id(0xc),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=0/0,tos=0/0,ttl=64,frag=no), packets:68, bytes:6664, used:0.166s, dp:ovs, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# Thanks & Regards, Abhiram R N Hi Abhiram, Which OS/Kernel did u use in your test? Hi @lariel , Below are the details. [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# uname -r 4.18.0-305.22.1.el8_4.x86_64 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 (Ootpa) [root@rhos-nfv-09 ~]# Thanks & Regards, Abhiram R N BTW - in what scope was this tested? According to the versions u mentioned it seems to be for OSP, is that correct? Hi Ariel, We were trying this as part of our testing in lab for 'port mirroring' use case on our setup where we have only OVN and OVS. (And without openstack). Thanks & Regards, Abhiram R N [covering only 1 of the issues in this bz here] (In reply to arn from comment #0) > ufid:9b591002-bd83-4e3c-93ef-0f3e0c0e2b67, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0), > ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/ > 0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800), > ipv4(src=192.168.0.2,dst=172.16.0.111,proto=1,tos=0/0,ttl=64,frag=no), > icmp(type=0/0,code=0/0), packets:19, bytes:1596, used:0.310s, dp:tc, > actions:enp4s0f0_2,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)), > set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0xc) Considering this flow has "actions:enp4s0f0_2,...,ct(),...", this is likely because of https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c#L3390 This code is present in RHEL 8.4: rhel8 ((kernel-4.18.0-305.30.1.el8_4))]$ git grep 'offload mirroring with action ct' drivers/net/ethernet/mellanox/mlx5/core/en_tc.c: "Can't offload mirroring with action ct"); And there's a good reason for that: ct() can trigger a miss. Considering it can't fallback to sw in the middle of the action list but only the entire flow, this means that if it had offloaded this flow and ct() triggered a miss, it would output the packet to enp4s0f0_2 twice. This is a design limitation. Yet, I can't see how OVN can avoid hitting this. I'm not aware of any way to tell OVS to generate that flow differently. (In reply to arn from comment #1) > Some more logs .. > > I tried the same scenario with tc policy as skip_hw. Here also same issue is Did you have hw-offload=true here? Because all flows pasted are using dp:ovs. I'm thinking this (traffic stopping) may be something else, unrelated to HWOL. (In reply to Marcelo Ricardo Leitner from comment #7) > (In reply to arn from comment #1) > > Some more logs .. > > > > I tried the same scenario with tc policy as skip_hw. Here also same issue is > > Did you have hw-offload=true here? Because all flows pasted are using dp:ovs. > I'm thinking this (traffic stopping) may be something else, unrelated to > HWOL. I think hw-offload was set to false. So, I tried today again just now. Still the ping not happening issue is there. From the below logs if you see before I create the mirrors things are looking fine. But once I create the mirrors the ping stops Initially when no mirror is present. [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:5183c14a-1d97-41cd-84c9-37215f0bff15, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:506, bytes:42504, used:0.010s, dp:tc, actions:pop_vlan,ct(zone=1,nat),recirc(0x5) ufid:5a5dcd56-1841-40f2-b858-88b1c6b19fa8, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0x5),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:505, bytes:42420, used:0.010s, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0 ufid:d1e6e22a-977f-4faf-9326-21d1a10c7df4, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:506, bytes:42504, used:0.010s, dp:tc, actions:set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)),set(ipv4(ttl=63)),ct(zone=1,nat),recirc(0xa) ufid:33452050-73ff-4796-8b95-d186447edbd2, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xa),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:505, bytes:42420, used:0.010s, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0 ufid:c71fc150-6105-4487-86f8-da71a3c23fc3, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(enp4s0f0_0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0806),arp(sip=192.168.0.2,tip=192.168.0.1,op=1/0xff,sha=6a:62:68:b2:61:f6,tha=00:00:00:00:00:00), packets:0, bytes:0, used:never, dp:ovs, actions:userspace(pid=3590968980,slow_path(action)) [root@rhos-nfv-09 ~]# Adding mirror configs. [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-vsctl -- --id=@m create mirror name=mymirror -- add bridge br-int mirrors @m 00dc7db5-d736-4afb-b6d1-121c8200f64e [root@rhos-nfv-09 ~]# ovs-vsctl -- --id=@enp4s0f0_0 get port enp4s0f0_0 -- set mirror mymirror select_src_port=@enp4s0f0_0 select_dst_port=@enp4s0f0_0 [root@rhos-nfv-09 ~]# ovs-vsctl -- --id=@enp4s0f0_2 get port enp4s0f0_2 -- set mirror mymirror output-port=@enp4s0f0_2 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:5183c14a-1d97-41cd-84c9-37215f0bff15, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:16, bytes:1344, used:0.950s, dp:tc, actions:pop_vlan,ct(zone=1,nat),recirc(0xb) ufid:c4d6fac1-12c7-44cf-a1dc-9dae6fc96650, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xb),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:15, bytes:1260, used:0.950s, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:5183c14a-1d97-41cd-84c9-37215f0bff15, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:22, bytes:1848, used:0.130s, dp:tc, actions:pop_vlan,ct(zone=1,nat),recirc(0xb) ufid:c4d6fac1-12c7-44cf-a1dc-9dae6fc96650, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xb),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:21, bytes:1764, used:0.130s, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-vsctl get Open_vSwitch . other_config:tc-policy skip_hw [root@rhos-nfv-09 ~]# ovs-vsctl get Open_vSwitch . other_config:hw-offload "true" [root@rhos-nfv-09 ~]# Thanks & Regards, Abhiram R N Considering this is reproducible with and without HWOL, it may be a network setup issue. When the mirroring gets configured, the switch gets confused on which port the MAC address belongs to, and forwards the packets to the wrong port. The switch in the case I think it is ovs itself. ARN, do we have any news here? Hi Marcelo, Was off on Paternity!.. Back now. Sorry for the delay. Maybe earlier what I mentioned might be confusing. 2 things are there here. a) For mirrored packets not all flows offloaded to hardware. b) Ping not happening.(this was not immediate but after a while) For now lets focus on a) . I checked it again and discussed with Haresh as well. Doesnt seem to be a Network setup issue. Let me try to explain with the flows where specifically we see a problem. See the below flows. Flows when ping initiated from remote side (external traffic). For ease of reading I have divided and put comments for each flow and then pasted the flow. [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m Incoming flows... First flow- incoming on enp4s0f0 . No mirroring involved here. Its offloaded and its fine. In actions we see ct(zone=7,nat) present ufid:32060819-a5a1-4a95-ae74-9be52a3d8d9a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:19, bytes:1596, used:0.310s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=7,nat),recirc(0xa) Second Flow - After NAT has happened packet was sent to enp4s0f0_0 properly also the packet is mirrored properly.(to enp4s0f0_2) . The flow is offloaded. So fine till here. ufid:c2c5d169-bc5f-48c5-b804-825e244fbef9, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xa),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:18, bytes:1512, used:0.310s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 Third Flow - Outgoing packet. In the actions we can see the packet is sent to enp4s0f0_2 and also see ct(zone=7,nat) is there in actions. There is where the problem is . Flow seems fine .But it is not offloaded . It is in dp:tc. So, clearly looks like when the NAT action and also the mirroring is involved on the same flow it didnt get offloaded. ufid:9b591002-bd83-4e3c-93ef-0f3e0c0e2b67, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:19, bytes:1596, used:0.310s, dp:tc, actions:enp4s0f0_2,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)),set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0xc) Fourth flow - Sending out packet on enp4s0f0 - Offloaded and its fine. ufid:3f934a12-867d-4101-b434-0237b329a02d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xc),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:19, bytes:1596, used:0.310s, offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0 [root@rhos-nfv-09 ~]# Thanks & Regards, Abhiram R N (In reply to arn from comment #12) > Flows when ping initiated from remote side (external traffic). For ease of > reading I have divided and put comments for each flow and then pasted the > flow. Thanks. > > [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m > > Incoming flows... > > First flow- incoming on enp4s0f0 . No mirroring involved here. Its offloaded > and its fine. In actions we see ct(zone=7,nat) present > ufid:32060819-a5a1-4a95-ae74-9be52a3d8d9a, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0), > ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0, > id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100), > vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255. > 192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), > packets:19, bytes:1596, used:0.310s, offloaded:yes, dp:tc, > actions:pop_vlan,ct(zone=7,nat),recirc(0xa) > > Second Flow - After NAT has happened packet was sent to enp4s0f0_0 properly > also the packet is mirrored properly.(to enp4s0f0_2) . The flow is > offloaded. So fine till here. > ufid:c2c5d169-bc5f-48c5-b804-825e244fbef9, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0), > ct_label(0/0),recirc_id(0xa),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/ > 0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800), > ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no), > icmp(type=0/0,code=0/0), packets:18, bytes:1512, used:0.310s, offloaded:yes, > dp:tc, > actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)), > set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 > > > Third Flow - Outgoing packet. In the actions we can see the packet is sent > to enp4s0f0_2 and also see ct(zone=7,nat) is there in actions. There is > where the problem is . Flow seems fine .But it is not offloaded . It is in > dp:tc. So, clearly looks like when the NAT action and also the mirroring is > involved on the same flow it didnt get offloaded. > ufid:9b591002-bd83-4e3c-93ef-0f3e0c0e2b67, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0), > ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/ > 0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800), > ipv4(src=192.168.0.2,dst=172.16.0.111,proto=1,tos=0/0,ttl=64,frag=no), > icmp(type=0/0,code=0/0), packets:19, bytes:1596, used:0.310s, dp:tc, > actions:enp4s0f0_2,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)), > set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0xc) This flow is not offloaded for a reason, yes. It is outputting a packet, mangling it, calling ct() and then doing recirc(). Problem is: ct() can miss in HW. If it misses, the entire action list needs to be done in sw, and not just the remaining part. That would mean the packet could get sent twice on enp4s0f0_2. So yes, I don't know why there's a ct() call in between the outputs here, but this is expected to not offload. If we need mirroring to be offloaded, we need to work with OVN and so to change the flows. Also, please mind that only TCP and UDP conntrack entries get offloaded. https://elixir.bootlin.com/linux/latest/source/net/sched/act_ct.c#L436 > > Fourth flow - Sending out packet on enp4s0f0 - Offloaded and its fine. > ufid:3f934a12-867d-4101-b434-0237b329a02d, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0), > ct_label(0/0),recirc_id(0xc),dp_hash(0/0),in_port(enp4s0f0_0), > packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10), > eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255. > 192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:19, bytes:1596, used:0.310s, > offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0 > [root@rhos-nfv-09 ~]# > > Thanks & Regards, > Abhiram R N Ah, this doesn't explain why it stops working, though. |
Description of problem: Network Topology Remote host ------------VLAN-------------------------local host with mellanox nic (4 VFs created) -- (2 VMs here ... vf 0 connected to VM1 and vf 1 connected to VM2) Configured Stateful NAT for external traffic from Remote host side. During working scenario before creating mirror................... Confgis.. [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-vsctl show 800e5315-5ef8-40c3-bd48-a35f90979838 Bridge br-ex Port patch-ln-public-to-br-int Interface patch-ln-public-to-br-int type: patch options: {peer=patch-br-int-to-ln-public} Port br-ex Interface br-ex type: internal Port enp4s0f0 Interface enp4s0f0 Bridge br-int fail_mode: secure datapath_type: system Port enp4s0f0_1 Interface enp4s0f0_1 Port enp4s0f0_0 Interface enp4s0f0_0 Port br-int Interface br-int type: internal Port enp4s0f0_2 Interface enp4s0f0_2 Port patch-br-int-to-ln-public Interface patch-br-int-to-ln-public type: patch options: {peer=patch-ln-public-to-br-int} ovs_version: "2.17.90" [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovn-nbctl show switch ef04eaec-cd80-42a4-b033-3bbd6af00f5d (public) port public-lr0 type: router router-port: lr0-public port ln-public type: localnet tag: 201 addresses: ["unknown"] switch 04994b58-5a1a-4774-8bc8-98799e31e590 (sw0) port sw0-port1 addresses: ["6a:62:68:b2:61:f6 dynamic"] port sw0-port2 addresses: ["66:f0:e4:1e:2c:bb dynamic"] port lrp0-attachment type: router router-port: lrp0 router 16c8181d-cfac-491a-a007-f2b869c67499 (lr0) port lr0-public mac: "0a:00:20:20:12:13" networks: ["172.16.0.1/24"] gateway chassis: [dummy] port lrp0 mac: "00:00:00:00:ff:01" networks: ["192.168.0.1/24"] nat 37f78c20-84ed-4c0a-a5c3-23ed21004f44 external ip: "172.16.0.8" logical ip: "192.168.0.3" type: "dnat_and_snat" nat 863687e2-ecbb-4a05-b069-55b5746abdf5 external ip: "172.16.0.7" logical ip: "192.168.0.2" type: "dnat_and_snat" nat e5926886-9330-4f6a-aa22-4f159800aa62 external ip: "172.16.0.2" logical ip: "192.168.0.0/24" type: "snat" [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovn-sbctl show Chassis "" hostname: rhos-nfv-09.lab.eng.rdu2.redhat.com Encap geneve ip: "10.8.2.160" options: {csum="true"} Port_Binding sw0-port1 Port_Binding sw0-port2 [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ovs-vsctl list bridge br-int _uuid : 6ce9f1ed-ef4a-4a51-82c0-164c93d7af89 auto_attach : [] controller : [] datapath_id : "00004e7193ac3192" datapath_type : system datapath_version : "<unknown>" external_ids : {ct-zone-1e48980b-5584-4c87-802b-727a0555de76_dnat="4", ct-zone-1e48980b-5584-4c87-802b-727a0555de76_snat="1", ct-zone-a0cd1fc6-3526-42a5-83de-b7607726a072_dnat="2", ct-zone-a0cd1fc6-3526-42a5-83de-b7607726a072_snat="5", ct-zone-a6b02f19-ca8e-4414-ab84-b14b4617546a_dnat="7", ct-zone-a6b02f19-ca8e-4414-ab84-b14b4617546a_snat="6", ct-zone-sw0-port1="3", ct-zone-sw0-port2="8", ovn-nb-cfg="3", ovn-nb-cfg-ts="1644827820510", ovn-startup-ts="1644829292265"} fail_mode : secure flood_vlans : [] flow_tables : {} ipfix : [] mcast_snooping_enable: false mirrors : [] name : br-int netflow : [] other_config : {disable-in-band="true", hwaddr="4e:71:93:ac:31:92"} ports : [16911803-41ac-4fba-a434-95ef5bde3eec, 23921f99-d6c4-45af-87ce-196024b5138e, 46211ef1-c787-4d40-a81c-da5eff85c277, adc20906-18ee-4492-9734-c5abce8cd38b, af5a5ed3-1af0-4b48-8a65-52a9eda740fb] protocols : [] rstp_enable : false rstp_status : {} sflow : [] status : {} stp_enable : false [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# ============================================= flows when ping initiated from remote side (external traffic) ============================================= [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:32060819-a5a1-4a95-ae74-9be52a3d8d9a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:27, bytes:2268, used:0.510s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=7,nat),recirc(0x4) ufid:0dd44d5c-c2eb-4d58-8772-0cfbf180407a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0x4),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:26, bytes:2184, used:0.510s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0 ufid:9b591002-bd83-4e3c-93ef-0f3e0c0e2b67, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:27, bytes:2268, used:0.510s, offloaded:yes, dp:tc, actions:set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)),set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0x9) ufid:5204d29c-b5e2-4e68-abb7-72a065b12817, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0x9),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:26, bytes:2184, used:0.510s, offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0 ========================================================== flows when ping initiated from same host from another VM [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:c7a01b58-8d81-4e1c-afcf-7870377bc9f3, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_1),packet_type(ns=0/0,id=0/0),eth(src=66:f0:e4:1e:2c:bb,dst=6a:62:68:b2:61:f6),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2/255.255.255.254,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:12, bytes:1176, used:0.620s, offloaded:yes, dp:tc, actions:enp4s0f0_0 ufid:c8b37c62-8fc3-4bc9-9bba-1fe1cf04c83e, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=66:f0:e4:1e:2c:bb),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2/255.255.255.254,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:12, bytes:1176, used:0.620s, offloaded:yes, dp:tc, actions:enp4s0f0_1 ========================================================== Added Ported mirroring confgs as below. ovs-vsctl -- --id=@m create mirror name=mymirror -- add bridge br-int mirrors @m ovs-vsctl -- --id=@enp4s0f0_0 get port enp4s0f0_0 -- set mirror mymirror select_src_port=@enp4s0f0_0 select_dst_port=@enp4s0f0_0 ovs-vsctl -- --id=@enp4s0f0_2 get port enp4s0f0_2 -- set mirror mymirror output-port=@enp4s0f0_2 [root@rhos-nfv-09 ~]# ovs-vsctl list mirror mymirror _uuid : 453b4cf2-e2d6-4c50-bd83-4b3044b67c06 external_ids : {} name : mymirror output_port : adc20906-18ee-4492-9734-c5abce8cd38b output_vlan : [] select_all : false select_dst_port : [23921f99-d6c4-45af-87ce-196024b5138e] select_src_port : [23921f99-d6c4-45af-87ce-196024b5138e] select_vlan : [] snaplen : [] statistics : {} [root@rhos-nfv-09 ~]# =========================================================== flows when ping initiated from same host from another VM -- No issue here. Working fine [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:c7a01b58-8d81-4e1c-afcf-7870377bc9f3, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_1),packet_type(ns=0/0,id=0/0),eth(src=66:f0:e4:1e:2c:bb,dst=6a:62:68:b2:61:f6),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2/255.255.255.254,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:19, bytes:1862, used:0.320s, offloaded:yes, dp:tc, actions:enp4s0f0_0,enp4s0f0_2 ufid:c8b37c62-8fc3-4bc9-9bba-1fe1cf04c83e, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=66:f0:e4:1e:2c:bb),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2/255.255.255.254,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:19, bytes:1862, used:0.320s, offloaded:yes, dp:tc, actions:enp4s0f0_2,enp4s0f0_1 Mirroring working fine... Can see the traffic on the mirrored output [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# tcpdump -i enp4s0f0v2 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp4s0f0v2, link-type EN10MB (Ethernet), capture size 262144 bytes 04:25:41.166644 IP 192.168.0.3 > 192.168.0.2: ICMP echo request, id 4944, seq 115, length 64 04:25:41.166732 IP 192.168.0.2 > 192.168.0.3: ICMP echo reply, id 4944, seq 115, length 64 04:25:42.190615 IP 192.168.0.3 > 192.168.0.2: ICMP echo request, id 4944, seq 116, length 64 04:25:42.190687 IP 192.168.0.2 > 192.168.0.3: ICMP echo reply, id 4944, seq 116, length 64 ============================================================ flows when ping initiated from remote side (external traffic) -- Issue observed here. 1 flow is not offloaded [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:c2c5d169-bc5f-48c5-b804-825e244fbef9, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xa),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:18, bytes:1512, used:0.310s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 ufid:32060819-a5a1-4a95-ae74-9be52a3d8d9a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:19, bytes:1596, used:0.310s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=7,nat),recirc(0xa) ufid:9b591002-bd83-4e3c-93ef-0f3e0c0e2b67, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=6a:62:68:b2:61:f6,dst=00:00:00:00:ff:01),eth_type(0x0800),ipv4(src=192.168.0.2,dst=172.16.0.111,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:19, bytes:1596, used:0.310s, dp:tc, actions:enp4s0f0_2,set(eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10)),set(ipv4(ttl=63)),ct(zone=7,nat),recirc(0xc) ufid:3f934a12-867d-4101-b434-0237b329a02d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xc),dp_hash(0/0),in_port(enp4s0f0_0),packet_type(ns=0/0,id=0/0),eth(src=0a:00:20:20:12:13,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=172.16.0.64/255.255.255.192,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:19, bytes:1596, used:0.310s, offloaded:yes, dp:tc, actions:ct_clear,push_vlan(vid=201,pcp=0),enp4s0f0 [root@rhos-nfv-09 ~]# Incoming flows are offloaded to HW and also mirrored. But Outgoing flow is not mirrored and not offloaded to HW [root@rhos-nfv-09 ~]# [root@rhos-nfv-09 ~]# tcpdump -i enp4s0f0v2 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp4s0f0v2, link-type EN10MB (Ethernet), capture size 262144 bytes 04:30:43.188466 IP 172.16.0.111 > 192.168.0.2: ICMP echo request, id 53508, seq 48, length 64 04:30:44.212394 IP 172.16.0.111 > 192.168.0.2: ICMP echo request, id 53508, seq 49, length 64 After a while even the ping request is not reaching the VM but still requests are seen on the mirrored port. Since the ping request is not reaching the VM the replies are also not sent and the flows are as below [root@rhos-nfv-08 ~]# [root@rhos-nfv-08 ~]# ping 172.16.0.7 PING 172.16.0.7 (172.16.0.7) 56(84) bytes of data. 64 bytes from 172.16.0.7: icmp_seq=1 ttl=63 time=30.7 ms [root@rhos-nfv-09 ~]# ovs-appctl dpctl/dump-flows -m ufid:32060819-a5a1-4a95-ae74-9be52a3d8d9a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x8100),vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=172.16.0.64/255.255.255.192,dst=172.16.0.7,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:893, bytes:75012, used:0.130s, offloaded:yes, dp:tc, actions:pop_vlan,ct(zone=7,nat),recirc(0xd) ufid:fd29df44-01a0-4a6b-90fb-5cb5628d9b89, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0xd),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=0a:00:20:20:12:13),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=192.168.0.2,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0), packets:891, bytes:74844, used:0.130s, offloaded:yes, dp:tc, actions:ct_clear,set(eth(src=00:00:00:00:ff:01,dst=6a:62:68:b2:61:f6)),set(ipv4(ttl=63)),enp4s0f0_0,enp4s0f0_2 [root@rhos-nfv-09 ~]# tcpdump -i enp4s0f0v2 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp4s0f0v2, link-type EN10MB (Ethernet), capture size 262144 bytes 04:46:00.500585 IP 172.16.0.111 > 192.168.0.2: ICMP echo request, id 53584, seq 112, length 64 04:46:01.524541 IP 172.16.0.111 > 192.168.0.2: ICMP echo request, id 53584, seq 113, length 64 04:46:02.548533 IP 172.16.0.111 > 192.168.0.2: ICMP echo request, id 53584, seq 114, length 64 Version-Release number of selected component (if applicable): OVS 2.17.90 How reproducible: Use below topology Remote host ------------VLAN-------------------------local host with mellanox nic (4 VFs created) -- (2 VMs here ... vf 0 given to VM1 and vf 1 given to VM2) Configured Stateful NAT for external traffic from Remote host side. Create mirror on OVS bridge to output traffic incoming and outgoing on vf 0 to vf 2. Steps to Reproduce: 1. Initiate ping from Remote host to the VM on the local host. 2. 3. Actual results: Not all flows are offloaded. Expected results: All flows should be offloaded. Additional info: