Description of problem:
Ping to floating ip associated with switchdev port from provider vlan network doesnt work. 
Same works well if switchdev port from geneve tenant network. 
Same works well if VirtIO port from provider vlan network.

Version-Release number of selected component (if applicable):
RHEL: rhel 8.4
Kernel: 4.18.0-305.12.1.el8_4.x86_64
Ovs: openvswitch2.15-2.15.0-26.el8fdp.x86_64
Ovn: ovn-2021-21.03.0-40.el8fdp.x86_64
     rhosp-ovn-2021-4.el8ost.1.noarch

How reproducible:
Always

Steps to Reproduce:
1. Attach floating ip to switchdev port of provider vlan network
2. Ping floating ip 

Actual results:
Ping fails

Expected results:
Ping should works

Additional info:
This is DVR disabled environment. 
This may evenutally fall to core-ovn.

Flow table at controller:

ufid:04589b76-9199-4504-a68e-3badd2c7a84a, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(ens12),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=72:f9:61:09:b9:79,dst=fa:16:3e:0b:8c:4b),eth_type(0x8100),vlan(vid=405,pcp=0),encap(eth_type(0x0800),ipv4(src=10.10.54.0/255.255.255.128,dst=10.10.54.187,proto=1,tos=0/0,ttl=64,frag=no),icmp(type=0/0,code=0/0)), packets:1358, bytes:138516, used:0.790s, dp:ovs, actions:ct_clear,pop_vlan,ct(zone=2,nat),recirc(0x14)

ufid:ea551a31-e55b-4980-a871-14ceda5a3eb2, recirc_id(0x14),dp_hash(0/0),skb_priority(0/0),in_port(ens12),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=10.10.54.187,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1358, bytes:133084, used:0.790s, dp:ovs, actions:ct(commit,zone=4,nat(dst=6.6.6.253)),recirc(0x15)

ufid:2ab96c60-4287-4eb5-9bca-fe99c3dab154, recirc_id(0x15),dp_hash(0/0),skb_priority(0/0),in_port(ens12),skb_mark(0/0),ct_state(0x21/0x3f),ct_zone(0/0),ct_mark(0/0),ct_label(0/0x1),eth(src=72:f9:61:09:b9:79,dst=fa:16:3e:0b:8c:4b),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=6.6.6.253,proto=0/0,tos=0/0x3,ttl=64,frag=no), packets:556, bytes:54488, used:0.790s, dp:ovs, actions:ct_clear,set(tunnel(tun_id=0x1,dst=10.10.51.150,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30004}),flags(df|csum|key))),set(eth(src=fa:16:3e:b3:fb:29,dst=f8:f2:1e:03:bf:f1)),set(ipv4(ttl=63)),genev_sys_6081


Now at compute:

ufid:4b036d8d-ab39-4960-bf49-45701ce4e977, skb_priority(0/0),tunnel(tun_id=0x1,src=10.10.51.171,dst=10.10.51.150,ttl=0/0,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30004/0x7fffffff}),flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(genev_sys_6081),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:b3:fb:29,dst=00:00:00:00:00:00/01:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:2375, bytes:232750, used:0.501s, offloaded:yes, dp:tc, actions:enp4s0f0_2

ufid:4e3b86c0-481e-4b54-bfc8-1a2cf494950a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_2),packet_type(ns=0/0,id=0/0),eth(src=f8:f2:1e:03:bf:f1,dst=fa:16:3e:b3:fb:29),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=8.0.0.0/248.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:2376, bytes:199584, used:0.260s, dp:tc, actions:push_vlan(vid=415,pcp=0),mx-bond,br-data

Back to controller

ufid:3f916f27-7ab7-4f48-b8cc-4ea9bad9cd16, recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(ens5),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=f8:f2:1e:03:bf:f1,dst=fa:16:3e:b3:fb:29),eth_type(0x8100),vlan(vid=415,pcp=0/0x0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no)), packets:1358, bytes:138516, used:0.790s, dp:ovs, actions:br-mgmt