Description of problem:
During RHOSP14 testing, I came across a discrepancy scenario in OpenFlow when I am testing L3 DVR. In my ODL L3 DVR configuration, my instance floating IP has configured properly with Compute external nic and egress & ingress traffic is accessible as expected.

But while analyzing flow it seems packet is not forward from DMAC to FIB table(21) and resubmitted twice to dispatcher again 17 which doesn't make sense for me and again it forward to ARP_CHK and ELAN tables which is not expected.

While reviewing Netvirt pipeline I noticed there is no change in flow table number w.r.t FIB and L3. But I am wondering why the flow is redirected to ARP_CHK and ELAN table from ACL table. Is there any changes in OVS2.10 flow pipeline.

How reproducible:
In Lab
RHOSP14 + OpenDaylight Oxygen

Steps to Reproduce:
1. Instance Details.
$ openstack server list
+--------------------------------------+-----------+--------+---------------------------------+--------+---------+
| ID                                   | Name      | Status | Networks                        | Image  | Flavor  |
+--------------------------------------+-----------+--------+---------------------------------+--------+---------+
| 43487f69-8085-4e5d-b36d-bbc1ed740c21 | instance1 | ACTIVE | internal=192.168.1.8, 10.0.0.50 | cirros | m1.tiny |
+--------------------------------------+-----------+--------+---------------------------------+--------+---------+
(overcloud) [stack@undercloud-0 ~]$ nova interface-list instance1
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| ACTIVE     | 4c9cd437-5418-48f8-b65f-faf65780d6bf | 1be640af-f38b-492a-b798-9d10ae1bd430 | 192.168.1.8  | fa:16:3e:e0:94:71 |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+


2. External Gateway IP and Mac.
# ifconfig external
external: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.1  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::5054:ff:feca:77d8  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:13b8::fe  prefixlen 64  scopeid 0x0<global>
        ether 52:54:00:ca:77:d8  txqueuelen 1000  (Ethernet)
        RX packets 19893  bytes 1375416 (1.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18291  bytes 14761114 (14.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


3. Neutron Router Details

$ openstack router list --long -f json
[
  {
    "Status": "ACTIVE",
    "External gateway info": "{\"network_id\": \"f020fa7a-f94a-444d-a415-6e53b2422353\", \"enable_snat\": true, \"external_fixed_ips\": [{\"subnet_id\": \"15c4dcb3-228a-4474-8b15-e1bc1841207e\", \"ip_address\": \"10.0.0.69\"}]}",
    "Name": "router1",
    "Tags": "",
    "Distributed": false,
    "Project": "f7e5b33741bf4422ada5108919f4dd30",
    "State": "UP",
    "Routes": "",
    "HA": null,
    "ID": "648c455c-0425-4ae6-95bd-047797cc20fd"
  }
]

4. VM Instance and port details.
# virsh domiflist instance-00000002
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tap4c9cd437-54 bridge     br-int     virtio      fa:16:3e:e0:94:71


5. Oftrace flow dump.

	# ovs-appctl ofproto/trace br-int "in_port=7,icmp,dl_src=fa:16:3e:e0:94:71,dl_dst=52:54:00:ca:77:d8,nw_src=192.168.1.8,nw_dst=10.0.0.1"
	Flow: icmp,in_port=7,vlan_tci=0x0000,dl_src=fa:16:3e:e0:94:71,dl_dst=52:54:00:ca:77:d8,nw_src=192.168.1.8,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0

	bridge("br-int")
	----------------
	 0. in_port=7,vlan_tci=0x0000/0x1fff, priority 4, cookie 0x8000000
	    write_metadata:0x180000000000/0xffffff0000000001
	    goto_table:17
	17. metadata=0x180000000000/0xffffff0000000000, priority 10, cookie 0x6900000
	    write_metadata:0x8000180000000000/0xfffffffffffffffe
	    goto_table:210
	210. ip,metadata=0x180000000000/0xfffff0000000000,dl_src=fa:16:3e:e0:94:71,nw_src=192.168.1.8, priority 61010, cookie 0x6900000
	    goto_table:211
	211. icmp, priority 100, cookie 0x6900000
	    write_metadata:0/0x2
	    goto_table:212
	212. ip,metadata=0x180000000000/0xfffff0000000000, priority 100, cookie 0x6900000
	    ct(table=213,zone=5502)
	    drop
	     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 213.
	     -> Sets the packet to an untracked state, and clears all the conntrack fields.

	Final flow: icmp,metadata=0x8000180000000000,in_port=7,vlan_tci=0x0000,dl_src=fa:16:3e:e0:94:71,dl_dst=52:54:00:ca:77:d8,nw_src=192.168.1.8,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0
	Megaflow: recirc_id=0,eth,icmp,in_port=7,vlan_tci=0x0000/0x1fff,dl_src=fa:16:3e:e0:94:71,nw_src=192.168.1.8,nw_frag=no
	Datapath actions: ct(zone=5502),recirc(0x15ab)

	===============================================================================
	recirc(0x15ab) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
	===============================================================================

	Flow: recirc_id=0x15ab,ct_state=new|trk,ct_zone=5502,eth,icmp,metadata=0x8000180000000000,in_port=7,vlan_tci=0x0000,dl_src=fa:16:3e:e0:94:71,dl_dst=52:54:00:ca:77:d8,nw_src=192.168.1.8,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0

	bridge("br-int")
	----------------
	    thaw
		Resuming from table 213
	213. priority 0, cookie 0x6900000
	    goto_table:214
	    214. ip,metadata=0x180000000000/0xfffff0000000000, priority 2001, cookie 0x6900000
		    goto_table:217
		217. ip,metadata=0x180000000000/0xfffff0000000002, priority 100, cookie 0x6900000
		    ct(commit,zone=5502,exec(set_field:0x1->ct_mark))
		    set_field:0x1->ct_mark
		     -> Sets the packet to an untracked state, and clears all the conntrack fields.
		    ct_clear
		    resubmit(,17)
		17. metadata=0x8000180000000000/0xffffff0000000000, priority 10, cookie 0x8000001
		    load:0x19258->NXM_NX_REG3[0..24]
		    write_metadata:0x90001800000324b0/0xfffffffffffffffe
		    goto_table:19
### >>> The discripnecy has started from here where it again resubmitted flow to dispatcher instead for FIB(21) and forward to ARP_CHK which is not accurate.

		19. priority 0, cookie 0x1080000
		    resubmit(,17)
		17. metadata=0x9000180000000000/0xffffff0000000000, priority 10, cookie 0x8040000
		    load:0x18->NXM_NX_REG1[0..19]
		    load:0x157e->NXM_NX_REG7[0..15]
		    write_metadata:0xa00018157e000000/0xfffffffffffffffe
		    goto_table:43
		43. priority 0, cookie 0x8220000
		    goto_table:48
		48. priority 0, cookie 0x8500000
		    resubmit(,49)
		    49. No match.
		            drop
		    resubmit(,50)
		50. metadata=0x18157e000000/0xfffffffff000000,dl_src=fa:16:3e:e0:94:71, priority 20, cookie 0x805157e
		    goto_table:51
		51. priority 0, cookie 0x8030000
		    goto_table:52
		52. metadata=0x157e000000/0xffff000001, priority 5, cookie 0x870157e
		    write_actions(group:211004)
		     -> action set is: group:211004
	--. Executing action set:
	    group:211004
	    bucket 0
		    group:211003
		    bucket 0
		            set_field:0x18->tun_id
		            resubmit(,55)
		        55. tun_id=0x18,metadata=0x180000000000/0xfffff0000000000, priority 10, cookie 0x8800018
		            write_actions(drop)
		             -> action set is empty
	    bucket 1
		    set_field:0x44->tun_id
		    load:0xa00->NXM_NX_REG6[]
		    resubmit(,220)
		220. reg6=0xa00, priority 9, cookie 0x8000007
		    output:6
		     -> output to kernel tunnel
	    bucket 2
		    set_field:0x44->tun_id
		    load:0x300->NXM_NX_REG6[]
		    resubmit(,220)
		220. reg6=0x300, priority 9, cookie 0x8000007
		    output:4
		     -> output to kernel tunnel
	    bucket 3
		    set_field:0x44->tun_id
		    load:0x500->NXM_NX_REG6[]
		    resubmit(,220)
		220. reg6=0x500, priority 9, cookie 0x8000007
		    output:5
		     -> output to kernel tunnel

	Final flow: recirc_id=0x15ab,eth,icmp,reg1=0x18,reg3=0x19258,reg7=0x157e,metadata=0xa00018157e000000,in_port=7,vlan_tci=0x0000,dl_src=fa:16:3e:e0:94:71,dl_dst=52:54:00:ca:77:d8,nw_src=192.168.1.8,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0
	Megaflow: recirc_id=0x15ab,ct_state=+new-est-rel-inv+trk,ct_mark=0/0x1,eth,icmp,tun_id=0,in_port=7,dl_src=fa:16:3e:e0:94:71,dl_dst=52:54:00:ca:77:d8,nw_ecn=0,nw_frag=no
	Datapath actions: ct(commit,zone=5502,mark=0x1/0xffffffff),ct_clear,set(tunnel(tun_id=0x44,src=172.17.2.19,dst=172.17.2.15,ttl=64,tp_dst=4789,flags(df|key))),9,set(tunnel(tun_id=0x44,src=172.17.2.19,dst=172.17.2.14,ttl=64,tp_dst=4789,flags(df|key))),9,set(tunnel(tun_id=0x44,src=172.17.2.19,dst=172.17.2.13,ttl=64,tp_dst=4789,flags(df|key))),9


Actual results:
Flow is not forwarding to L3 FIB table (21), then how the floating IP is accessible in ODL L3 DVR scenario.


Expected results:
With expected scenario, DMAC(19) should forward to L3-FIB(21).

Additional info:
For reference Netvirt Flow table captured by odltools.
# odltools netvirt show tables -i 172.17.1.28 -t 8081 -u admin -w redhat -p
  0:INGRESS
 17:DISPATCHER
 18:DHCP_EXT_TUN
 19:L3_GW_MAC
 20:L3_LFIB
 21:L3_FIB
 22:L3_SUBNET_RT
 23:L3VNI_EXT_TUN
 24:L2VNI_EXT_TUN
 25:PDNAT
 26:PSNAT
 27:DNAT
 28:SNAT
 36:INT_TUN
 38:EXT_TUN
 43:ARP_CHK
 44:IN_NAPT
 45:IPV6
 46:OUT_NAPT
 47:NAPT_FIB
 48:ELAN_BASE
 50:ELAN_SMAC
 51:ELAN_DMAC
 52:ELAN_UNK_DMAC
 55:ELAN_FILTER
 60:DHCP
 80:L3_INTF
 81:ARP_RESPONDER
 90:QOS_DSCP
210:IN_ACL_ASPF
211:IN_ACL_CTRK_CLASS
212:IN_ACL_CTRK_SNDR
213:IN_ACL_EXISTING
214:IN_ACL_FLTR_DISP
215:IN_ACL_RULE_FLTR
216:IN_ACL_REM
217:IN_ACL_CMTR
220:EG_LPORT_DISP
239:EG_ACL_DUMMY
240:EG_ACL_ASPF
241:EG_ACL_CTRK_CLASS
242:EG_ACL_CTRK_SNDR
243:EG_ACL_EXISTING
244:EG_ACL_FLTR_DISP
245:EG_ACL_RULE_FLTR
246:EG_ACL_REM
247:EG_ACL_CMTR

Regards,
Pradipta