Created attachment 1739682 [details] dbs and logs When a port is replugged several times to the OVN bridge, its ofport will change. ovn-controller doesn't seem to honor these changes in table 65 and traffic from such port will not work anymore. The issue gets fixed triggering a recompute via: ovn-appctl -t ovn-controller recompute so everything seems to point to an issue of the incremental processing framework. Tests have reported this failing with ovn2.13-20.06/20.09 and a scratch build rebased on 20.12. Example: Port with ofport 745 shows in table 0: bash-4.4# ovs-ofctl dump-flows br-int table=0 | grep 745 cookie=0x38937d8e, duration=40387.372s, table=0, n_packets=1863, n_bytes=111678, idle_age=1, priority=100,in_port=745 actions=load:0x4b->NXM_NX_REG13[],load:0x6a->NXM_NX_REG11[],load:0x69->NXM_NX_REG12[],load:0x18d->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) However 737 is used in table 65: bash-4.4# ovs-ofctl dump-flows br-int table=65 | grep 8937d8e cookie=0x38937d8e, duration=40593.699s, table=65, n_packets=1848, n_bytes=98960, idle_age=2599, priority=100,reg15=0x1,metadata=0x18d actions=output:737 Older entry in the OVS database shows that 737 was the previous ofport of this port before it was re-added to the bridge: record 6516: 2020-12-14 22:22:54.184 table Interface row "tap71a5dfc1-10" (073801e2): ofport=737 I'm attaching NB/SB/OVS databases as well as physical and logical flows dumps for analysis.
Created attachment 1739685 [details] OVN controller logs
Fix posted for review: http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dceara@redhat.com/
Hi Dumitru I failed to reproduce it. Could you help to check if I am missing something? Steps: ovn-nbctl ls-add sw ovn-nbctl lsp-add sw lsp1 ovn-nbctl lsp-add sw lsp2 ovs-vsctl \ -- add-port br-int vm1 \ -- set Interface vm1 external_ids:iface-id=lsp1 \ ofport-request=1 ovs-vsctl \ -- add-port br-int vm2 \ -- set Interface vm2 external_ids:iface-id=lsp2 \ ofport-request=2 for i in `seq 3 100` do ovs-vsctl \ -- del-port vm1 \ -- del-port vm2 ovs-vsctl \ -- add-port br-int vm1 \ -- set Interface vm1 external_ids:iface-id=lsp1 \ ofport-request=$i \ -- add-port br-int vm2 \ -- set Interface vm2 external_ids:iface-id=lsp2 \ ofport-request=$(($i+1)) echo $i done # ovs-ofctl dump-flows br-int table=0 |grep in_port cookie=0xc5e16eb4, duration=25.042s, table=0, n_packets=0, n_bytes=0, idle_age=25, priority=100,in_port=101 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) cookie=0x1192e79a, duration=25.042s, table=0, n_packets=0, n_bytes=0, idle_age=25, priority=100,in_port=100 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) # ovs-ofctl dump-flows br-int table=65|grep output cookie=0xc5e16eb4, duration=36.695s, table=65, n_packets=0, n_bytes=0, idle_age=36, priority=100,reg15=0x2,metadata=0x1 actions=output:101 cookie=0x1192e79a, duration=36.695s, table=65, n_packets=0, n_bytes=0, idle_age=36, priority=100,reg15=0x1,metadata=0x1 actions=output:100 [root@dell-per730-17 ~]# rpm -qa|grep ovn ovn2.13-20.09.0-8.el8fdp.x86_64 ovn2.13-central-20.09.0-8.el8fdp.x86_64 ovn2.13-host-20.09.0-8.el8fdp.x86_64 [root@dell-per730-17 ~]# rpm -qa|grep openv python3-openvswitch2.13-2.13.0-9.el8fdp.x86_64 openvswitch2.13-2.13.0-9.el8fdp.x86_64 [root@dell-per730-17 ~]#
Hi Fei Liu, With ovn2.13-20.09.0-23.el8fdp I did: ovn-nbctl ls-add sw ovn-nbctl lsp-add sw lsp1 ovn-nbctl lsp-add sw lsp2 ovs-vsctl \ -- add-port br-int vm1 \ -- set Interface vm1 type=internal \ -- set Interface vm1 external_ids:iface-id=lsp1 \ ofport-request=1 ovs-vsctl \ -- add-port br-int vm2 \ -- set Interface vm2 type=internal \ -- set Interface vm2 external_ids:iface-id=lsp2 \ ofport-request=2 while [ $(ovn-sbctl show | grep -c Port_Binding) -ne 2 ]; do sleep 1; done; ovn-appctl -t ovn-controller debug/pause ovs-vsctl \ -- del-port vm1 \ -- del-port vm2 ovs-vsctl \ -- add-port br-int vm1 \ -- set Interface vm1 type=internal \ -- set Interface vm1 external_ids:iface-id=lsp1 \ ofport-request=$i \ -- add-port br-int vm2 \ -- set Interface vm2 type=internal \ -- set Interface vm2 external_ids:iface-id=lsp2 \ ofport-request=$(($i+1)) ovn-appctl -t ovn-controller debug/resume $ ovs-ofctl dump-flows br-int table=0 |grep in_port cookie=0xa3cb8b6d, duration=13.346s, table=0, n_packets=0, n_bytes=0, idle_age=13, priority=100,in_port=1 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0xbd13d6c3, duration=13.334s, table=0, n_packets=0, n_bytes=0, idle_age=13, priority=100,in_port=2 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) cookie=0xa3cb8b6d, duration=13.263s, table=0, n_packets=0, n_bytes=0, idle_age=13, priority=100,in_port=3 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0xbd13d6c3, duration=13.263s, table=0, n_packets=0, n_bytes=0, idle_age=13, priority=100,in_port=4 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) $ ovs-ofctl dump-flows br-int table=65|grep output cookie=0xa3cb8b6d, duration=19.024s, table=65, n_packets=0, n_bytes=0, idle_age=19, priority=100,reg15=0x1,metadata=0x1 actions=output:1 cookie=0xbd13d6c3, duration=19.013s, table=65, n_packets=0, n_bytes=0, idle_age=19, priority=100,reg15=0x2,metadata=0x1 actions=output:2 # Now if we trigger a manual recompute we see the correct flows: $ ovn-appctl -t ovn-controller recompute $ ovs-ofctl dump-flows br-int table=0 |grep in_port cookie=0xa3cb8b6d, duration=194.976s, table=0, n_packets=0, n_bytes=0, idle_age=194, priority=100,in_port=3 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0xbd13d6c3, duration=194.976s, table=0, n_packets=0, n_bytes=0, idle_age=194, priority=100,in_port=4 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) $ ovs-ofctl dump-flows br-int table=65|grep output cookie=0xa3cb8b6d, duration=212.088s, table=65, n_packets=0, n_bytes=0, idle_age=212, hard_age=36, priority=100,reg15=0x1,metadata=0x1 actions=output:3 cookie=0xbd13d6c3, duration=212.076s, table=65, n_packets=0, n_bytes=0, idle_age=212, hard_age=36, priority=100,reg15=0x2,metadata=0x1 actions=output:4 Regards, Dumitru
(In reply to Dumitru Ceara from comment #6) > Hi Fei Liu, > > With ovn2.13-20.09.0-23.el8fdp I did: > > ovn-nbctl ls-add sw > ovn-nbctl lsp-add sw lsp1 > ovn-nbctl lsp-add sw lsp2 > > ovs-vsctl \ > -- add-port br-int vm1 \ > -- set Interface vm1 type=internal \ > -- set Interface vm1 external_ids:iface-id=lsp1 \ > ofport-request=1 > ovs-vsctl \ > -- add-port br-int vm2 \ > -- set Interface vm2 type=internal \ > -- set Interface vm2 external_ids:iface-id=lsp2 \ > ofport-request=2 > > while [ $(ovn-sbctl show | grep -c Port_Binding) -ne 2 ]; do sleep 1; done; > > ovn-appctl -t ovn-controller debug/pause > ovs-vsctl \ > -- del-port vm1 \ > -- del-port vm2 > ovs-vsctl \ > -- add-port br-int vm1 \ > -- set Interface vm1 type=internal \ > -- set Interface vm1 external_ids:iface-id=lsp1 \ > ofport-request=$i \ > -- add-port br-int vm2 \ > -- set Interface vm2 type=internal \ > -- set Interface vm2 external_ids:iface-id=lsp2 \ > ofport-request=$(($i+1)) > ovn-appctl -t ovn-controller debug/resume > > $ ovs-ofctl dump-flows br-int table=0 |grep in_port > cookie=0xa3cb8b6d, duration=13.346s, table=0, n_packets=0, n_bytes=0, > idle_age=13, priority=100,in_port=1 > actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2- > >NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[], > resubmit(,8) > cookie=0xbd13d6c3, duration=13.334s, table=0, n_packets=0, n_bytes=0, > idle_age=13, priority=100,in_port=2 > actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2- > >NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[], > resubmit(,8) > cookie=0xa3cb8b6d, duration=13.263s, table=0, n_packets=0, n_bytes=0, > idle_age=13, priority=100,in_port=3 > actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2- > >NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[], > resubmit(,8) > cookie=0xbd13d6c3, duration=13.263s, table=0, n_packets=0, n_bytes=0, > idle_age=13, priority=100,in_port=4 > actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2- > >NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[], > resubmit(,8) > > $ ovs-ofctl dump-flows br-int table=65|grep output > cookie=0xa3cb8b6d, duration=19.024s, table=65, n_packets=0, n_bytes=0, > idle_age=19, priority=100,reg15=0x1,metadata=0x1 actions=output:1 > cookie=0xbd13d6c3, duration=19.013s, table=65, n_packets=0, n_bytes=0, > idle_age=19, priority=100,reg15=0x2,metadata=0x1 actions=output:2 > > # Now if we trigger a manual recompute we see the correct flows: > $ ovn-appctl -t ovn-controller recompute > > $ ovs-ofctl dump-flows br-int table=0 |grep in_port > cookie=0xa3cb8b6d, duration=194.976s, table=0, n_packets=0, n_bytes=0, > idle_age=194, priority=100,in_port=3 > actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2- > >NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[], > resubmit(,8) > cookie=0xbd13d6c3, duration=194.976s, table=0, n_packets=0, n_bytes=0, > idle_age=194, priority=100,in_port=4 > actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2- > >NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[], > resubmit(,8) > > $ ovs-ofctl dump-flows br-int table=65|grep output > cookie=0xa3cb8b6d, duration=212.088s, table=65, n_packets=0, n_bytes=0, > idle_age=212, hard_age=36, priority=100,reg15=0x1,metadata=0x1 > actions=output:3 > cookie=0xbd13d6c3, duration=212.076s, table=65, n_packets=0, n_bytes=0, > idle_age=212, hard_age=36, priority=100,reg15=0x2,metadata=0x1 > actions=output:4 > > Regards, > Dumitru Thanks for your reply,it help me a lot. BTW , On which version is this issue fixed??
(In reply to Fei Liu from comment #7) [...] > > Thanks for your reply,it help me a lot. BTW , On which version is this issue > fixed?? This is fixed in ovn2.13-20.12.0-1.
# cat reproducer.sh ovn-nbctl ls-add sw ovn-nbctl lsp-add sw lsp1 ovn-nbctl lsp-add sw lsp2 ovs-vsctl \ -- add-port br-int vm1 \ -- set Interface vm1 type=internal \ -- set Interface vm1 external_ids:iface-id=lsp1 \ ofport-request=1 ovs-vsctl \ -- add-port br-int vm2 \ -- set Interface vm2 type=internal \ -- set Interface vm2 external_ids:iface-id=lsp2 \ ofport-request=2 while [ $(ovn-sbctl show | grep -c Port_Binding) -ne 2 ]; do sleep 1; done; ovn-appctl -t ovn-controller debug/pause ovs-vsctl \ -- del-port vm1 \ -- del-port vm2 ovs-vsctl \ -- add-port br-int vm1 \ -- set Interface vm1 type=internal \ -- set Interface vm1 external_ids:iface-id=lsp1 \ ofport-request=3 \ -- add-port br-int vm2 \ -- set Interface vm2 type=internal \ -- set Interface vm2 external_ids:iface-id=lsp2 \ ofport-request=4 ovn-appctl -t ovn-controller debug/resume ovs-ofctl dump-flows br-int table=0 |grep in_port ovs-ofctl dump-flows br-int table=65|grep output ovn-appctl -t ovn-controller recompute ovs-ofctl dump-flows br-int table=0 |grep in_port ovs-ofctl dump-flows br-int table=65|grep output reproduce with ovn2.13-20.09.0-23 # ./reproducer.sh cookie=0xd4c894f6, duration=0.100s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=1 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0xd017a563, duration=0.085s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=2 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) cookie=0xd4c894f6, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0xd017a563, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=4 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) cookie=0xd4c894f6, duration=0.105s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x1,metadata=0x1 actions=output:1 cookie=0xd017a563, duration=0.090s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:2 cookie=0xd4c894f6, duration=0.016s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0xd017a563, duration=0.016s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=4 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) cookie=0xd4c894f6, duration=0.116s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x1,metadata=0x1 actions=output:3 cookie=0xd017a563, duration=0.101s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:4 verify with ovn2.13-20.12.0-1 both in rhel7 and rhel8 # ./reproducer.sh cookie=0x3237e55e, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=4 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) cookie=0x5a7d20b0, duration=0.004s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0x3237e55e, duration=0.009s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:4 cookie=0x5a7d20b0, duration=0.009s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x1,metadata=0x1 actions=output:3 cookie=0x3237e55e, duration=0.015s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=4 actions=load:0x4->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8) cookie=0x5a7d20b0, duration=0.015s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x1->NXM_NX_REG13[],load:0x3->NXM_NX_REG11[],load:0x2->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8) cookie=0x3237e55e, duration=0.020s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:4 cookie=0x5a7d20b0, duration=0.020s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x1,metadata=0x1 actions=output:3
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0407
*** Bug 1918616 has been marked as a duplicate of this bug. ***
*** Bug 1889419 has been marked as a duplicate of this bug. ***
*** Bug 1976792 has been marked as a duplicate of this bug. ***