2129866 – After a minor update, connectiviy issues between a new instance and metadata.

Bug 2129866 - After a minor update, connectiviy issues between a new instance and metadata.

Summary: After a minor update, connectiviy issues between a new instance and metadata.

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux Fast Datapath
Classification:	Red Hat
Component:	ovn-2021
Sub Component:
Version:	FDP 21.F
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	xsimonar
QA Contact:	Ehsan Elahi
Docs Contact:
URL:
Whiteboard:
Depends On:	2076604
Blocks:
TreeView+	depends on / blocked

Reported:	2022-09-26 13:41 UTC by Miguel Lavalle
Modified:	2023-02-09 00:26 UTC (History)
CC List:	20 users (show)
Fixed In Version:	ovn-2021-21.12.0-99.el8fdp
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	2093901
Environment:
Last Closed:	2023-02-09 00:26:28 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	FD-2313	0	None	None	None	2022-09-26 14:02:11 UTC
Red Hat Product Errata	RHBA-2023:0678	0	None	None	None	2023-02-09 00:26:31 UTC

Comment 4 xsimonar 2022-10-17 12:29:19 UTC

Fix posted upstream for review: http://patchwork.ozlabs.org/project/ovn/patch/20221017122046.1381341-1-xsimonar@redhat.com/

Comment 5 Yatin Karel 2022-10-21 11:13:30 UTC

(In reply to xsimonar from comment #4)
> Fix posted upstream for review:
> http://patchwork.ozlabs.org/project/ovn/patch/20221017122046.1381341-1-
> xsimonar/

Thanks Xavier, I am no longer seeing the issue with your patch ^ and then i realized you have pushed v3[1], that too worked in my tests. But i noticed different behavior[2] when i applied the v1/v3 patch on branch-21.12, seems it will need more changes when backporting the fix to branch-21.12?

[1] http://patchwork.ozlabs.org/project/ovn/patch/20221020145302.563272-1-xsimonar@redhat.com/
[2] table=0 and table=65 do not have references to new ofport after ifaced-id is set on new interface, and these set to new ofport after older interface get's deleted.
# bash switch.sh                                                                                                  
**********Flows after iface-id set for the new interface**********                                                                                
 cookie=0xba7cb07e, duration=28.899s, table=0, n_packets=7, n_bytes=566, idle_age=14, priority=100,in_port=2 actions=load:0x4->NXM_NX_REG13[],load
:0x3->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)        
 cookie=0xba7cb07e, duration=28.899s, table=37, n_packets=7, n_bytes=566, idle_age=14, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)  
 cookie=0xba7cb07e, duration=28.900s, table=38, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg15=0x2,metadata=0x1 actions=load:0x4->NXM_NX_
REG13[],load:0x3->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)                                                                           
 cookie=0xba7cb07e, duration=28.899s, table=39, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 act
ions=drop                                                                
 cookie=0xba7cb07e, duration=28.899s, table=64, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=pus
h:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xba7cb07e, duration=28.899s, table=65, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg15=0x2,metadata=0x1 actions=output:2         
**********Flows after old interface is deleted*********
 cookie=0xba7cb07e, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x4->NXM_NX_REG13[],load:0x3
->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xba7cb07e, duration=28.925s, table=37, n_packets=7, n_bytes=566, idle_age=14, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)  
 cookie=0xba7cb07e, duration=28.926s, table=38, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg15=0x2,metadata=0x1 actions=load:0x4->NXM_NX_
REG13[],load:0x3->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xba7cb07e, duration=28.925s, table=39, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 act
ions=drop
 cookie=0xba7cb07e, duration=28.925s, table=64, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=pus
h:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xba7cb07e, duration=0.002s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:3

Comment 6 xsimonar 2022-11-02 12:01:01 UTC

Thanks for the test.
Yes, the v3 patch is not backportable as such. I posted a v4 which should be backportable from that perspective.
However, it is still being discussed upstream.
I'll update this bz when the proper solution is accepted uspstream
Thanks

Comment 7 Yatin Karel 2022-11-02 12:40:18 UTC

(In reply to xsimonar from comment #6)
> Thanks for the test.
> Yes, the v3 patch is not backportable as such. I posted a v4 which should be
> backportable from that perspective.
> However, it is still being discussed upstream.
> I'll update this bz when the proper solution is accepted uspstream
> Thanks

Thanks for the update Xavier, yes v4 looks good with branch-21.12 too.

Comment 8 OVN Bot 2022-11-24 05:04:05 UTC

ovn22.09 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147494
ovn22.09 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147495
ovn22.06 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147496
ovn22.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147497
ovn22.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147498
ovn22.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147499

Comment 9 Ehsan Elahi 2022-12-05 20:28:59 UTC

Using the reproducer mentioned in comment 1

Reproduced on:
[root@@bz-2129866 ~]# rpm -qa |grep -E 'ovn|openvswitch'
ovn22.03-22.03.0-106.el8fdp.x86_64
ovn22.03-central-22.03.0-106.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-29.el8fdp.noarch
ovn22.03-host-22.03.0-106.el8fdp.x86_64
openvswitch2.17-2.17.0-57.el8fdp.x86_64

========= After running create.sh ===========
[root@@bz-2129866 ~]# ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9
 e2466ff3
[root@@bz-2129866 ~]# COOKIE=$(ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9)
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xe2466ff3, duration=40.898s, table=0, n_packets=2, n_bytes=140, idle_age=25, priority=100,in_port=2 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xe2466ff3, duration=40.898s, table=37, n_packets=2, n_bytes=140, idle_age=25, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xe2466ff3, duration=40.898s, table=38, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xe2466ff3, duration=40.898s, table=39, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xe2466ff3, duration=40.898s, table=64, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xe2466ff3, duration=40.898s, table=65, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg15=0x2,metadata=0x1 actions=output:2

============= After running switch.sh ================================
============= same in_port, flows remain unchanged =======================
[root@@bz-2129866 ~]# ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9
 e2466ff3
[root@@bz-2129866 ~]# COOKIE=$(ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9)
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xe2466ff3, duration=82.937s, table=0, n_packets=3, n_bytes=210, idle_age=38, priority=100,in_port=2 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xe2466ff3, duration=82.937s, table=37, n_packets=3, n_bytes=210, idle_age=38, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xe2466ff3, duration=82.937s, table=38, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xe2466ff3, duration=82.937s, table=39, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xe2466ff3, duration=82.937s, table=64, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xe2466ff3, duration=82.937s, table=65, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg15=0x2,metadata=0x1 actions=output:2
[root@@bz-2129866 ~]# ovs-vsctl del-port br-int ls1lp
================= No flows showing ======================
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
[root@@bz-2129866 ~]# ovn-appctl -t ovn-controller recompute
================ After recompute, flow are back with new in_port ==============
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xe2466ff3, duration=2.577s, table=0, n_packets=0, n_bytes=0, idle_age=2, priority=100,in_port=3 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xe2466ff3, duration=2.577s, table=37, n_packets=0, n_bytes=0, idle_age=2, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xe2466ff3, duration=2.577s, table=38, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xe2466ff3, duration=2.577s, table=39, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xe2466ff3, duration=2.577s, table=64, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xe2466ff3, duration=2.577s, table=65, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg15=0x2,metadata=0x1 actions=output:3

Verified on:
[root@@bz-2129866 ~]# rpm -qa |grep -E 'ovn|openvswitch'
ovn22.03-22.03.0-125.el8fdp.x86_64
ovn22.03-central-22.03.0-125.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-29.el8fdp.noarch
ovn22.03-host-22.03.0-125.el8fdp.x86_64
openvswitch2.17-2.17.0-65.el8fdp.x86_64

========= After running create.sh =======================
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xa0c4f4f3, duration=15.651s, table=0, n_packets=6, n_bytes=476, idle_age=0, priority=100,in_port=2 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xa0c4f4f3, duration=15.651s, table=37, n_packets=6, n_bytes=476, idle_age=0, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xa0c4f4f3, duration=15.651s, table=38, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],resubmit(,39)
 cookie=0xa0c4f4f3, duration=15.651s, table=39, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xa0c4f4f3, duration=15.651s, table=64, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xa0c4f4f3, duration=15.651s, table=65, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg15=0x2,metadata=0x1 actions=output:2

============= After running switch.sh ================================
============= different in_port, flows changed =======================

[root@@bz-2129866 ~]# COOKIE=$(ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9)
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xa0c4f4f3, duration=0.014s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xa0c4f4f3, duration=29.248s, table=37, n_packets=6, n_bytes=476, idle_age=14, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xa0c4f4f3, duration=29.248s, table=38, n_packets=0, n_bytes=0, idle_age=29, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],resubmit(,39)
 cookie=0xa0c4f4f3, duration=29.248s, table=39, n_packets=0, n_bytes=0, idle_age=29, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xa0c4f4f3, duration=29.248s, table=64, n_packets=0, n_bytes=0, idle_age=29, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xa0c4f4f3, duration=0.014s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:3

================= Flows are shown, no need to recompute ====================== 
[root@@bz-2129866 ~]# ovs-vsctl del-port br-int ls1lp
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xa0c4f4f3, duration=70.906s, table=0, n_packets=9, n_bytes=706, idle_age=9, priority=100,in_port=3 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xa0c4f4f3, duration=100.140s, table=37, n_packets=15, n_bytes=1182, idle_age=9, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xa0c4f4f3, duration=100.140s, table=38, n_packets=0, n_bytes=0, idle_age=100, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],resubmit(,39)
 cookie=0xa0c4f4f3, duration=100.140s, table=39, n_packets=0, n_bytes=0, idle_age=100, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xa0c4f4f3, duration=100.140s, table=64, n_packets=0, n_bytes=0, idle_age=100, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xa0c4f4f3, duration=70.906s, table=65, n_packets=0, n_bytes=0, idle_age=70, priority=100,reg15=0x2,metadata=0x1 actions=output:3

Comment 10 OVN Bot 2023-01-11 15:34:06 UTC

ovn-2021 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2160157

Comment 13 Ehsan Elahi 2023-01-17 18:40:46 UTC

Verified on:

openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
openvswitch2.17-2.17.0-50.el8fdp.x86_64
ovn-2021-central-21.12.0-99.el8fdp.x86_64
ovn-2021-21.12.0-99.el8fdp.x86_64
ovn-2021-host-21.12.0-99.el8fdp.x86_64

Comment 15 errata-xmlrpc 2023-02-09 00:26:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn-2021 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:0678

Note You need to log in before you can comment on or make changes to this bug.

apevec
bcafarel
ctrautma
dhill
eelahi
egarciar
ekuris
jiji
jishi
jmarcian
lhh
lmiccini
majopela
mmichels
mtomaska
rhos-maint
scohen
skaplons
xsimonar
ykarel