Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2129866

Summary: After a minor update, connectiviy issues between a new instance and metadata.
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Miguel Lavalle <mlavalle>
Component: ovn-2021Assignee: xsimonar
Status: CLOSED ERRATA QA Contact: Ehsan Elahi <eelahi>
Severity: high Docs Contact:
Priority: high    
Version: FDP 21.FCC: apevec, bcafarel, ctrautma, dhill, eelahi, egarciar, ekuris, jiji, jishi, jmarcian, lhh, lmiccini, majopela, mmichels, mtomaska, rhos-maint, scohen, skaplons, xsimonar, ykarel
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn-2021-21.12.0-99.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2093901 Environment:
Last Closed: 2023-02-09 00:26:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2076604    
Bug Blocks:    

Comment 4 xsimonar 2022-10-17 12:29:19 UTC 
Fix posted upstream for review: http://patchwork.ozlabs.org/project/ovn/patch/20221017122046.1381341-1-xsimonar@redhat.com/
Comment 5 Yatin Karel 2022-10-21 11:13:30 UTC 
(In reply to xsimonar from comment #4)
> Fix posted upstream for review:
> http://patchwork.ozlabs.org/project/ovn/patch/20221017122046.1381341-1-
> xsimonar/

Thanks Xavier, I am no longer seeing the issue with your patch ^ and then i realized you have pushed v3[1], that too worked in my tests. But i noticed different behavior[2] when i applied the v1/v3 patch on branch-21.12, seems it will need more changes when backporting the fix to branch-21.12?

[1] http://patchwork.ozlabs.org/project/ovn/patch/20221020145302.563272-1-xsimonar@redhat.com/
[2] table=0 and table=65 do not have references to new ofport after ifaced-id is set on new interface, and these set to new ofport after older interface get's deleted.
# bash switch.sh                                                                                                  
**********Flows after iface-id set for the new interface**********                                                                                
 cookie=0xba7cb07e, duration=28.899s, table=0, n_packets=7, n_bytes=566, idle_age=14, priority=100,in_port=2 actions=load:0x4->NXM_NX_REG13[],load
:0x3->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)        
 cookie=0xba7cb07e, duration=28.899s, table=37, n_packets=7, n_bytes=566, idle_age=14, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)  
 cookie=0xba7cb07e, duration=28.900s, table=38, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg15=0x2,metadata=0x1 actions=load:0x4->NXM_NX_
REG13[],load:0x3->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)                                                                           
 cookie=0xba7cb07e, duration=28.899s, table=39, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 act
ions=drop                                                                
 cookie=0xba7cb07e, duration=28.899s, table=64, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=pus
h:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xba7cb07e, duration=28.899s, table=65, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg15=0x2,metadata=0x1 actions=output:2         
**********Flows after old interface is deleted*********
 cookie=0xba7cb07e, duration=0.002s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x4->NXM_NX_REG13[],load:0x3
->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xba7cb07e, duration=28.925s, table=37, n_packets=7, n_bytes=566, idle_age=14, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)  
 cookie=0xba7cb07e, duration=28.926s, table=38, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg15=0x2,metadata=0x1 actions=load:0x4->NXM_NX_
REG13[],load:0x3->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xba7cb07e, duration=28.925s, table=39, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 act
ions=drop
 cookie=0xba7cb07e, duration=28.925s, table=64, n_packets=0, n_bytes=0, idle_age=28, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=pus
h:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xba7cb07e, duration=0.002s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:3
Comment 6 xsimonar 2022-11-02 12:01:01 UTC 
Thanks for the test.
Yes, the v3 patch is not backportable as such. I posted a v4 which should be backportable from that perspective.
However, it is still being discussed upstream.
I'll update this bz when the proper solution is accepted uspstream
Thanks
Comment 7 Yatin Karel 2022-11-02 12:40:18 UTC 
(In reply to xsimonar from comment #6)
> Thanks for the test.
> Yes, the v3 patch is not backportable as such. I posted a v4 which should be
> backportable from that perspective.
> However, it is still being discussed upstream.
> I'll update this bz when the proper solution is accepted uspstream
> Thanks

Thanks for the update Xavier, yes v4 looks good with branch-21.12 too.
Comment 8 OVN Bot 2022-11-24 05:04:05 UTC 
ovn22.09 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147494
ovn22.09 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147495
ovn22.06 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147496
ovn22.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147497
ovn22.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147498
ovn22.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2147499
Comment 9 Ehsan Elahi 2022-12-05 20:28:59 UTC 
Using the reproducer mentioned in comment 1

Reproduced on:
[root@@bz-2129866 ~]# rpm -qa |grep -E 'ovn|openvswitch'
ovn22.03-22.03.0-106.el8fdp.x86_64
ovn22.03-central-22.03.0-106.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-29.el8fdp.noarch
ovn22.03-host-22.03.0-106.el8fdp.x86_64
openvswitch2.17-2.17.0-57.el8fdp.x86_64

========= After running create.sh ===========
[root@@bz-2129866 ~]# ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9
 e2466ff3
[root@@bz-2129866 ~]# COOKIE=$(ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9)
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xe2466ff3, duration=40.898s, table=0, n_packets=2, n_bytes=140, idle_age=25, priority=100,in_port=2 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xe2466ff3, duration=40.898s, table=37, n_packets=2, n_bytes=140, idle_age=25, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xe2466ff3, duration=40.898s, table=38, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xe2466ff3, duration=40.898s, table=39, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xe2466ff3, duration=40.898s, table=64, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xe2466ff3, duration=40.898s, table=65, n_packets=0, n_bytes=0, idle_age=40, priority=100,reg15=0x2,metadata=0x1 actions=output:2

============= After running switch.sh ================================
============= same in_port, flows remain unchanged =======================
[root@@bz-2129866 ~]# ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9
 e2466ff3
[root@@bz-2129866 ~]# COOKIE=$(ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9)
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xe2466ff3, duration=82.937s, table=0, n_packets=3, n_bytes=210, idle_age=38, priority=100,in_port=2 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xe2466ff3, duration=82.937s, table=37, n_packets=3, n_bytes=210, idle_age=38, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xe2466ff3, duration=82.937s, table=38, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xe2466ff3, duration=82.937s, table=39, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xe2466ff3, duration=82.937s, table=64, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xe2466ff3, duration=82.937s, table=65, n_packets=0, n_bytes=0, idle_age=82, priority=100,reg15=0x2,metadata=0x1 actions=output:2
[root@@bz-2129866 ~]# ovs-vsctl del-port br-int ls1lp
================= No flows showing ======================
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
[root@@bz-2129866 ~]# ovn-appctl -t ovn-controller recompute
================ After recompute, flow are back with new in_port ==============
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xe2466ff3, duration=2.577s, table=0, n_packets=0, n_bytes=0, idle_age=2, priority=100,in_port=3 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xe2466ff3, duration=2.577s, table=37, n_packets=0, n_bytes=0, idle_age=2, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xe2466ff3, duration=2.577s, table=38, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x4->NXM_NX_REG11[],load:0x1->NXM_NX_REG12[],resubmit(,39)
 cookie=0xe2466ff3, duration=2.577s, table=39, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xe2466ff3, duration=2.577s, table=64, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xe2466ff3, duration=2.577s, table=65, n_packets=0, n_bytes=0, idle_age=2, priority=100,reg15=0x2,metadata=0x1 actions=output:3

Verified on:
[root@@bz-2129866 ~]# rpm -qa |grep -E 'ovn|openvswitch'
ovn22.03-22.03.0-125.el8fdp.x86_64
ovn22.03-central-22.03.0-125.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-29.el8fdp.noarch
ovn22.03-host-22.03.0-125.el8fdp.x86_64
openvswitch2.17-2.17.0-65.el8fdp.x86_64

========= After running create.sh =======================
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xa0c4f4f3, duration=15.651s, table=0, n_packets=6, n_bytes=476, idle_age=0, priority=100,in_port=2 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xa0c4f4f3, duration=15.651s, table=37, n_packets=6, n_bytes=476, idle_age=0, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xa0c4f4f3, duration=15.651s, table=38, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],resubmit(,39)
 cookie=0xa0c4f4f3, duration=15.651s, table=39, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xa0c4f4f3, duration=15.651s, table=64, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xa0c4f4f3, duration=15.651s, table=65, n_packets=0, n_bytes=0, idle_age=15, priority=100,reg15=0x2,metadata=0x1 actions=output:2

============= After running switch.sh ================================
============= different in_port, flows changed =======================

[root@@bz-2129866 ~]# COOKIE=$(ovn-sbctl find port_binding logical_port=ls1lp|grep uuid|cut -d: -f2| cut -c1-9)
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xa0c4f4f3, duration=0.014s, table=0, n_packets=0, n_bytes=0, idle_age=0, priority=100,in_port=3 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xa0c4f4f3, duration=29.248s, table=37, n_packets=6, n_bytes=476, idle_age=14, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xa0c4f4f3, duration=29.248s, table=38, n_packets=0, n_bytes=0, idle_age=29, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],resubmit(,39)
 cookie=0xa0c4f4f3, duration=29.248s, table=39, n_packets=0, n_bytes=0, idle_age=29, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xa0c4f4f3, duration=29.248s, table=64, n_packets=0, n_bytes=0, idle_age=29, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xa0c4f4f3, duration=0.014s, table=65, n_packets=0, n_bytes=0, idle_age=0, priority=100,reg15=0x2,metadata=0x1 actions=output:3

================= Flows are shown, no need to recompute ====================== 
[root@@bz-2129866 ~]# ovs-vsctl del-port br-int ls1lp
[root@@bz-2129866 ~]# ovs-ofctl dump-flows br-int|grep $COOKIE
 cookie=0xa0c4f4f3, duration=70.906s, table=0, n_packets=9, n_bytes=706, idle_age=9, priority=100,in_port=3 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],load:0x1->NXM_NX_REG10[10],resubmit(,8)
 cookie=0xa0c4f4f3, duration=100.140s, table=37, n_packets=15, n_bytes=1182, idle_age=9, priority=150,reg14=0x2,metadata=0x1 actions=resubmit(,38)
 cookie=0xa0c4f4f3, duration=100.140s, table=38, n_packets=0, n_bytes=0, idle_age=100, priority=100,reg15=0x2,metadata=0x1 actions=load:0x3->NXM_NX_REG13[],load:0x1->NXM_NX_REG11[],load:0x4->NXM_NX_REG12[],resubmit(,39)
 cookie=0xa0c4f4f3, duration=100.140s, table=39, n_packets=0, n_bytes=0, idle_age=100, priority=100,reg10=0/0x1,reg14=0x2,reg15=0x2,metadata=0x1 actions=drop
 cookie=0xa0c4f4f3, duration=100.140s, table=64, n_packets=0, n_bytes=0, idle_age=100, priority=100,reg10=0x1/0x1,reg15=0x2,metadata=0x1 actions=push:NXM_OF_IN_PORT[],load:0xffff->NXM_OF_IN_PORT[],resubmit(,65),pop:NXM_OF_IN_PORT[]
 cookie=0xa0c4f4f3, duration=70.906s, table=65, n_packets=0, n_bytes=0, idle_age=70, priority=100,reg15=0x2,metadata=0x1 actions=output:3
Comment 10 OVN Bot 2023-01-11 15:34:06 UTC 
ovn-2021 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2160157
Comment 13 Ehsan Elahi 2023-01-17 18:40:46 UTC 
Verified on:

openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
openvswitch2.17-2.17.0-50.el8fdp.x86_64
ovn-2021-central-21.12.0-99.el8fdp.x86_64
ovn-2021-21.12.0-99.el8fdp.x86_64
ovn-2021-host-21.12.0-99.el8fdp.x86_64
Comment 15 errata-xmlrpc 2023-02-09 00:26:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn-2021 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:0678