Bug 1939524 - Changing rate on meter does not take effect unless ovn-controller is restarted
Summary: Changing rate on meter does not take effect unless ovn-controller is restarted
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: ovn2.13
Version: FDP 20.H
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: FDP 22.C
Assignee: lorenzo bianconi
QA Contact: Ehsan Elahi
URL:
Whiteboard:
Depends On:
Blocks: 1619266 2051954 2073462
TreeView+ depends on / blocked
 
Reported: 2021-03-16 15:08 UTC by ffernand
Modified: 2022-04-25 14:27 UTC (History)
8 users (show)

Fixed In Version: ovn-2021-21.12.0-42.el8fdp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2073462 (view as bug list)
Environment:
Last Closed: 2022-04-25 14:26:54 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FD-1133 0 None None None 2021-09-24 13:52:50 UTC
Red Hat Product Errata RHBA-2022:1523 0 None None None 2022-04-25 14:27:10 UTC

Description ffernand 2021-03-16 15:08:57 UTC
Using ovs/ovn master as well as openvswitch2.13-2.13.0-79.5.el8fdp.x86_64, I notice what while I am able to successfully change the rate attribute on nb and sb dbs, the update is not making its way to ofctl by the ovn-controller(s).

NOTE: **even "ovn-appctl -t ovn-controller recompute" doesn't "fix" it.**, which makes this issue a high priority.

The only work around at this point is to restart the ovn-controller, which can be a very disruptive approach.

Reference deployment taken from this document:
https://docs.google.com/document/d/1L7u_VPRSvSiD4wUMNAgTctagLq72ar7ZwtVvo2QG04o/edit?usp=sharing

[root@compute-1 ~]# rpm -qa | grep openvswitch2
openvswitch2.13-2.13.0-79.5.el8fdp.x86_64
network-scripts-openvswitch2.13-2.13.0-79.5.el8fdp.x86_64

[root@compute-1 ~]# sudo podman exec ovn_controller rpm -qa | grep ovn2.
ovn2.13-20.12.0-17.el8fdp.x86_64
ovn2.13-host-20.12.0-17.el8fdp.x86_64

looking at meter from controller
[root@controller-0 ~]# ovn-nbctl list meter
_uuid               : 10f37ada-576e-45a7-932b-bc1f3ff2e828
bands               : [e5528fca-0d33-4b7d-aa28-e524c889f570]
external_ids        : {}
fair                : []
name                : sgm
unit                : pktps
[root@controller-0 ~]# ovn-nbctl list meter-band ;  # north
_uuid               : e5528fca-0d33-4b7d-aa28-e524c889f570
action              : drop
burst_size          : 0
external_ids        : {}
rate                : 100
[root@controller-0 ~]# ovn-sbctl list meter-band   ; # SOUTH
_uuid               : b7f598a1-d207-4230-9ca5-3906e877042e
action              : drop
burst_size          : 0
rate                : 100

## looking at the meter at compute
[root@compute-1 ~]# sudo ovs-ofctl -OOpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=100    ; # CURRENTLY SET VALUE

## changing the meter 
[root@controller-0 ~]# MB=$(ovn-nbctl --bare --column _uuid list meter-band) ; echo $MB
e5528fca-0d33-4b7d-aa28-e524c889f570
[root@controller-0 ~]# ovn-nbctl set meter-band ${MB} rate=222
[root@controller-0 ~]# ovn-nbctl list meter
_uuid               : 10f37ada-576e-45a7-932b-bc1f3ff2e828
bands               : [e5528fca-0d33-4b7d-aa28-e524c889f570]
external_ids        : {}
fair                : []
name                : sgm
unit                : pktps
[root@controller-0 ~]# ovn-nbctl list meter-band
_uuid               : e5528fca-0d33-4b7d-aa28-e524c889f570
action              : drop
burst_size          : 0
external_ids        : {}
rate                : 222
[root@controller-0 ~]# ovn-sbctl list meter-band
_uuid               : f564c7af-64e6-4567-8c8f-b02c4ad99890
action              : drop
burst_size          : 0
rate                : 222

## compute node(s) never the get the memo
[root@compute-0 ~]# sudo ovs-ofctl -OOpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=100    ; # OLD VALUE

[root@compute-1 ~]# sudo ovs-ofctl -OOpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=100    ; # OLD VALUE

## I also tried:
sudo podman exec ovn_controller ovn-appctl -t ovn-controller recompute

## And on computes
ovs-vsctl set open . external_ids:ovn-monitor-all=true

but only this seemed to make it take effect:

[root@compute-0 ~]# systemctl restart tripleo_ovn_controller.service


[root@compute-0 ~]# sudo ovs-ofctl -OOpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=222   ; # EXPECTED value after ovn-controller restart

Comment 1 Mark Michelson 2021-09-24 13:48:51 UTC
Hi Flavio, this came up during OVN issue prioritization. Given the age of this bug report, do you know if this is still relevant?

Comment 2 Alex Katz 2021-09-29 06:44:02 UTC
Hello Mark,

Flavio is not working on this project anymore.
I can confirm that we are still facing that issue in OpenStack environments.

Comment 4 Andreas Karis 2022-03-03 11:16:17 UTC
Hi there,

Just as an FYI: We are running into this in ovn-kubernetes, too: https://bugzilla.redhat.com/show_bug.cgi?id=2051954#c11

- Andreas

Comment 5 lorenzo bianconi 2022-03-10 16:44:20 UTC
upstream fix: https://github.com/ovn-org/ovn/commit/885655e16e63384484af4607e43fa4c69a1089c5

Comment 8 Jianlin Shi 2022-04-14 08:31:51 UTC
tested with following script:

systemctl start openvswitch                                                                           
systemctl start ovn-northd                                                                            
ovn-nbctl set-connection ptcp:6641                                                                    
ovn-sbctl set-connection ptcp:6642                                                                    
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.177.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.177.25
systemctl restart ovn-controller                                                                      

ovn-nbctl ls-add ls1                                                                                  
ovn-nbctl lsp-add ls1 ls1p1                                                                           
ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1 2001::1"                             

ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1      

ip netns add ls1p1                     
ip link set ls1p1 netns ls1p1              
ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01                                       
ip netns exec ls1p1 ip link set ls1p1 up
ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1                                              
ip netns exec ls1p1 ip addr add 2001::1/64 dev ls1p1
ip netns exec ls1p1 ip route add default via 192.168.1.254 dev ls1p1                                  
ip netns exec ls1p1 ip -6 route add default via 2001::a dev ls1p1                                     

ovn-nbctl meter-add meter0 drop 50 pktps
ovn-nbctl --log --severity=alert --name=bz1939524 --meter=meter0 acl-add ls1 to-lport 1000 'icmp4' drop
ovn-nbctl --wait=hv sync                                                                              
ovs-ofctl -O OpenFlow15 dump-meters br-int                                                            
band_id=$(ovn-nbctl list meter_band | awk '/_uuid/{print $3}' )                                       
for meter in 30 20 10 -1 5                                                                            
do                                                                                                    
        ovn-nbctl set meter_band $band_id rate=$meter                                                 
                                          
        ovn-nbctl --wait=hv sync                                                                      
        ovs-ofctl -O OpenFlow15 dump-meters br-int                                                    
done 

reproduced on ovn-2021-21.12.0-32.el8:

+ ovn-nbctl set meter_band 4f49cde0-febe-4dac-88d4-8e34657b6fcb rate=30
+ ovn-nbctl --wait=hv sync                  
+ ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=                         
type=drop rate=50                                                                                     
+ for meter in 30 20 10 -1 5                  
+ ovn-nbctl set meter_band 4f49cde0-febe-4dac-88d4-8e34657b6fcb rate=20
+ ovn-nbctl --wait=hv sync                         
+ ovs-ofctl -O OpenFlow15 dump-meters br-int       
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):        
meter=1 pktps stats bands=                                                                                                                                                                                  
type=drop rate=50

<==== rate not changed

...


Verified on ovn-2021-21.12.0-42:

+ ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=50
++ ovn-nbctl list meter_band
++ awk '/_uuid/{print $3}'
+ band_id=74c0eec8-0b1e-4c2b-a0bc-48c3391a5908
+ for meter in 30 20 10 -1 5
+ ovn-nbctl set meter_band 74c0eec8-0b1e-4c2b-a0bc-48c3391a5908 rate=30
+ ovn-nbctl --wait=hv sync
+ ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=30
+ for meter in 30 20 10 -1 5
+ ovn-nbctl set meter_band 74c0eec8-0b1e-4c2b-a0bc-48c3391a5908 rate=20
+ ovn-nbctl --wait=hv sync
+ ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=20
+ for meter in 30 20 10 -1 5
+ ovn-nbctl set meter_band 74c0eec8-0b1e-4c2b-a0bc-48c3391a5908 rate=10
+ ovn-nbctl --wait=hv sync
+ ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=10
+ for meter in 30 20 10 -1 5
+ ovn-nbctl set meter_band 74c0eec8-0b1e-4c2b-a0bc-48c3391a5908 rate=-1
ovn-nbctl: constraint violation: -1 is not in the valid range 1 to 4294967295 (inclusive)
+ ovn-nbctl --wait=hv sync
+ ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=10
+ for meter in 30 20 10 -1 5
+ ovn-nbctl set meter_band 74c0eec8-0b1e-4c2b-a0bc-48c3391a5908 rate=5
+ ovn-nbctl --wait=hv sync
+ ovs-ofctl -O OpenFlow15 dump-meters br-int
OFPST_METER_CONFIG reply (OF1.5) (xid=0x2):
meter=1 pktps stats bands=
type=drop rate=5

<=== rate changed

+ exit 0
[root@wsfd-advnetlab16 bz1939524]# rpm -qa | grep -E "openvswitch2.15|ovn-2021"
ovn-2021-host-21.12.0-42.el8fdp.x86_64
ovn-2021-central-21.12.0-42.el8fdp.x86_64
ovn-2021-21.12.0-42.el8fdp.x86_64

Comment 10 errata-xmlrpc 2022-04-25 14:26:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn-2021 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1523


Note You need to log in before you can comment on or make changes to this bug.