Description of problem: Currently below command is applicable only to normal ports which has phynet attachment (localnet port) and not to ports with representator ovn-nbctl set Logical_Switch_Port $portname options:qos_min_rate=2000000 In order support some of the customers requirement (min QoS hw-offload) we need to extend this to support for setting this to ports with representators also. Example: We need to be able to set something as below. ovn-nbctl set Logical_Switch_Port sw0-port2 options:qos_min_rate=9000000000 In OVS the external-ids iface-id is set as below and the representator port is attached to the OVS bridge. ovs-vsctl add-port br-int enp4s0f0_1 -- set Interface enp4s0f0_1 external-ids:iface-id=sw0-port2 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: Remote node 1 ------- |--------VF 1 --------|------------| |-----link 1 - 10G -----vlan------| S | |--------VF 2 --------| | |-----link 2 - 10G -----vlan------| W | |--------VF 3 --------| | | I | |----- 25 G-----[Compute Node, DUT | Remote node 2 | T | | has Mellanox NIC | |-----link 3 -10G ------vlan-----| C | | Running OVN and OVS | -- H-- ------------------------------------- On the DUT we have a mellanox NIC and we have created VFs. From the OVN we have created 3 logical ports and mapped them to 3 different VFs We have created 3 separate domains 5.5.5.X , 6.6.6.X and 7.7.7.X and each of them has an iperf server running on them to which from the compute/DUT we try to send and receive data using iperf clients. Initially when NO QOS is applied almost all 3 will be getting Transfer rates around 7.9 Gbits/sec... When we set a min qos on one of the logical ports then expectation is on the VF corresponding to the Logical port sw0-port2 the min transfer rates will be around 9 Gbits/sec ovn-nbctl set Logical_Switch_Port sw0-port2 options:qos_min_rate=9000000000 But as of now there is no effect We would want to have support in OVN by which we can set the min_tx_rate and also have hw-offload working with it. For this we need to have support in OVN. (possibly with the commands like below) ovn-nbctl set Logical_Switch_Port sw0-port2 options:qos_min_rate=9000000000 In OVS the external-ids iface-id is set as below and the representator port is attached to the OVS bridge. ovs-vsctl add-port br-int enp4s0f0_1 -- set Interface enp4s0f0_1 external-ids:iface-id=sw0-port2 Like mentioned earlier at the beginning currently we are handling this qos_min_rate for normal ports (only with phynet attachment (localnet port)). For the other types of ports currently there is no action taken except that it is set to the DB. Actual results: For port with representator ports currently no action taken when qos_min_rate is set. Expected results: Need to handle qos_min_rate for port with representator. Additional info: Note: For this feature to work fully end to end driver support is also needed. Hence we have raised a BZ on NVIDIA. Below is the BZ for reference --> https://bugzilla.redhat.com/show_bug.cgi?id=2105263 . But this shouldnt be a blocker to add support in OVN as we can verify by querying tc show command and see if we have set to representator ports expected qos_min_rate.
Hi, thanks for the detailed report. I had a look at the OVN code, and from what I could tell, there is no restriction in the OVN code that applies qos_min_rate only to localnet ports. It should also apply to virtual ports, container ports, and ports with no type specified (i.e. a VIF). > But this shouldnt be a blocker to add support in OVN as we can verify by querying tc show command and see if we have set to representator ports expected qos_min_rate. Does this mean that when you try to apply qos_min_rate to a representator port, `tc show dev enp4s0f0_1` does not show that the setting has been applied? If the setting is present here, then it means that OVN attempted to create the qdisc as requested and the min_rate is not being applied due to an issue at a different layer. If it's not there, then it means that OVN is not creating the qdisc and we need to make an update in OVN so the qdisc is created. can you clarify which is true at present? And just for reference, what is the "type" column of the sw0-port2 logical switch port set to? Thanks.
Hi Mark, Thanks for responding. For your questions I have put answers below also later I have explained why the setting is not getting applied. Please read through. Does this mean that when you try to apply qos_min_rate to a representator port, `tc show dev enp4s0f0_1` does not show that the setting has been applied? YES. It doesnt show it. (When I tried setting to enp4s0f0_1 it doesnt show in tc show dev enp4s0f0_1.) "type" of sw0-port2 is "" as below type : "" I had written a mail to Ihar and Rodolfo where they had confirmed that currently it is expected as currently this is supported only to normal ports and not to ports with representator. Further when I tried checking the OVN code for my understanding. From the higher level it "seems" supported but please see the call sequence below which I traced on why it doesnt set. I do see the below code in function consider_vif_lport_ where it tries to get the qos params which will be later set. if (b_lport->lbinding->iface && qos_map && b_ctx_in->ovs_idl_txn) { get_qos_params(pb, qos_map); } But here what I observed was it is not able to get the params as qos_map was NULL. So, if we go back to check where this gets filled.. We need to take 2 steps back binding_run case LP_VIF: consider_vif_lport |---->consider_vif_lport_ i.e. we need to look at binding_run function consider_vif_lport(pb, b_ctx_in, b_ctx_out, NULL, qos_map_ptr); We need to see how the last param qos_map_ptr is getting filled. In the same binding_run function at the beginning we have below code struct hmap *qos_map_ptr = !sset_is_empty(b_ctx_out->egress_ifaces) ? &qos_map : NULL; In my case since the b_ctx_out->egress_ifaces was empty qos_map_ptr was getting set to NULL So, the next thing was to see where is b_ctx_out->egress_ifaces getting updated So I started looking into the code where this gets filled. It is getting filled in 2 places 1) In function add_localnet_egress_interface_mappings as below bool is_egress_iface = smap_get_bool(&iface_rec->external_ids, "ovn-egress-iface", false); if (!is_egress_iface) { continue; } sset_add(egress_ifaces, iface_rec->name); As I have mentioned in my earlier description of the bug this is specific to localnet So, this never gets called for our case. 2) In function build_local_bindings its as below /* Check if this is a tunnel interface. */ if (smap_get(&iface_rec->options, "remote_ip")) { const char *tunnel_iface = smap_get(&iface_rec->status, "tunnel_egress_iface"); if (tunnel_iface) { sset_add(b_ctx_out->egress_ifaces, tunnel_iface); } } Here again, since our interface is not a tunnel interface this doesnt get hit either. So, if you see for my case 'b_ctx_out->egress_ifaces' remains empty and hence the qos doesnt gets picked up. Hence never gets set also. Lastly, iF you check the BZ which I have given as reference in bug description is assigned to Driver in that ofcourse I had to do some hacks in OVN by removing few checks and make it set. And hence I have put this statement there "I tried doing changes in OVN to set QoS to the representator port. Due to changes, I can see it is setting to 'tc' ." Hope I am able to explain it. Please feel free to ping me if you need to discuss more. Thanks & Regards, Abhiram R N
Thank you very much for the explanation. I'm marking triage complete on the issue since I think we have everything we need to address this.
upstream patch: https://patchwork.ozlabs.org/project/ovn/patch/6799f37ba7e7e1c827d2aa85c9bb3df1c6ea864e.1667567227.git.lorenzo.bianconi@redhat.com/
ovn23.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2208426
verified on version: # rpm -qa|grep ovn ovn23.06-central-23.06.1-60.el8fdp.x86_64 ovn23.06-23.06.1-60.el8fdp.x86_64 ovn23.06-host-23.06.1-60.el8fdp.x86_64 # ovn-nbctl show switch 83b7ecd7-2afc-4e5f-86d4-657972509044 (ls) port vm1 addresses: ["00:00:00:00:00:01 42.42.42.2/24 2001:1::2/64"] port ln_p1 type: localnet addresses: ["unknown"] port vm2 addresses: ["00:00:00:00:00:02 42.42.42.3/24 2001:1::3/64"] [root@dell-per730-19 ~]# ovs-vsctl show 71fe32f7-b82d-4ac9-85aa-1fae6a7cf3e7 Bridge nat_test Port nat_test Interface nat_test type: internal Port enp5s0f1 Interface enp5s0f1 Port patch-ln_p1-to-br-int Interface patch-ln_p1-to-br-int type: patch options: {peer=patch-br-int-to-ln_p1} Bridge br-int fail_mode: secure datapath_type: system Port patch-br-int-to-ln_p1 Interface patch-br-int-to-ln_p1 type: patch options: {peer=patch-ln_p1-to-br-int} Port br-int Interface br-int type: internal Port vm1 Interface vm1 type: internal Port ovn-hv0-0 Interface ovn-hv0-0 type: geneve options: {csum="true", key=flow, remote_ip="20.0.9.26"} ovs_version: "3.1.4" then add qos on vm1. ovn-nbctl set Logical_Switch_Port vm1 options:qos_max_rate=2000000 ovn-nbctl set Logical_Switch_Port vm1 options:qos_min_rate=1000000 on unfixed version: check qdisc: #tc class show dev enp5s0f1|grep 'rate 1Mbit ceil 2Mbit' # on fixed version: #tc class show dev enp5s0f1|grep 'rate 1Mbit ceil 2Mbit' #class htb 1:2 parent 1:fffe prio 0 rate 1Mbit ceil 2Mbit burst 1564b cburst 1564b set verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn23.06 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:0388