Hide Forgot
Created attachment 1356729 [details] 10Gbit interface detected on the host with right speed Description of problem: We have hosts with 10Gbit. I am not so sure if it is a bug or luck of the future. When we define a network for the VMs with QoS > 1Gbit it is failing The VM with Nic from 10Gbit network saturating 100-400% during the high load. Expecting behavior: 1) add 10G interface to VM 1.1) set performance limits on the vm to 8Gbit not default 1Gbit. 2) define QoS to 80% of the host 10Gbit Nic 3) Profit 10Gbit interface on VM without polluting the logs with overloaded network. Version-Release number of selected component (if applicable): 4.1.7.6
Seems like you're seeing a dup of bug 1316568. But your definition is more concise. Yevgeny Zaspitsky 2016-07-04 13:27:06 IDT Looks like the overflow is on the engine side. It holds values as Integers, which are signed 32-bit in Java. The ul.m2 and rt.m2 values are multiplied by 8 before sending to VDSM, so that could be an additional limitation.
*** Bug 1316568 has been marked as a duplicate of this bug. ***
Moving back to post because referenced patch https://gerrit.ovirt.org/#/c/91245/ is still open.
(In reply to Sandro Bonazzola from comment #3) > Moving back to post because referenced patch > https://gerrit.ovirt.org/#/c/91245/ is still open. Now I assume it can move to MODIFIED.
Eitan ,Dan This fix has caused us a regression. The values that passed to libvirt now are not match what was configured in the UI. This was discovered during the first automation run on the new build on tier2. For example, in the UI i set: In Average=10 In Peak=1000 In burst=100 Out Average=10 Out Peak=1000 Out Burst=100 In the xml i get -> <bandwidth> <inbound average='1250' peak='125000' burst='102400'/> <outbound average='1250' peak='125000' burst='102400'/> </bandwidth> My question is : should i report new regression bug that will block this one or should i fail this bug? any how this can't be verified as it is now.
The conversion performed by engine was fixed and that's why you have a 'regression': Engine expects the user to insert rate values in mbps - megabits per sec (as declared in the UI and in engine config) but before the fix engine multiplied the rates provided by the user by 128. Since libvirt declares it expects kilobytes [1], there are two possibilities: 1. there was an error in the units declared by engine for user input. 2. the multiplication factor was wrong Since the documentation [2] declares that user input should be in mbps as well, option 1 is not valid. So this fix changes the multiplication factor to 125: from mbps to KB. Therefore: 10 -> 1250 1000 -> 125000 [1] https://libvirt.org/formatnetwork.html#elementQoS [2] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html-single/administration_guide/#Creating_a_Host_Network_Quality_of_Service_Entry
(In reply to eraviv from comment #6) > The conversion performed by engine was fixed and that's why you have a > 'regression': > > Engine expects the user to insert rate values in mbps - megabits per sec (as > declared in the UI and in engine config) but before the fix engine > multiplied the rates provided by the user by 128. > Since libvirt declares it expects kilobytes [1], there are two possibilities: > 1. there was an error in the units declared by engine for user input. > 2. the multiplication factor was wrong > Since the documentation [2] declares that user input should be in mbps as > well, option 1 is not valid. > > So this fix changes the multiplication factor to 125: from mbps to KB. > Therefore: > 10 -> 1250 > 1000 -> 125000 > > > [1] https://libvirt.org/formatnetwork.html#elementQoS > [2] > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ > html-single/administration_guide/ > #Creating_a_Host_Network_Quality_of_Service_Entry I see, Eitan, next time you doing such changes please let us know. You made a lot of changes here, some are affecting our automation tests that expect to specific values. This change was not even part of the origin bugs reports. As i look more carefully now, this fix includes multiple changes both for host and VMs QoS, i was not aware of it and this require more testing effort from me. This bug should have been split to several bugs BTW. And i think that summary should changed here as well. So, the things to test here are(notes for my self): - Enable 10Gb host QoS on both engine and vdsm, verify that vdsm except the values and report them - Test 10Gb on hosts - Enable 10Gb on VM QoS - Test 10Gb on Vms - Test new bw reported in xml and change automation test for VM QoS.
As well i see that you changed the default engine-config values to -> MaxPeakNetworkQoSValue: 34359 version: general MaxAverageNetworkQoSValue: 17179 version: general As i wrote in previous comment, this should have been split to at least 4 different bugs. Next time please do.
The fix for this bug contains several changes: 1. Set default MaxPeakNetworkQoSValue in engine-config to 34359 Megabits per sec - the max. QoS values supported by traffic control (see man tc #PARAMETERS). This change will allow the user to set higher bit rates in engine. 2. Set default MaxAverageNetworkQoSValue in engine-config to half the peak (as was half before the change) - namely 17179 Megabits per sec. 3. Correct the conversion factor employed by engine to convert user input of VM Qos rates from Megabits per sec to Kilobytes per sec - the value required by libvirt. conversion factor was: 128. now: 125 (1 Megabit = 125 KiloByte) 4. Enable engine to process values larger than java#Integer.MAX_INT. Before the fix this was a limitation that caused engine to send negative values to vdsm as Host QoS rates. 5. Enable engine db to store values larger than postgres#SMALL_INT for VM QoS. Before the fix this was a limitation that prevented engine from storing large values of VM QoS. Notes: 1. The fix for both VM QoS and Host QoS was executed under one bug because MaxPeakNetworkQoSValue and MaxAverageNetworkQoSValue influence both VM QoS and Host QoS and therefore changing them without solving all the above issues for both VM and Host QoS would have created regressions. 2. Verfications I performed for this bug during development: - setting large values within the 34359 Mbps limit is possible for both types of QoS without any errors in engine or vdsm. - making sure tc on the host reflects the values configured by engine (tc class show dev eth0) - making sure libvirt accepts the bandwidth values passed to it be engine (dumpxml#bandwidth) and powers up the VM with them. If not, a traffic control error is returned to engine (engine.log). - setting higher values than 34359 Mbps invokes errors.
Ok, so based on this -> VM QoS UI - It's now possible to set 34359 Mbps for Peak and 17179 Megabits for Average. Host QoS UI - It's now possible to set 34359 Mbps for Peak and 17179 Megabits for Average. Note that in host QoS, there is validation for 17179 Megabits for peak, with the new default values, not like on VM QoS. If we want to set max in the Peak, we need to update the Average to max as well via engine-config. Host QoS - "hostQos": { "out": { "rt": { "m1": 0, "d": 0, "m2": 17179000000 }, "ul": { "m1": 0, "d": 0, "m2": 34359000000 }, "ls": { "m1": 0, "d": 0, "m2": 80 [root@orchid-vds1 ~]# tc class show dev enp4s0 class hfsc 1389: root class hfsc 1389:1388 parent 1389: leaf 1388: rt m1 0bit d 0us m2 17179Mbit ls m1 0bit d 0us m2 640bit ul m1 0bit d 0us m2 34359Mbit "stp": "off", "hostQos": { "out": { "rt": { "m1": 0, "d": 0, "m2": 10000000000 }, "ul": { "m1": 0, "d": 0, "m2": 10000000000 }, "ls": { "m1": 0, "d": 0, "m2": 80 [root@orchid-vds1 ~]# tc class show dev enp4s0 class hfsc 1389: root class hfsc 1389:1388 parent 1389: leaf 1388: rt m1 0bit d 0us m2 10Gbit ls m1 0bit d 0us m2 640bit ul m1 0bit d 0us m2 10Gbit VM Host - conversion factor was: 128. now: 125 <bandwidth> <inbound average='1250000' peak='1875000' burst='1024000'/> <outbound average='1250000' peak='1875000' burst='1024000'/> </bandwidth> <bandwidth> <inbound average='2147375' peak='4294875' burst='102400'/> <outbound average='2147375' peak='4294875' burst='102400'/> </bandwidth> - burst still reported wrong, will be handled in BZ 1580285 Verified on - 4.2.4.2-0.1.el7_3 and vdsm-4.20.30-1.el7ev.x86_64
This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.
*** Bug 1600248 has been marked as a duplicate of this bug. ***
*** Bug 1464047 has been marked as a duplicate of this bug. ***