Bug 1574771 - [RFE] Provide a friendly interface (UI + REST) for virtio multiqueue network interfaces
Summary: [RFE] Provide a friendly interface (UI + REST) for virtio multiqueue network ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: unspecified
Hardware: All
OS: All
high
high
Target Milestone: ovirt-4.2.5
: ---
Assignee: Alona Kaplan
QA Contact: Michael Burman
URL:
Whiteboard:
: 1309274 (view as bug list)
Depends On: 1588643
Blocks: 1529493
TreeView+ depends on / blocked
 
Reported: 2018-05-04 02:21 UTC by Jamie Bainbridge
Modified: 2020-08-03 15:38 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
This release enables multiple queues and creates up to four queues per vNIC depending on the number of available vCPUs.
Clone Of:
Environment:
Last Closed: 2018-11-28 12:36:33 UTC
oVirt Team: Network
Target Upstream Version:
lsvaty: testing_plan_complete-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1506903 0 None None None 2018-06-04 02:38:19 UTC
oVirt gerrit 92007 0 'None' MERGED engine: Add multiQueuesEnabled to VmBase 2021-02-14 09:34:11 UTC
oVirt gerrit 92009 0 'None' MERGED engine: Update libvirtVmXml to consider vmBase.multiQueuesEnabled attribute 2021-02-14 09:34:11 UTC
oVirt gerrit 92149 0 'None' MERGED webadmin: Add 'Multi Queues Enabled' to vm dialog 2021-02-14 09:34:11 UTC
oVirt gerrit 92205 0 'None' MERGED engine: Add "MultiQueuesEnabled" to ovf reader and writer 2021-02-14 09:34:11 UTC
oVirt gerrit 92307 0 'None' MERGED api-model: Add 'Multi Queues Enabled' to the relevant mappers 2021-02-14 09:34:11 UTC
oVirt gerrit 92318 0 'None' MERGED restapi: Add 'Multi Queues Enabled' to the relevant mappers 2021-02-14 09:34:12 UTC
oVirt gerrit 92574 0 'None' MERGED api-model: Add 'Multi Queues Enabled' to the relevant mappers 2021-02-14 09:34:13 UTC
oVirt gerrit 92587 0 'None' MERGED restapi: Update to model 4.3.13 2021-02-14 09:34:13 UTC
oVirt gerrit 92588 0 'None' MERGED engine: Add multiQueuesEnabled to VmBase 2021-02-14 09:34:13 UTC
oVirt gerrit 92589 0 'None' MERGED engine: Introduce algorithm for calculating how many queues asign per vnic 2021-02-14 09:34:13 UTC
oVirt gerrit 92590 0 'None' MERGED engine: Update libvirtVmXml to consider vmBase.multiQueuesEnabled attribute 2021-02-14 09:34:12 UTC
oVirt gerrit 92591 0 'None' MERGED engine: Add "MultiQueuesEnabled" to ovf reader and writer 2021-02-14 09:34:13 UTC
oVirt gerrit 92592 0 'None' MERGED webadmin: Add 'Multi Queues Enabled' to vm dialog 2021-02-14 09:34:13 UTC
oVirt gerrit 92593 0 'None' MERGED restapi: Add 'Multi Queues Enabled' to the relevant mappers 2021-02-14 09:34:14 UTC
oVirt gerrit 92595 0 'None' MERGED restapi: Update to model 4.2.34 2021-02-14 09:34:14 UTC
oVirt gerrit 92626 0 'None' MERGED engine: Specify non custom queues only if 'vmStatic.multiQueuesEnabled' 2021-02-14 09:34:14 UTC
oVirt gerrit 92627 0 'None' ABANDONED engine: Send 'queues' to the vdsm as a 'custom' value 2021-02-14 09:34:15 UTC
oVirt gerrit 92629 0 'None' MERGED engine: Specify non custom queues only if 'vmStatic.multiQueuesEnabled' 2021-02-14 09:34:15 UTC
oVirt gerrit 92656 0 'None' MERGED engine: Not reporting queues=1 since it is the default. 2021-02-14 09:34:14 UTC
oVirt gerrit 92706 0 'None' MERGED engine: Not reporting queues=1 since it is the default. 2021-02-14 09:34:15 UTC
oVirt gerrit 92712 0 'None' MERGED engine: Remove the 'isPlugged' check when adding queues 2021-02-14 09:34:15 UTC

Description Jamie Bainbridge 2018-05-04 02:21:23 UTC
1. Proposed title of this feature request

Provide a GUI interface for virtio multiqueue network interfaces

3. What is the nature and description of the request?

RHEV gained support for configuring virtio-multiqueue for NICs on Bug 821493.

However, the feature was implemented as CustomDeviceProperties which the user needs to enter on the RHEV-M's commandline with engine-config.

This is not ideal.

Please add a GUI tickbox to turn the feature on, and a text field to define the number of queues. If the user does not enter a number, default the number of queues to the number of VCPUs the VM has.

4. Why does the customer need this? (List the business requirements here)

Easier configuration of RHEV for customers.

5. How would the customer like to achieve this? (List the functional requirements here)

RHEV devel to add a GUI option for configuration of virtio-net mq

6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.

Create a RHEV VM with multiple VCPUs and one virtio NIC
Boot the VM and observe the number of queues created in /proc/interrupts

Power off the VM
Tick the box for virtio-net multiqueue
Boot the VM and observe the number of queues created in /proc/interrupts

Power off the VM
Enter the number of virtio-net multiqueue queues
Boot the VM and observe the number of queues created in /proc/interrupts

7. Is there already an existing RFE upstream or in Red Hat Bugzilla?

No

8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)?

No

9. Is the sales team involved in this request and do they have any additional input?

No

10. List any affected packages or components.

ovirt-engine

11. Would the customer be able to assist in testing this functionality if implemented?

No

Comment 14 Alona Kaplan 2018-06-03 13:16:28 UTC
A new "multi queues" checkbox will be added to the Add/Edit Vm Resource Allocation tab.
For "High Performance" vms "multi queues" will be true by default.

If multi queues is enabled, the algorithm for specifying how many queues each virtio vnic will have will be as follows -
1. If the number of vCpus is lower than the number of virtio vnics an audit log warning saying multi queues cannot be applied, will be displayed.
2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
3. Dividing the queues between the virtio vnics. All the vnics will get at least 1 queue, some random vnics will get 2 queues (depending on the number_of_queues).

* CustomDeviceProperties will override the "multi queues" algorithm.

Comment 15 Yaniv Lavi 2018-06-06 07:47:56 UTC
*** Bug 1309274 has been marked as a duplicate of this bug. ***

Comment 16 Yaniv Lavi 2018-06-11 13:37:33 UTC
(In reply to Alona Kaplan from comment #14)
> A new "multi queues" checkbox will be added to the Add/Edit Vm Resource
> Allocation tab.
> For "High Performance" vms "multi queues" will be true by default.
> 
> If multi queues is enabled, the algorithm for specifying how many queues
> each virtio vnic will have will be as follows -
> 1. If the number of vCpus is lower than the number of virtio vnics an audit
> log warning saying multi queues cannot be applied, will be displayed.
> 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> 3. Dividing the queues between the virtio vnics. All the vnics will get at
> least 1 queue, some random vnics will get 2 queues (depending on the
> number_of_queues).
> 
> * CustomDeviceProperties will override the "multi queues" algorithm.

There should be a max of 4 queues for up to 100 cores or 8 queues for above that.

Comment 17 Alona Kaplan 2018-06-14 12:31:37 UTC
(In reply to Yaniv Lavi from comment #16)
> (In reply to Alona Kaplan from comment #14)
> > A new "multi queues" checkbox will be added to the Add/Edit Vm Resource
> > Allocation tab.
> > For "High Performance" vms "multi queues" will be true by default.
> > 
> > If multi queues is enabled, the algorithm for specifying how many queues
> > each virtio vnic will have will be as follows -
> > 1. If the number of vCpus is lower than the number of virtio vnics an audit
> > log warning saying multi queues cannot be applied, will be displayed.
> > 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> > 3. Dividing the queues between the virtio vnics. All the vnics will get at
> > least 1 queue, some random vnics will get 2 queues (depending on the
> > number_of_queues).
> > 
> > * CustomDeviceProperties will override the "multi queues" algorithm.
> 
> There should be a max of 4 queues for up to 100 cores or 8 queues for above
> that.

Updated the algorithm to the following -

If the number of vCpus is larger than 100, each vnic will get 8 queues. Otherwise, each vnic will get 4 queues. If there are no enough vcpus, each vnic will get vcpus/num_of_vnics queues. The extra queues will be devided randomly between the vnics. Each vnic will have a least one queue. Please notice the hotplug algorithm does best effort to find out the optimal number of queues to assign to the hot plugged nic. But since we don't persist the number of queues assigned per each vnic, there may be mistakes. Only re-running the vm promises the best allocation of the queues.

Comment 18 Yaniv Lavi 2018-06-14 12:59:15 UTC
Can you please ack the policy in comment #17 based on your testing?

Comment 19 Bob Sibley 2018-06-14 14:31:30 UTC
(In reply to Alona Kaplan from comment #17)
> (In reply to Yaniv Lavi from comment #16)
> > (In reply to Alona Kaplan from comment #14)
> > > A new "multi queues" checkbox will be added to the Add/Edit Vm Resource
> > > Allocation tab.
> > > For "High Performance" vms "multi queues" will be true by default.
> > > 
> > > If multi queues is enabled, the algorithm for specifying how many queues
> > > each virtio vnic will have will be as follows -
> > > 1. If the number of vCpus is lower than the number of virtio vnics an audit
> > > log warning saying multi queues cannot be applied, will be displayed.
> > > 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> > > 3. Dividing the queues between the virtio vnics. All the vnics will get at
> > > least 1 queue, some random vnics will get 2 queues (depending on the
> > > number_of_queues).
> > > 
> > > * CustomDeviceProperties will override the "multi queues" algorithm.
> > 
> > There should be a max of 4 queues for up to 100 cores or 8 queues for above
> > that.
> 
> Updated the algorithm to the following -
> 
> If the number of vCpus is larger than 100, each vnic will get 8 queues.
> Otherwise, each vnic will get 4 queues. If there are no enough vcpus, each
> vnic will get vcpus/num_of_vnics queues. The extra queues will be devided
> randomly between the vnics. Each vnic will have a least one queue. Please
> notice the hotplug algorithm does best effort to find out the optimal number
> of queues to assign to the hot plugged nic. But since we don't persist the
> number of queues assigned per each vnic, there may be mistakes. Only
> re-running the vm promises the best allocation of the queues.

AlL of our testing was done with 8 queues (10Gb, 25Gb, 100Gb)
 
This new algorithm will be available to users as out of the box tuning correct?
If so this  should work fine for the standard customer wanting more throughput that multi-queue can provide. 

The more aggressive customer wanting higher que depth can still change muti-queue through CustomDeviceProperties, correct?

Comment 20 Jamie Bainbridge 2018-06-14 22:16:59 UTC
(In reply to Bob Sibley from comment #19)
> This new algorithm will be available to users as out of the box tuning
> correct?

Yes, in the High Performance VM type.

> If so this should work fine for the standard customer wanting more
> throughput that multi-queue can provide.

That's the idea :)

> The more aggressive customer wanting higher que depth can still change
> muti-queue through CustomDeviceProperties, correct?

Yes, mentioned above in Comment 14.

Comment 21 Yaniv Kaul 2018-06-15 08:15:32 UTC
(In reply to Jamie Bainbridge from comment #20)
> (In reply to Bob Sibley from comment #19)
> > This new algorithm will be available to users as out of the box tuning
> > correct?
> 
> Yes, in the High Performance VM type.

And Server VM type as well.

Comment 22 Alona Kaplan 2018-06-17 11:48:37 UTC
(In reply to Yaniv Kaul from comment #21)
> (In reply to Jamie Bainbridge from comment #20)
> > (In reply to Bob Sibley from comment #19)
> > > This new algorithm will be available to users as out of the box tuning
> > > correct?
> > 
> > Yes, in the High Performance VM type.
> 
> And Server VM type as well.

Why not setting multi-queues as default for all types of vms?
Can it harm desktop vms?

Comment 23 Yaniv Kaul 2018-06-17 12:21:43 UTC
(In reply to Alona Kaplan from comment #22)
> (In reply to Yaniv Kaul from comment #21)
> > (In reply to Jamie Bainbridge from comment #20)
> > > (In reply to Bob Sibley from comment #19)
> > > > This new algorithm will be available to users as out of the box tuning
> > > > correct?
> > > 
> > > Yes, in the High Performance VM type.
> > 
> > And Server VM type as well.
> 
> Why not setting multi-queues as default for all types of vms?
> Can it harm desktop vms?

You tell me. My concern is that for VDI use case, the density might be a factor - how much additional memory and CPU do the queues consume (and does VDI really need that higher perform at some cost) ?

Comment 24 Jamie Bainbridge 2018-06-17 22:52:17 UTC
I doubt there would be a problem enabling mq for all VMs but I have not tested common use cases.

Memory usage is negligible, just a few small kernel data structures for each additional queue.

Theoretically, mq would be *better* for CPU usage as it allows network processing load to spread more evenly across CPUs instead of one queue all being handled by one core. There is no additional CPU usage, the system receives the amount of traffic it receives, mq just spreads that CPU usage around across more cores.

The only reason I didn't suggest it is that one queue tends to be "good enough" for most common desktop and even mid-power server usage.

I'd be happy to see mq used on all RHV VMs.

Perhaps ask QE to do a sanity check on a moderately busy VDI. If no negative latency or usage impact is seen, make it the default for all.

Comment 25 Jamie Bainbridge 2018-06-20 05:11:52 UTC
(In reply to Alona Kaplan from comment #14)
> 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)

This is wrong and not what was discussed prior. This is not what we want. Don't do this.

(In reply to Yaniv Lavi from comment #16)
> There should be a max of 4 queues for up to 100 cores or 8 queues for above
> that.

Having though about this more, I suggest not to include this logic.

A 100-core VM is well past the "one size fits most" sort of usage this rule is intended to hit.

Someone with a 100-core VM is likely going to do significant work with physical CPU pinning, virtual NUMA nodes, and a significant amount of work inside the VM and their application to properly manage CPU usage. If such an advanced user wants to add more net queues, that's a very minor and simple step compared to everything else they have to do.


Just keep the rule simple and relevant to majority usage:

 queues = min(num_of_vCpus, 4)

Comment 26 Alona Kaplan 2018-06-20 06:21:50 UTC
(In reply to Jamie Bainbridge from comment #25)
> (In reply to Alona Kaplan from comment #14)
> > 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> 

The algorithm is comment #14 is outdated. Please refer to the algorithm is comment #17.

> This is wrong and not what was discussed prior. This is not what we want.
> Don't do this.
> 
> (In reply to Yaniv Lavi from comment #16)
> > There should be a max of 4 queues for up to 100 cores or 8 queues for above
> > that.
> 
> Having though about this more, I suggest not to include this logic.
> 
> A 100-core VM is well past the "one size fits most" sort of usage this rule
> is intended to hit.
> 
> Someone with a 100-core VM is likely going to do significant work with
> physical CPU pinning, virtual NUMA nodes, and a significant amount of work
> inside the VM and their application to properly manage CPU usage. If such an
> advanced user wants to add more net queues, that's a very minor and simple
> step compared to everything else they have to do.
> 
> 
> Just keep the rule simple and relevant to majority usage:
> 
>  queues = min(num_of_vCpus, 4)

What do you mean by 'queues'? The number of queues that each vnic should get or the total amount of queues that should be divided between the vnics?

Are you ok with this algorithm?
Each vnic will get 4 queues. If there are no enough vcpus, each vnic will get vcpus/num_of_vnics queues. The extra queues will be divided randomly between the vnics. Each vnic will have at least one queue.

For example -
1. numOfVnics = 5, numOfCpus = 12
nic1=3queues , nic2==3queues, nic3=2queues, nic4=2queues, nic5=2queues

2. numOfVnics = 5, numOfCpus = 40
nic1=4queues , nic2==4queues, nic3=4queues, nic4=4queues, nic5=4queues

Comment 27 Germano Veit Michel 2018-06-20 06:25:04 UTC
Hi Alone,

(In reply to Alona Kaplan from comment #26)
> For example -
> 1. numOfVnics = 5, numOfCpus = 12
> nic1=3queues , nic2==3queues, nic3=2queues, nic4=2queues, nic5=2queues

As in the gerrit comment, I think this should have 4 queues on every vNIC.

Lets see if Jamie agrees.

Comment 29 Jamie Bainbridge 2018-06-20 06:53:25 UTC
(In reply to Alona Kaplan from comment #26)
> >  queues = min(num_of_vCpus, 4)
> 
> What do you mean by 'queues'? The number of queues that each vnic should get
> or the total amount of queues that should be divided between the vnics?

The number of queues that each NIC should get.

The number of total NICs/queues isn't a concern. It's fine if some NICs "double up" and use the same cores. It is unlikely that two queues would become so busy they would pin the CPU on softirq processing, and if that does happen then the irqbalance service will move the IRQ handling for one or both queues, so CPU usage will reduce soon. There is no need to do complex algorithms to create different numbers of queues based on total core count.

Again we are aiming for "most common" customer usage. In my experience it is very rare for customers to use more than three interfaces within a VM, most have just one NIC.

> Are you ok with this algorithm?

No, far too complex.

Following the algo above:

 num_of_nic_queues = min(num_of_vCpus, 4)

Examples:

1. numOfVnics = 5, numOfCpus = 12

Each NIC = 4 queues.
Total queues = 20.

2. numOfVnics = 5, numOfCpus = 40

Any NIC = 4 queues.
Total queues = 20.

Comment 30 RHV bug bot 2018-07-02 15:34:18 UTC
WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops@redhat.comINFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops@redhat.com

Comment 31 Michael Burman 2018-07-05 12:41:49 UTC
Verified on - 4.2.5.1_SNAPSHOT-71.g54dde01.0.scratch.master.el7ev

Comment 32 RHV bug bot 2018-07-24 12:27:01 UTC
WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops@redhat.comINFO: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops@redhat.com

Comment 35 Michal Skrivanek 2018-08-23 04:53:15 UTC
Why was this bug not cloned to zstream?

IIUC we went with up to 4 mq per NIC, correct? Just making sure since several comments discuss no upper bound and that the amount is not a concern. Reading e.g. https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html it doesn’t seem to be true

Comment 36 Jamie Bainbridge 2018-08-23 05:57:38 UTC
(In reply to Michal Skrivanek from comment #35)
> IIUC we went with up to 4 mq per NIC, correct?

Correct. Per-NIC we should calculating: min(num_of_vCpus, 4)

With common customer usage, where at most we see 3 virtio NICs, we won't come anywhere near a limit of 200 IRQs. A customer would need 40 virtio NICs to hit that limitation. (40+(40*4)=200)

Comment 39 Germano Veit Michel 2018-08-30 23:21:28 UTC
Restoring Michal's needinfo (comment #35)

Comment 40 Alona Kaplan 2018-09-02 06:43:32 UTC
(In reply to Michal Skrivanek from comment #35)
> Why was this bug not cloned to zstream?
> 
> IIUC we went with up to 4 mq per NIC, correct? Just making sure since
> several comments discuss no upper bound and that the amount is not a
> concern. Reading e.g.
> https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/
> libvirt-virtiomq.html it doesn’t seem to be true

Yes, the amount of the queues will be up to 4.

Comment 41 Emma Heftman 2018-09-04 12:21:24 UTC
As this bug hasn't been closed yet, can you please change the name of the field in the UI to Multi Queues Enabled with a capital E.

It needs to be consistent with the other fields.

Comment 42 Emma Heftman 2018-09-04 13:04:24 UTC
Dan, do we still need/want to document the customer property now that we have this field?

Is there any point to changing the value? From what I understood from Alona, virt  testing did not produce better performance by changing this value.

Comment 43 Sandro Bonazzola 2018-11-28 12:36:33 UTC
This bugzilla is included in oVirt 4.2.5 release, published on July 30th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.5 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.