Bug 1574771

A new "multi queues" checkbox will be added to the Add/Edit Vm Resource Allocation tab.
For "High Performance" vms "multi queues" will be true by default.

If multi queues is enabled, the algorithm for specifying how many queues each virtio vnic will have will be as follows -
1. If the number of vCpus is lower than the number of virtio vnics an audit log warning saying multi queues cannot be applied, will be displayed.
2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
3. Dividing the queues between the virtio vnics. All the vnics will get at least 1 queue, some random vnics will get 2 queues (depending on the number_of_queues).

* CustomDeviceProperties will override the "multi queues" algorithm.

*** Bug 1309274 has been marked as a duplicate of this bug. ***

(In reply to Alona Kaplan from comment #14)
> A new "multi queues" checkbox will be added to the Add/Edit Vm Resource
> Allocation tab.
> For "High Performance" vms "multi queues" will be true by default.
> 
> If multi queues is enabled, the algorithm for specifying how many queues
> each virtio vnic will have will be as follows -
> 1. If the number of vCpus is lower than the number of virtio vnics an audit
> log warning saying multi queues cannot be applied, will be displayed.
> 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> 3. Dividing the queues between the virtio vnics. All the vnics will get at
> least 1 queue, some random vnics will get 2 queues (depending on the
> number_of_queues).
> 
> * CustomDeviceProperties will override the "multi queues" algorithm.

There should be a max of 4 queues for up to 100 cores or 8 queues for above that.

(In reply to Yaniv Lavi from comment #16)
> (In reply to Alona Kaplan from comment #14)
> > A new "multi queues" checkbox will be added to the Add/Edit Vm Resource
> > Allocation tab.
> > For "High Performance" vms "multi queues" will be true by default.
> > 
> > If multi queues is enabled, the algorithm for specifying how many queues
> > each virtio vnic will have will be as follows -
> > 1. If the number of vCpus is lower than the number of virtio vnics an audit
> > log warning saying multi queues cannot be applied, will be displayed.
> > 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> > 3. Dividing the queues between the virtio vnics. All the vnics will get at
> > least 1 queue, some random vnics will get 2 queues (depending on the
> > number_of_queues).
> > 
> > * CustomDeviceProperties will override the "multi queues" algorithm.
> 
> There should be a max of 4 queues for up to 100 cores or 8 queues for above
> that.

Updated the algorithm to the following -

If the number of vCpus is larger than 100, each vnic will get 8 queues. Otherwise, each vnic will get 4 queues. If there are no enough vcpus, each vnic will get vcpus/num_of_vnics queues. The extra queues will be devided randomly between the vnics. Each vnic will have a least one queue. Please notice the hotplug algorithm does best effort to find out the optimal number of queues to assign to the hot plugged nic. But since we don't persist the number of queues assigned per each vnic, there may be mistakes. Only re-running the vm promises the best allocation of the queues.

Can you please ack the policy in comment #17 based on your testing?

(In reply to Alona Kaplan from comment #17)
> (In reply to Yaniv Lavi from comment #16)
> > (In reply to Alona Kaplan from comment #14)
> > > A new "multi queues" checkbox will be added to the Add/Edit Vm Resource
> > > Allocation tab.
> > > For "High Performance" vms "multi queues" will be true by default.
> > > 
> > > If multi queues is enabled, the algorithm for specifying how many queues
> > > each virtio vnic will have will be as follows -
> > > 1. If the number of vCpus is lower than the number of virtio vnics an audit
> > > log warning saying multi queues cannot be applied, will be displayed.
> > > 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> > > 3. Dividing the queues between the virtio vnics. All the vnics will get at
> > > least 1 queue, some random vnics will get 2 queues (depending on the
> > > number_of_queues).
> > > 
> > > * CustomDeviceProperties will override the "multi queues" algorithm.
> > 
> > There should be a max of 4 queues for up to 100 cores or 8 queues for above
> > that.
> 
> Updated the algorithm to the following -
> 
> If the number of vCpus is larger than 100, each vnic will get 8 queues.
> Otherwise, each vnic will get 4 queues. If there are no enough vcpus, each
> vnic will get vcpus/num_of_vnics queues. The extra queues will be devided
> randomly between the vnics. Each vnic will have a least one queue. Please
> notice the hotplug algorithm does best effort to find out the optimal number
> of queues to assign to the hot plugged nic. But since we don't persist the
> number of queues assigned per each vnic, there may be mistakes. Only
> re-running the vm promises the best allocation of the queues.

AlL of our testing was done with 8 queues (10Gb, 25Gb, 100Gb)
 
This new algorithm will be available to users as out of the box tuning correct?
If so this  should work fine for the standard customer wanting more throughput that multi-queue can provide. 

The more aggressive customer wanting higher que depth can still change muti-queue through CustomDeviceProperties, correct?

(In reply to Bob Sibley from comment #19)
> This new algorithm will be available to users as out of the box tuning
> correct?

Yes, in the High Performance VM type.

> If so this should work fine for the standard customer wanting more
> throughput that multi-queue can provide.

That's the idea :)

> The more aggressive customer wanting higher que depth can still change
> muti-queue through CustomDeviceProperties, correct?

Yes, mentioned above in Comment 14.

(In reply to Jamie Bainbridge from comment #20)
> (In reply to Bob Sibley from comment #19)
> > This new algorithm will be available to users as out of the box tuning
> > correct?
> 
> Yes, in the High Performance VM type.

And Server VM type as well.

(In reply to Yaniv Kaul from comment #21)
> (In reply to Jamie Bainbridge from comment #20)
> > (In reply to Bob Sibley from comment #19)
> > > This new algorithm will be available to users as out of the box tuning
> > > correct?
> > 
> > Yes, in the High Performance VM type.
> 
> And Server VM type as well.

Why not setting multi-queues as default for all types of vms?
Can it harm desktop vms?

(In reply to Alona Kaplan from comment #22)
> (In reply to Yaniv Kaul from comment #21)
> > (In reply to Jamie Bainbridge from comment #20)
> > > (In reply to Bob Sibley from comment #19)
> > > > This new algorithm will be available to users as out of the box tuning
> > > > correct?
> > > 
> > > Yes, in the High Performance VM type.
> > 
> > And Server VM type as well.
> 
> Why not setting multi-queues as default for all types of vms?
> Can it harm desktop vms?

You tell me. My concern is that for VDI use case, the density might be a factor - how much additional memory and CPU do the queues consume (and does VDI really need that higher perform at some cost) ?

I doubt there would be a problem enabling mq for all VMs but I have not tested common use cases.

Memory usage is negligible, just a few small kernel data structures for each additional queue.

Theoretically, mq would be *better* for CPU usage as it allows network processing load to spread more evenly across CPUs instead of one queue all being handled by one core. There is no additional CPU usage, the system receives the amount of traffic it receives, mq just spreads that CPU usage around across more cores.

The only reason I didn't suggest it is that one queue tends to be "good enough" for most common desktop and even mid-power server usage.

I'd be happy to see mq used on all RHV VMs.

Perhaps ask QE to do a sanity check on a moderately busy VDI. If no negative latency or usage impact is seen, make it the default for all.

(In reply to Alona Kaplan from comment #14)
> 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)

This is wrong and not what was discussed prior. This is not what we want. Don't do this.

(In reply to Yaniv Lavi from comment #16)
> There should be a max of 4 queues for up to 100 cores or 8 queues for above
> that.

Having though about this more, I suggest not to include this logic.

A 100-core VM is well past the "one size fits most" sort of usage this rule is intended to hit.

Someone with a 100-core VM is likely going to do significant work with physical CPU pinning, virtual NUMA nodes, and a significant amount of work inside the VM and their application to properly manage CPU usage. If such an advanced user wants to add more net queues, that's a very minor and simple step compared to everything else they have to do.


Just keep the rule simple and relevant to majority usage:

 queues = min(num_of_vCpus, 4)

(In reply to Jamie Bainbridge from comment #25)
> (In reply to Alona Kaplan from comment #14)
> > 2. num_of_queues = min(num_of_vCpus, num_of_vnics * 2)
> 

The algorithm is comment #14 is outdated. Please refer to the algorithm is comment #17.

> This is wrong and not what was discussed prior. This is not what we want.
> Don't do this.
> 
> (In reply to Yaniv Lavi from comment #16)
> > There should be a max of 4 queues for up to 100 cores or 8 queues for above
> > that.
> 
> Having though about this more, I suggest not to include this logic.
> 
> A 100-core VM is well past the "one size fits most" sort of usage this rule
> is intended to hit.
> 
> Someone with a 100-core VM is likely going to do significant work with
> physical CPU pinning, virtual NUMA nodes, and a significant amount of work
> inside the VM and their application to properly manage CPU usage. If such an
> advanced user wants to add more net queues, that's a very minor and simple
> step compared to everything else they have to do.
> 
> 
> Just keep the rule simple and relevant to majority usage:
> 
>  queues = min(num_of_vCpus, 4)

What do you mean by 'queues'? The number of queues that each vnic should get or the total amount of queues that should be divided between the vnics?

Are you ok with this algorithm?
Each vnic will get 4 queues. If there are no enough vcpus, each vnic will get vcpus/num_of_vnics queues. The extra queues will be divided randomly between the vnics. Each vnic will have at least one queue.

For example -
1. numOfVnics = 5, numOfCpus = 12
nic1=3queues , nic2==3queues, nic3=2queues, nic4=2queues, nic5=2queues

2. numOfVnics = 5, numOfCpus = 40
nic1=4queues , nic2==4queues, nic3=4queues, nic4=4queues, nic5=4queues

Hi Alone,

(In reply to Alona Kaplan from comment #26)
> For example -
> 1. numOfVnics = 5, numOfCpus = 12
> nic1=3queues , nic2==3queues, nic3=2queues, nic4=2queues, nic5=2queues

As in the gerrit comment, I think this should have 4 queues on every vNIC.

Lets see if Jamie agrees.

(In reply to Alona Kaplan from comment #26)
> >  queues = min(num_of_vCpus, 4)
> 
> What do you mean by 'queues'? The number of queues that each vnic should get
> or the total amount of queues that should be divided between the vnics?

The number of queues that each NIC should get.

The number of total NICs/queues isn't a concern. It's fine if some NICs "double up" and use the same cores. It is unlikely that two queues would become so busy they would pin the CPU on softirq processing, and if that does happen then the irqbalance service will move the IRQ handling for one or both queues, so CPU usage will reduce soon. There is no need to do complex algorithms to create different numbers of queues based on total core count.

Again we are aiming for "most common" customer usage. In my experience it is very rare for customers to use more than three interfaces within a VM, most have just one NIC.

> Are you ok with this algorithm?

No, far too complex.

Following the algo above:

 num_of_nic_queues = min(num_of_vCpus, 4)

Examples:

1. numOfVnics = 5, numOfCpus = 12

Each NIC = 4 queues.
Total queues = 20.

2. numOfVnics = 5, numOfCpus = 40

Any NIC = 4 queues.
Total queues = 20.

WARN: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

Verified on - 4.2.5.1_SNAPSHOT-71.g54dde01.0.scratch.master.el7ev

WARN: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2.z': '?'}', ]

For more info please contact: rhv-devops

Why was this bug not cloned to zstream?

IIUC we went with up to 4 mq per NIC, correct? Just making sure since several comments discuss no upper bound and that the amount is not a concern. Reading e.g. https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html it doesn’t seem to be true

(In reply to Michal Skrivanek from comment #35)
> IIUC we went with up to 4 mq per NIC, correct?

Correct. Per-NIC we should calculating: min(num_of_vCpus, 4)

With common customer usage, where at most we see 3 virtio NICs, we won't come anywhere near a limit of 200 IRQs. A customer would need 40 virtio NICs to hit that limitation. (40+(40*4)=200)

Restoring Michal's needinfo (comment #35)

(In reply to Michal Skrivanek from comment #35)
> Why was this bug not cloned to zstream?
> 
> IIUC we went with up to 4 mq per NIC, correct? Just making sure since
> several comments discuss no upper bound and that the amount is not a
> concern. Reading e.g.
> https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/
> libvirt-virtiomq.html it doesn’t seem to be true

Yes, the amount of the queues will be up to 4.

As this bug hasn't been closed yet, can you please change the name of the field in the UI to Multi Queues Enabled with a capital E.

It needs to be consistent with the other fields.

Dan, do we still need/want to document the customer property now that we have this field?

Is there any point to changing the value? From what I understood from Alona, virt  testing did not produce better performance by changing this value.

This bugzilla is included in oVirt 4.2.5 release, published on July 30th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.5 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.