About the following spec:
https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html


I've tried it:

[root@controller1 ~(keystone_admin)]# nova boot --image rhel --flavor m1.medium.6cpus --nic net-id=46489bf3-eefc-4dff-b4d5-21d904d75016,queues=3 test1

ERROR (CommandError): Invalid nic argument 'net-id=46489bf3-eefc-4dff-b4d5-21d904d75016,queues=3'. Nic arguments must be of the form --nic <net-id=net-uuid,v4-fixed-ip=ip-addr,v6-fixed-ip=ip-addr,port-id=port-uuid>, with at minimum net-id or port-id (but not both) specified.

[root@controller1 ~(keystone_admin)]# nova interface-attach --net-id=46489bf3-eefc-4dff-b4d5-21d904d75016 --queues=3 test1
error: unrecognized arguments: --queues=3

But unfortunately, on liberty, there is no such an option.

(In reply to Eyal Dannon from comment #2)
> About the following spec:
> https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/
> libvirt-virtiomq.html
> 
> 
> I've tried it:
> 
> [root@controller1 ~(keystone_admin)]# nova boot --image rhel --flavor
> m1.medium.6cpus --nic net-id=46489bf3-eefc-4dff-b4d5-21d904d75016,queues=3
> test1
> 
> ERROR (CommandError): Invalid nic argument
> 'net-id=46489bf3-eefc-4dff-b4d5-21d904d75016,queues=3'. Nic arguments must
> be of the form --nic
> <net-id=net-uuid,v4-fixed-ip=ip-addr,v6-fixed-ip=ip-addr,port-id=port-uuid>,
> with at minimum net-id or port-id (but not both) specified.
> 
> [root@controller1 ~(keystone_admin)]# nova interface-attach
> --net-id=46489bf3-eefc-4dff-b4d5-21d904d75016 --queues=3 test1
> error: unrecognized arguments: --queues=3
> 
> But unfortunately, on liberty, there is no such an option.

Yes that option has not been implemented I put the URL has a simple reference. One of our engineer has worked on that implementation (vromanso). I'm going to see with him what can we do here. Probably pushing new spec to provide such option.

(In reply to Eyal Dannon from comment #2)
> About the following spec:
> https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/
> libvirt-virtiomq.html
> 
> 
> I've tried it:
> 
> [root@controller1 ~(keystone_admin)]# nova boot --image rhel --flavor
> m1.medium.6cpus --nic net-id=46489bf3-eefc-4dff-b4d5-21d904d75016,queues=3
> test1
> 
> ERROR (CommandError): Invalid nic argument
> 'net-id=46489bf3-eefc-4dff-b4d5-21d904d75016,queues=3'. Nic arguments must
> be of the form --nic
> <net-id=net-uuid,v4-fixed-ip=ip-addr,v6-fixed-ip=ip-addr,port-id=port-uuid>,
> with at minimum net-id or port-id (but not both) specified.
> 
> [root@controller1 ~(keystone_admin)]# nova interface-attach
> --net-id=46489bf3-eefc-4dff-b4d5-21d904d75016 --queues=3 test1
> error: unrecognized arguments: --queues=3
> 
> But unfortunately, on liberty, there is no such an option.

Not sure what you expected here, the queues= approach was listed as an alternative in the spec - one that was previously rejected by the community as exposing too much detail to the cloud user.

In the description you state "On OVS 2.5 , VHostUser will reject the connection if the number of queues is greater than the number configured from OVS." - what number of queues is OVS configuring? Are you saying we can't use multi-queue with OVS 2.5? That seems like a step backwards...

OVS 2.5 works with multi-queue, the problem is the number of queues MUST be equal to the number of vCPUs.
As I wrote earlier in case we boot an instance with 5 vcpus, while 6 queues are configured on the OVS, the VHU port will reject the connection, that's why they need to be equal.
Meanwhile I'm booting an instance with number of vCPUs equal to the number of queues and modify the number of queues as I wish "on the fly".

The spec you mentioned could solve this issue, if I could specify the number of queues per NIC but unfortunately as you wrote it rejected by community.

We need a way to assign number of queues in the instance, independent from its vCPU number.

Thanks,

(In reply to Eyal Dannon from comment #5)
> OVS 2.5 works with multi-queue, the problem is the number of queues MUST be
> equal to the number of vCPUs.
> As I wrote earlier in case we boot an instance with 5 vcpus, while 6 queues
> are configured on the OVS, the VHU port will reject the connection, that's
> why they need to be equal.
> Meanwhile I'm booting an instance with number of vCPUs equal to the number
> of queues and modify the number of queues as I wish "on the fly".
> 
> The spec you mentioned could solve this issue, if I could specify the number
> of queues per NIC but unfortunately as you wrote it rejected by community.
> 
> We need a way to assign number of queues in the instance, independent from
> its vCPU number.
> 
> Thanks,

So we can't really give to users possibility to configure such low level option related to the hosts. Probably queues have to be seen in Nova as a resources so we can apply quota per tenants or something similar...

Hi Amnon, with virtio: the number of TX queues and RX queues has (is) the same, no option to have different values. Right?

Also, vhost-user or vhost-net should not be different, right?

I'm asking as physical NIC can have a different RX and TX queue number, and this is very handy. This doesn't answer to the question: should it be visible at nova's level. But the other way around could be to modify virtio for the quest to be able to configure (negotiate) the number of queues per NIC, and hopefully, a different number on RX and TX side.

Thanks!
Franck

This issue is only for vhostuser.  The only requirement should be that the number of queues configured for the VM does not exceed the number of queues OVS currently supports for the running configuration -this is for DPDK only, does not affect vhost_net in any way.  There should not be a requirement that the number of queues match exactly, only that the number of queues the VM wished to use not exceed the number of queues OVS has to offer.

Ideally there should be no user input at all for this.  Nova should simply query OVS for the number of available queues for a vhostuser interface, and use the minimum of ( vcpus, ovs-queues).

Where is the OVS number of queues coming from ?  IMHO the right thing is to configure OVS to have the number of queues that Nova is using with the guest, not the other way around.

> Hi Amnon, with virtio: the number of TX queues and RX queues has (is) the
> same, no option to have different values. Right?

Right

> I'm asking as physical NIC can have a different RX and TX queue number, and
> this is very handy. This doesn't answer to the question: should it be
> visible at nova's level. But the other way around could be to modify virtio
> for the quest to be able to configure (negotiate) the number of queues per
> NIC, and hopefully, a different number on RX and TX side.

Currently it's the same number for Tx and Rx.
Can you elaborate on the need to have it different?

The actual question here is not Tx vs Rx, but rather the number of queues, and 
the way to have it automatically configured and synced.
From QEMU level (not libvirt or Nova), as of today, it should be manually configured with the same value (same number of queues) on both QEMU and OVS (separately). 
Next it should be configured on the guest side, to a number which is less or 
equal to the number configured on QEMU/OVS.

This still enables libvirt/Nova to do some automatic configuration, so that 
the guest number of queues can be selected to be the same or different from 
the number of vCPUs, and the only limitation is to have it less or 
equal to the number of queues configured on QEMU/OVS.

(In reply to Amnon Ilan from comment #10)
> 
> > Hi Amnon, with virtio: the number of TX queues and RX queues has (is) the
> > same, no option to have different values. Right?
> 
> Right
> 
> > I'm asking as physical NIC can have a different RX and TX queue number, and
> > this is very handy. This doesn't answer to the question: should it be
> > visible at nova's level. But the other way around could be to modify virtio
> > for the quest to be able to configure (negotiate) the number of queues per
> > NIC, and hopefully, a different number on RX and TX side.
> 
> Currently it's the same number for Tx and Rx.
> Can you elaborate on the need to have it different?

Physical NIC permit to have different RX and TX queues number, and the main benefits are: usually, in DPDK world, applications have more pthreads that can send packets than threads that receive packets. Typically, you have one TX queue per online CPU, while only one or two CPU are processing RX queues. Some applications are based on a pipeline, where you have only one RX pthread and another one TX pthread.

DPDK applications have been designed on bare-metal initially, and do assume that the number of RX queues and TX queues are different and configured at PMD init time. So there is a porting effort on virtIO, and the challenge there is that you want to avoid a un-even packet distribution among RX pthreads: how to manage 5 queues with 2 pthreads?



> 
> The actual question here is not Tx vs Rx, but rather the number of queues,
> and 
> the way to have it automatically configured and synced.
> From QEMU level (not libvirt or Nova), as of today, it should be manually
> configured with the same value (same number of queues) on both QEMU and OVS
> (separately). 
> Next it should be configured on the guest side, to a number which is less or 
> equal to the number configured on QEMU/OVS.
> 
> This still enables libvirt/Nova to do some automatic configuration, so that 
> the guest number of queues can be selected to be the same or different from 
> the number of vCPUs, and the only limitation is to have it less or 
> equal to the number of queues configured on QEMU/OVS.

My recommendation, until I get formal RFEs from the VNF vendors (OVS-DPDK is not yet in the wild, but feedback will come with RHOSP10), we should configure the number of queues to be equal to the numbers of vCPUs. But we need to find a way for the VM to be able to configure the number of queues (ethtool for Linux guests, PMD init parameter for DPDK guests)

(In reply to Franck Baudin from comment #11)
> 
> But we
> need to find a way for the VM to be able to configure the number of queues
> (ethtool for Linux guests, 

This seems to be:
ethtool -L eth0 combined <queues_nr>

> PMD init parameter for DPDK guests)

testpmd has the parameter --rxq 
In fact it has also --txq (maybe it can support different number)

(In reply to Daniel Berrange from comment #9)
> Where is the OVS number of queues coming from ?

It generally comes from capacity planning, knowing the packet rate each queue can do (for example 4Mpps per queue), and then deciding on a multi-queue-maximum based on what the VMs need to handle.

> IMHO the right thing is to
> configure OVS to have the number of queues that Nova is using with the
> guest, not the other way around.

How can OVS anticipate this?  Couldn't a guest have 16, 32, or even 64 vcpus?  The number of queues in OVS is typically set at OVS start time.  I don't think OVS could confidently set a number-of-queues value unless it used a very high number.

(In reply to Andrew Theurer from comment #13)
> (In reply to Daniel Berrange from comment #9)
> > Where is the OVS number of queues coming from ?
> 
> It generally comes from capacity planning, knowing the packet rate each
> queue can do (for example 4Mpps per queue), and then deciding on a
> multi-queue-maximum based on what the VMs need to handle.
> 
> > IMHO the right thing is to
> > configure OVS to have the number of queues that Nova is using with the
> > guest, not the other way around.
> 
> How can OVS anticipate this?  Couldn't a guest have 16, 32, or even 64
> vcpus?  The number of queues in OVS is typically set at OVS start time.  I
> don't think OVS could confidently set a number-of-queues value unless it
> used a very high number.

It seems like there is a loop here in that we are saying the # of OVS queues comes from capacity planning for a specific workload - but that at the time that's being done it's not known what size the guests will be because they haven't been launched yet? How can we ever be sure that we have an optimal configuration for what is actually running then?

I guess what we mean is that part of that capacity planning is knowing exactly which workloads will land on which box, and that each box would potentially have a different number of queues at the OVS level based on the characteristics of the workloads expected to land there?

This all sounds extremely non-cloudy/inelastic. It seems to me like there is a desperate need for the OVS side of this equation to be much more dynamic than it is today if using multi-queues in a cloud environment is going to be realistic.

(In reply to Stephen Gordon from comment #14)
> (In reply to Andrew Theurer from comment #13)
> > (In reply to Daniel Berrange from comment #9)
> > > Where is the OVS number of queues coming from ?
> > 
> > It generally comes from capacity planning, knowing the packet rate each
> > queue can do (for example 4Mpps per queue), and then deciding on a
> > multi-queue-maximum based on what the VMs need to handle.
> > 
> > > IMHO the right thing is to
> > > configure OVS to have the number of queues that Nova is using with the
> > > guest, not the other way around.
> > 
> > How can OVS anticipate this?  Couldn't a guest have 16, 32, or even 64
> > vcpus?  The number of queues in OVS is typically set at OVS start time.  I
> > don't think OVS could confidently set a number-of-queues value unless it
> > used a very high number.
> 
> It seems like there is a loop here in that we are saying the # of OVS queues
> comes from capacity planning for a specific workload - but that at the time
> that's being done it's not known what size the guests will be because they
> haven't been launched yet? How can we ever be sure that we have an optimal
> configuration for what is actually running then?
> 
> I guess what we mean is that part of that capacity planning is knowing
> exactly which workloads will land on which box, and that each box would
> potentially have a different number of queues at the OVS level based on the
> characteristics of the workloads expected to land there?
> 
> This all sounds extremely non-cloudy/inelastic. It seems to me like there is
> a desperate need for the OVS side of this equation to be much more dynamic
> than it is today if using multi-queues in a cloud environment is going to be
> realistic.

Agreed, requiring queues to be decided at the time OVS is started is very inflexible and not a good fit for cloud. I would rather expect number of queues to be something that could be set per port, at the time we create the OVS port to back a VM's NIC.

Still no clear path forward here, has there been any progress on this from the OVS POV?

We have on-going investigations on OVS 2.6, lead by KVM team and NFV integration team, we expect not not modify Nova, so closing this BZ. We will open a new RFE when/if we have a concrete ask, once our homework will be done.