Bug 2270005

If I'm understanding the use case correctly, it sounds like Nova's NUMA-aware vSwitch feature should fulfill it [1]. You can skim the design spec (at least the initial parts) at your convenience, but the summary is that in nova.conf, we can specify the NUMA affinity of either Neutron physical networks, or Neutron tunnelled networks [2]. When configured correctly, instances with OVS-DPDK NICs should end up affined to the correct NUMA node, and should leave NUMA nodes with SRIOV devices free for other instances.

However, there is currently a bug that causes the NUMA-aware vSwitches feature to not work: https://bugzilla.redhat.com/show_bug.cgi?id=2240011

If I've understood the use case correctly, and it is in fact addressed by NUMA-aware vSwitches, then we can prioritize fixing that bug in a future 17.1 z-stream.

[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/numa-aware-vswitches.html
[2] https://docs.openstack.org/nova/wallaby/configuration/config.html#neutron

Hi Artom,

I discussed with the customer. The customer thinks that NUMA-aware vSwitch is not the solution for their use-case.
In their use-case, there is an instance which has SR-IOV port and OVS-DPDK interface. In such case, NUMA-aware vSwitch won't be a solution.

The requested feature is 
1. If preferred NUMA is configured in nova.conf or in flavor, nova tries to allocate vcpus from the specified numa in the option.
2. If there is no node when nova tries to allocate vcpu on the specified numa, then nova tries to use other numas for the instance.

Would you please consider this feature?

Best Regads,
Keigo Noha

Hi Keigo,

Nova has always refused to allow a specific NUMA node to be configurable as a fallback for instances. There are many issues with this. If it's done universally for the whole cloud, what happens if some compute hosts don't have the specified NUMA node? If it's done on each compute host individually, how to handle that NUMA node becoming full? These are just some of the questions that make implementing such a feature essentially impossible.

Back to your customer - you mentioned that the instance for which they want this has two interfaces: an SRIOV port, and an OVS-DPDK interface. In both cases, Nova has features that allow the user to specify a NUMA affinity policy: numa_affinity_policy [1] for SRIOV Neutron ports, and NUMA-aware vSwitches for OVS-DPDK. Can you help me understand why the customer is unwilling or unable to use these features? Are they trying to achieve something that is impossible with those features as they're currently implemented, or have they found bugs or edge cases?

Thanks in advance!

[1] https://docs.openstack.org/api-ref/network/v2/index.html#numa-affinity-policy

*** This bug has been marked as a duplicate of bug 2269653 ***

Hi Artom,

I'm sorry for reopening this bugzilla.

I and the customer discussed this topic in weekly meeting.
The customer doesn't want to use numa-aware vswitch in their deployment. The nova scheduler enforces the specific NUMA to use it. However, the customer's case just requires to assigned the preferred NUMA.
They ask us to implement a new option to change the order of CPU list in descending order in NUMA.
Currently, nova tries to get a cpu from ascending order of NUMA(NUMA0, 1 and so on).
However, there is a deployment that it should take the reversed order, NUMA N, N-1...1, 0.
The option just change the order of CPU list for scheduling the instance. It won't affect to the other part of nova.

Would you please share your thoughts to introduce the new option for CPU ordering?

Best Regards,
Keigo Noha

Keigo, can you please look at the bug that Artom marked this a duplicate of? That references a patch that we think will be better for the customer's use-case if we backport it to 17. It does not simply reverse the allocation order (which I think we've been clear we're not going to do) but instead favors non-PCI-attached NUMA nodes for instances that don't require it. If you only have two NUMA nodes and one has a PCI device attached, then it will have the same effect. Instances without a request for that PCI device will favor the other NUMA node. We'd like to get clarification that (once they understand it) the customer's use-case would indeed be solved by this behavior.