Description of problem: This is meant to track the issue found originally when testing [1] for 17.0. I have also tried doing this without the mixed dedicated policy and I ran into scheduling issues as well. Below is the working and failing scenarios tried: Compute Host NUMA Topology: NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 Hosts Dedicated/Shared Configurations: [tripleo-admin@computesriov-1 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_dedicated_set 24-39 [tripleo-admin@computesriov-1 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_shared_set 20-23 [tripleo-admin@computesriov-0 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_dedicated_set 4-19 [tripleo-admin@computesriov-0 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf compute cpu_shared_set 0,1,2,3 ########################################################################## # Working deployment (3 vCPU, asymmetric, and no mixed dedicated policy) # ########################################################################## (overcloud) [stack@undercloud-0 ~]$ openstack flavor show tempest-MixedCPUPolicyTestMultiNuma-flavor-1395833240 +----------------------------+-------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------------+-------------------------------------------------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 1 | | id | 862970624 | | name | tempest-MixedCPUPolicyTestMultiNuma-flavor-1395833240 | | os-flavor-access:is_public | True | | properties | hw:numa_cpus.0='0', hw:numa_cpus.1='1,2', hw:numa_mem.0='256', hw:numa_mem.1='768', hw:numa_nodes='2' | | ram | 1024 | | rxtx_factor | 1.0 | | swap | | | vcpus | 3 | +----------------------------+-------------------------------------------------------------------------------------------------------+ XML Output <vcpu placement='static'>3</vcpu> <cputune> <vcpupin vcpu='0' cpuset='20,22'/> <vcpupin vcpu='1' cpuset='21,23'/> <vcpupin vcpu='2' cpuset='21,23'/> <emulatorpin cpuset='20-23'/> </cputune> <numatune> <memory mode='strict' nodeset='0-1'/> <memnode cellid='0' mode='strict' nodeset='0'/> <memnode cellid='1' mode='strict' nodeset='1'/> </numatune> ###################################################################### # Failed deployment (3 vCPU, asymmetric, and mixed dedicated policy) # ###################################################################### (overcloud) [stack@undercloud-0 ~]$ openstack flavor show tempest-MixedCPUPolicyTestMultiNuma-flavor-1583852539 /usr/lib/python3.9/site-packages/openstack/config/cloud_region.py:452: UserWarning: You have a configured API_VERSION with 'latest' in it. In the context of openstacksdk this doesn't make any sense. warnings.warn( +----------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 1 | | id | 1495385529 | | name | tempest-MixedCPUPolicyTestMultiNuma-flavor-1583852539 | | os-flavor-access:is_public | True | | properties | hw:cpu_dedicated_mask='^0', hw:cpu_policy='mixed', hw:numa_cpus.0='0', hw:numa_cpus.1='1,2', hw:numa_mem.0='256', hw:numa_mem.1='768', hw:numa_nodes='2' | | ram | 1024 | | rxtx_factor | 1.0 | | swap | | | vcpus | 3 | +----------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ nova-scheduler.log:2022-10-17 16:25:17.163 12 DEBUG nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy='mixed',cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([0]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([])) on host_cell NUMACell(cpu_usage=0,cpuset=set([20,22]),id=0,memory=128210,memory_usage=0,mempages=[NUMAPagesTopology,NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pcpuset=set([32,34,36,38,24,26,28,30]),pinned_cpus=set([]),siblings=[set([36]),set([24]),set([20]),set([30]),set([38]),set([22]),set([26]),set([28]),set([32]),set([34])],socket=0) _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:929 nova-scheduler.log:2022-10-17 16:25:17.163 12 DEBUG nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] No specific pagesize requested for instance, selected pagesize: 4 _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:956 nova-scheduler.log:2022-10-17 16:25:17.163 12 DEBUG nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] Instance has requested pinned CPUs _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:1021 nova-scheduler.log:2022-10-17 16:25:17.164 12 DEBUG nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] Packing an instance onto a set of siblings: host_cell_free_siblings: [{36}, {24}, set(), {30}, {38}, set(), {26}, {28}, {32}, {34}] instance_cell: InstanceNUMACell(cpu_pinning_raw=None,cpu_policy='mixed',cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([0]),cpuset_reserved=None,id=0,memory=256,pagesize=None,pcpuset=set([])) host_cell_id: 0 threads_per_core: 1 num_cpu_reserved: 0 _pack_instance_onto_cores /usr/lib/python3.9/site-packages/nova/virt/hardware.py:658 nova-scheduler.log:2022-10-17 16:25:17.164 12 DEBUG nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] Built sibling_sets: defaultdict(<class 'list'>, {1: [{36}, {24}, {30}, {38}, {26}, {28}, {32}, {34}]}) _pack_instance_onto_cores /usr/lib/python3.9/site-packages/nova/virt/hardware.py:679 nova-scheduler.log:2022-10-17 16:25:17.164 12 DEBUG nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] User did not specify a thread policy. Using default for 1 cores _pack_instance_onto_cores /usr/lib/python3.9/site-packages/nova/virt/hardware.py:794 nova-scheduler.log:2022-10-17 16:25:17.164 12 INFO nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] Computed NUMA topology CPU pinning: usable pCPUs: [[36], [24], [30], [38], [26], [28], [32], [34]], vCPUs mapping: [] nova-scheduler.log:2022-10-17 16:25:17.165 12 INFO nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] Computed NUMA topology CPU pinning: usable pCPUs: [[36], [24], [30], [38], [26], [28], [32], [34]], vCPUs mapping: [] nova-scheduler.log:2022-10-17 16:25:17.165 12 DEBUG nova.virt.hardware [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] Failed to map instance cell CPUs to host cell CPUs _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:1049 nova-scheduler.log:2022-10-17 16:25:17.165 12 DEBUG nova.scheduler.filters.numa_topology_filter [req-6307d41d-02a6-40f7-88d8-c7fcc2714202 0bfbbc8dfa1548819b0af96786b1ad43 12a4c6a7df9947a39f54dedf04b22fea - default default] [instance: fa4f0399-3507-42e8-a02d-5fd1269b3762] computesriov-1.localdomain, computesriov-1.localdomain fails NUMA topology requirements. The instance does not fit on this host. host_passes /usr/lib/python3.9/site-packages/nova/scheduler/filters/numa_topology_filter.py:106 ########################################################### # Failed deployment (4 vCPU, asymmetric, no mixed policy) # ########################################################### (overcloud) [stack@undercloud-0 ~]$ openstack flavor show tempest-MixedCPUPolicyTestMultiNuma-flavor-1023626242 /usr/lib/python3.9/site-packages/openstack/config/cloud_region.py:452: UserWarning: You have a configured API_VERSION with 'latest' in it. In the context of openstacksdk this doesn't make any sense. warnings.warn( +----------------------------+---------------------------------------------------------------------------------------------------------+ | Field | Value | +----------------------------+---------------------------------------------------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 1 | | id | 2041661630 | | name | tempest-MixedCPUPolicyTestMultiNuma-flavor-1023626242 | | os-flavor-access:is_public | True | | properties | hw:numa_cpus.0='0', hw:numa_cpus.1='1,2,3', hw:numa_mem.0='256', hw:numa_mem.1='768', hw:numa_nodes='2' | | ram | 1024 | | rxtx_factor | 1.0 | | swap | | | vcpus | 4 | +----------------------------+---------------------------------------------------------------------------------------------------------+ nova-scheduler.log:2022-10-17 16:32:58.191 12 DEBUG nova.virt.hardware [req-96e2e8a3-8ea4-49ba-94e5-44bc00cb3081 e0c95053ea3742a79ff16dcdf2c40b0c b29448db925f45c58bd7aa693df43f9f - default default] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy=None,cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([1,2,3]),cpuset_reserved=None,id=1,memory=768,pagesize=None,pcpuset=set([])) on host_cell NUMACell(cpu_usage=0,cpuset=set([20,22]),id=0,memory=128210,memory_usage=0,mempages=[NUMAPagesTopology,NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pcpuset=set([32,34,36,38,24,26,28,30]),pinned_cpus=set([]),siblings=[set([36]),set([24]),set([20]),set([30]),set([38]),set([22]),set([26]),set([28]),set([32]),set([34])],socket=0) _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:929 nova-scheduler.log:2022-10-17 16:32:58.191 12 DEBUG nova.virt.hardware [req-96e2e8a3-8ea4-49ba-94e5-44bc00cb3081 e0c95053ea3742a79ff16dcdf2c40b0c b29448db925f45c58bd7aa693df43f9f - default default] No specific pagesize requested for instance, selected pagesize: 4 _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:956 nova-scheduler.log:2022-10-17 16:32:58.192 12 DEBUG nova.virt.hardware [req-96e2e8a3-8ea4-49ba-94e5-44bc00cb3081 e0c95053ea3742a79ff16dcdf2c40b0c b29448db925f45c58bd7aa693df43f9f - default default] Not enough host cell CPUs to fit instance cell; required: 3, actual: 2 _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:1010 Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20221004.n.1 How reproducible: 100% Steps to Reproduce: 1. Deploy 17.1 and attempt to deploy guest with above flavors 2. 3. Actual results: Asymmetric topologies with mixed dedicated polices and/or 4+ vCPUs fail to schedule Expected results: Above failing flavors schedule correctly Additional info: Bed is available upon request [1] https://bugzilla.redhat.com/show_bug.cgi?id=1693377
########################################################### # Failed deployment (4 vCPU, asymmetric, no mixed policy) # ########################################################### This flavor does not request pinning and asks for 3 vCPUs from one NUMA node (hw:numa_cpus.1='1,2,3'). Both computes have cpu_shared_set set to 4 pCPUs symmetrically. I.e. 2 pCPUs per NUMA node. So this flavor does not fit to these computes as there is no NUMA node with 3 shared CPUs available. I think in this case the scheduler is correctly rejects the request.
###################################################################### # Failed deployment (3 vCPU, asymmetric, and mixed dedicated policy) # ###################################################################### * hw:numa_cpus.0='0', hw:numa_cpus.1='1,2', hw:cpu_dedicated_mask='^0' => vCPU 1,2 needs to be pinned, i.e. coming from the cpu_dedicated_set => vCPU 0 needs to come from the cpu_shared_set => vCPU 0 and vCPU 1,2 should come from two different NUMA nodes. This should be possible on the existing computes. For example on computesriov-1: vCPU 0 => pCPU 20 (NUMA 0, cpu_shared_set) vCPU 1 => pCPU 25 (NUMA 1, cpu_dedicated_set) vCPU 2 => pCPU 27 (NUMA 1, cpu_dedicated_set) is a valid solution. So this is a bug.
Cloned to BZ 2152723 to get the fix in 17.1. Leaving this open targetted at z2 in case there's a 17.0.2, or a customer needs a hotfix.
*** Bug 2152723 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577