Bug 1954920 - Auto Pinning Policy results in division by zero on hosts with 1 NUMA node.
Summary: Auto Pinning Policy results in division by zero on hosts with 1 NUMA node.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.4.6
Hardware: x86_64
OS: Linux
unspecified
medium vote
Target Milestone: ovirt-4.4.7
: 4.4.7
Assignee: Liran Rotenberg
QA Contact: Polina
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-29 04:56 UTC by Germano Veit Michel
Modified: 2021-07-06 07:28 UTC (History)
5 users (show)

Fixed In Version: ovirt-engine-4.4.7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-06 07:28:10 UTC
oVirt Team: Virt
pm-rhel: ovirt-4.4+
pm-rhel: devel_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 6006061 0 None None None 2021-04-30 04:21:38 UTC
oVirt gerrit 114549 0 master MERGED core: validate auto pinning adjust with one core 2021-05-04 14:05:04 UTC

Description Germano Veit Michel 2021-04-29 04:56:57 UTC
Description of problem:

Auto Pinning Policy produces incorrect configuration when the Host has 1 NUMA node.

1. This is the Host, very simple, 1 NUMA and 4 CPUs:

# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3
node 0 size: 3922 MB
node 0 free: 2820 MB
node distances:
node   0 
  0:  10 

2. Create 1 VMs with 4 vCPUs, pin the VM to the host, everything else default.

There is no pinning at this point:

engine=# select vm_name,cpu_pinning,num_of_sockets,cpu_per_socket,threads_per_cpu from vm_static where vm_name = 'VM3';
 vm_name | cpu_pinning | num_of_sockets | cpu_per_socket | threads_per_cpu 
---------+-------------+----------------+----------------+-----------------
 VM3     |             |              4 |              1 |               1
(1 row)

3. VM -> Edit -> Host -> Auto Pinning Policy -> Adjust -> OK

Now we expect it to pin the VM 4 CPUs to the Host 4 CPUs, both with 1 NUMA.

This is the result: no pinning and 0 CPUs per socket.

engine=# select vm_name,cpu_pinning,num_of_sockets,cpu_per_socket,threads_per_cpu from vm_static where vm_name = 'VM3';
 vm_name | cpu_pinning | num_of_sockets | cpu_per_socket | threads_per_cpu 
---------+-------------+----------------+----------------+-----------------
 VM3     |             |              4 |              0 |               1
 
 The VM fails to run, division by zero:
 
 2021-04-29 14:53:45,949+10 ERROR [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-13572) [d2f65764-8ce7-4b45-956c-394ee23dbb7e] Failed to create VM: java.lang.ArithmeticException: / by zero
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.builder.vminfo.LibvirtVmXmlBuilder.writeCpu(LibvirtVmXmlBuilder.java:470)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.builder.vminfo.LibvirtVmXmlBuilder.buildCreateVm(LibvirtVmXmlBuilder.java:290)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand.generateDomainXml(CreateBrokerVDSCommand.java:99)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand.createInfo(CreateBrokerVDSCommand.java:53)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.CreateBrokerVDSCommand.executeVdsBrokerCommand(CreateBrokerVDSCommand.java:44)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
	at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
	at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:398)

   463          if ((boolean) Config.getValue(ConfigValues.SendSMPOnRunVm)) {
   464              writer.writeStartElement("topology");
   465              writer.writeAttributeString("cores", Integer.toString(vm.getCpuPerSocket()));
   466              writer.writeAttributeString("threads", Integer.toString(vm.getThreadsPerCpu()));
   467              int vcpus = FeatureSupported.supportedInConfig(ConfigValues.HotPlugCpuSupported, vm.getCompatibilityVersion(), vm.getClusterArch()) ?
   468                      VmCpuCountHelper.calcMaxVCpu(vm, vm.getClusterCompatibilityVersion())
   469                      : vm.getNumOfCpus();
   470              writer.writeAttributeString("sockets", String.valueOf(vcpus / vm.getCpuPerSocket() / vm.getThreadsPerCpu()));
   471              writer.writeEndElement();

Expected with that cpu_per_socket = 0.

Version-Release number of selected component (if applicable):
ovirt-engine-4.4.6.5-0.17.el8ev.noarch

How reproducible:
Always

Steps to Reproduce:
As above

Actual results:
* Inconsistent configuration

Expected results:
* Pin the 4 CPUs of the VM to the 4 CPUs on the host
* VM with 1 cpu per socket, or some other valid combination

Comment 1 Liran Rotenberg 2021-04-29 07:26:49 UTC
The problem here is that there is a use of Adjust policy and the host has pretty bad hardware to use this feature with.
If you have too many pCPU, you might not get a CPU pinning at all (if vCPUs / host sockets / host threads == 0) and 1 thread is optional but isn't entirely suitable to this feature as well.

When using adjust the VM topology is adjusted to the host hardware.
It will set the CPUs like this:
vSockets = pSockets
vCores = pCores - 1
vThreads = pThreads

Our problem is that this host had 1 core. We should not allow to set 0 as cores. It is also might be an indicator for the host not really fitting the feature.
In such case we need to do one of these:
1. Nothing, and just return like you didn't set adjust policy.
2. Do vCores = pCores without dropping one.
3. Look for optional -1 in sockets/threads.

Note that the 1 core decrease is to let the host enough breathing space.

Comment 2 Arik 2021-04-29 08:14:28 UTC
Right, but still - we shouldn't fail to run the VM in this case
It's better to fail AddVm/UpdateVm than getting into a situation we have a VM that cannot run due to invalid configuration

Comment 3 Liran Rotenberg 2021-04-29 08:24:01 UTC
(In reply to Arik from comment #2)
> Right, but still - we shouldn't fail to run the VM in this case
> It's better to fail AddVm/UpdateVm than getting into a situation we have a
> VM that cannot run due to invalid configuration

Yes, checking for 1 core combined with adjust in the validation phase is also a good solution.

Comment 4 Michal Skrivanek 2021-04-30 07:42:47 UTC
why did we choose blocking it instead of keeping the pinning to host cores(i.e there would be no additional "space", sure, but at least you do not add a failure that someone needs to deal with, understand, etc)

Comment 5 Arik 2021-04-30 10:19:57 UTC
Michal, it's not that simple - in order to support auto-adjusting with such hardware we'll need to change the logic of the auto-adjust to produce valid configuration
not that it's so complicated, but it's just redundant - better to just block it for hardware that it doesn't fit for

Comment 6 Michal Skrivanek 2021-04-30 10:27:35 UTC
blocking anything is also a major source of frustration. We do that when we have no choice and there is a risk of wrong behavior.

Comment 7 Arik 2021-04-30 10:32:03 UTC
yes, for the most part
but we also block things that we don't test or that doesn't really make sense - like high-availability for vms that are pinned to a single host ;)

Comment 8 Germano Veit Michel 2021-05-02 21:54:41 UTC
The only good use case for this is a big VM on a big 1 NUMA host that requires high performance. So the user can click a button instead of manually typing that not user friendly at all pinning string in the resource allocation tab.

Comment 9 Arik 2021-05-03 07:18:14 UTC
I think we're talking about two different things - (A) a single numa node; (B) a single core per socket
IIUC, the auto-pinning assignment failed with a single core per socket and should work well with a single NUMA host where core-per-socket > 1
Liran, please keep me honest

Comment 10 Liran Rotenberg 2021-05-03 08:04:24 UTC
(In reply to Arik from comment #9)
> I think we're talking about two different things - (A) a single numa node;
> (B) a single core per socket
> IIUC, the auto-pinning assignment failed with a single core per socket and
> should work well with a single NUMA host where core-per-socket > 1
> Liran, please keep me honest

Correct.

Comment 11 Germano Veit Michel 2021-05-03 21:11:22 UTC
In that case I'd agree blocking is a fine resolution for this.

Comment 12 Polina 2021-06-02 13:20:22 UTC
verified on ovirt-engine-4.4.7-0.23.el8ev.noarch.

found a host with 1 core.
host lscpu: 

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           4
NUMA node(s):        1
Vendor ID:           GenuineIntel
BIOS Vendor ID:      Red Hat
CPU family:          6
Model:               85
Model name:          Intel Xeon Processor (Skylake)
BIOS Model name:     RHEL 7.5.0 PC (i440FX + PIIX, 1996)
Stepping:            4
CPU MHz:             2294.606
BogoMIPS:            4589.21
Virtualization:      VT-x
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-3

In UI - Error while executing action:
golden_env_mixed_virtio_1_0:
Cannot edit VM. The Auto Pinning policy Adjust cannot be applied to a VM that is pinned to a single core host.

In Rest API - 
2021-06-02 16:11:48,108+03 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-8) [] Operation Failed: [Cannot add VM. The Auto Pinning policy Adjust cannot be applied to a VM that is pinned to a single core host.]

Comment 13 Sandro Bonazzola 2021-07-06 07:28:10 UTC
This bugzilla is included in oVirt 4.4.7 release, published on July 6th 2021.

Since the problem described in this bug report should be resolved in oVirt 4.4.7 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.