Bug 1792944
| Summary: | VM fails to start after previous unsuccessful invalid attempt of CPU unplugging . | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Polina <pagranat> | ||||
| Component: | BLL.Virt | Assignee: | Andrej Krejcir <akrejcir> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | meital avital <mavital> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.4.0 | CC: | akrejcir, bugs, rbarry | ||||
| Target Milestone: | ovirt-4.4.0 | Flags: | pm-rhel:
ovirt-4.4+
|
||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-05-20 19:59:43 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1437559 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Polina
2020-01-20 12:37:08 UTC
This is NOTABUG. We could provide better logging about the operation failing, but the inability to hot unplug without an agent is well-documented. The VM shows a pending configuration change in the UI, which is applied on shutdown. After this, the scheduler (correctly) will not start it. Where is the bug here except better audit logs in RHVM if this is an expected behavior, how the user is supposed to start such VM? We would expect them to read the logs for the scheduler error, and to not set the number of CPUs less than what their NUMA config specified. Does this work without a manual NUMA config? Passing scheduler messages back up to RHVM is something we can definitely improve on in 4.5, though also, it is not the case when the VM shows the pending configuration in UI (Desktop with newer configuration for the next run . Pending virtual machine changes). There is no this yellow UI icon for the case I describe. There is also no snapshot for the next configuration in the snapshots table. The CPU number remains unchanged when shut down. It looks for the user that nothing was changed in VM and yet the VM could not be restarted. (In reply to Ryan Barry from comment #3) > We would expect them to read the logs for the scheduler error, and to not > set the number of CPUs less than what their NUMA config specified. Does this > work without a manual NUMA config? > > Passing scheduler messages back up to RHVM is something we can definitely > improve on in 4.5, though happens with numa configuration only. please see the comment 4 also , we don't set here numa config less than the cpu number. numa is 2 Ok, so there are 2 potential bugs (no pending configuration change and maybe scheduler). Andrej, want to take a look? I think that pending configuration must be involved when the option 'Aply later' is chosen. but this is not the case. The issue is probably the CPU hotplug, not unplug. The validation of CPU hotplug does not consider the host, where the VM is running. It is possible to increase the number of CPU cores to more than the current host has. When the VM is shut down and started again, the scheduler checks which host has enough cores to run the VM. In this case it appears that the host has less CPU cores than the VM requires. Can you check? yes, it could be the case. [root@puma43 qemu]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Model name: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz Stepping: 7 CPU MHz: 2317.255 CPU max MHz: 2800.0000 CPU min MHz: 1200.0000 BogoMIPS: 4600.03 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-5,12-17 NUMA node1 CPU(s): 6-11,18-23 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts flush_l1d Created attachment 1654826 [details]
engine.log
reproduce in engine.log attached
2020-01-23 13:37:13,577+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-67116) [6779f16c] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM reproduce_start_problem (User: admin@internal-authz)
The cause is an incorrect domain XML sent to libvirt. In the <numa> tag there are more CPUs specified than the VM has.
<domain>
<vcpu current="8">16</vcpu>
...
<cpu match="exact">
<model>Westmere</model>
<topology cores="1" threads="1" sockets="16"/>
<numa>
<cell id="0" cpus="0,8-14" memory="524288"/>
<cell id="1" cpus="1,15-21" memory="524288"/>
</numa>
</cpu>
...
</domain>
It is probably caused by a patch from Bug 1437559.
The patches on Bug 1437559 have been merged and they should fix this bug too. this bug is targeting 4.4.2 and is in modified state. Can we retarget to 4.4.0 and move to QE? This bugzilla is included in oVirt 4.4.0 release, published on May 20th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.0 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |