Bug 1863615

Summary: High Performance, headless VM fails to run when having graphic consoles devices
Product: [oVirt] ovirt-engine Reporter: Roy Golan <rgolan>
Component: BLL.VirtAssignee: Liran Rotenberg <lrotenbe>
Status: CLOSED CURRENTRELEASE QA Contact: Nisim Simsolo <nsimsolo>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.4.2CC: ahadas, bugs, gchakkar, gdeolive, kmajumde, lrotenbe, michal.skrivanek, nsimsolo, sigbjorn.lie, vpagar
Target Milestone: ovirt-4.4.2Keywords: Regression, TestBlocker
Target Release: 4.4.2.3Flags: pm-rhel: ovirt-4.4+
pm-rhel: blocker?
ahadas: devel_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhv-4.4.2-3, ovirt-engine-4.4.2.3 Doc Type: Bug Fix
Doc Text:
Previously, when having a VM set with High Performance profile by API, the VM couldn't start, having no USB controller but USB devices. Also, for 4.3 clusters, the host don't report the TSC frequency. Now, TSC is dropped for 4.3 clusters and the VM won't have USB devices when there is no USB contoller, allowing VMs to run normally.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-18 07:12:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine.log none

Description Roy Golan 2020-08-03 16:58:22 UTC
HP VM fails to start, VDSM fails it with:

2020-08-03 19:02:06,081+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-15) [d3a0767] EVENT_ID: VM_DOWN_ERROR(119), VM catapult-xhfrp-m
aster-1 is down with error. Exit message: unsupported configuration: Can't add USB input device. USB bus is disabled.

HP VM, cluster compat 4.3, headless mode.


Seems like there is a usb controller and video and graphic device in the device list, eventhought it should be headless. 

  <devices>
    ...
    <controller type="usb" model="none" index="0">
      <alias name="ua-8690459e-9323-4080-abfc-c5527bacd9da"/>
    </controller>
    <video>
      <model type="qxl" vram="32768" heads="1" ram="65536" vgamem="16384"/>
      <alias name="ua-9187c283-3bb9-4a27-8ea3-b3b89668abc1"/>
    </video>
    <graphics type="vnc" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" keymap="en-us">
      <listen type="network" network="vdsm-ovirtmgmt"/>
    </graphics>

Comment 1 Roy Golan 2020-08-03 17:02:23 UTC
Created attachment 1704443 [details]
engine.log

Comment 2 Michal Skrivanek 2020-08-04 06:40:44 UTC
You can't create a VM which is set as headless (maybe coming from the HP profile) and enable console at the the same time (apparently you set VNC+SPICE). If that happened automatically we need to see the API request details.

Comment 3 Roy Golan 2020-08-04 09:42:19 UTC
(In reply to Michal Skrivanek from comment #2)
> You can't create a VM which is set as headless (maybe coming from the HP
> profile) and enable console at the the same time (apparently you set
> VNC+SPICE). If that happened automatically we need to see the API request
> details.

This VM is created using the API - with vm type to HP and os RHCOS. I didn't mess with the console at all

Creating a VM from the UI with HP type yields an NPE while trying to run it:


2020-08-04 12:41:14,086+03 ERROR [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23756) [ee0b9693-9238-4fd9-a1aa-997
8d1dcf208] Command 'CreateVDSCommand( CreateVDSCommandParameters:{hostId='d5cee641-f873-4729-b2f6-f2db7a0092e3', vmId='e2690b65-e830-432e-990c-02fb97e81656', 
vm='VM [HP-USB-FAIL]'})' execution failed: java.lang.NullPointerException
2020-08-04 12:41:14,086+03 DEBUG [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-23756) [ee0b9693-9238-4fd9-a1aa-997
8d1dcf208] Exception: java.lang.RuntimeException: java.lang.NullPointerException
        at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.CreateVDSCommand.executeVmCommand(CreateVDSCommand.java:59)
        at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ManagingVmCommand.executeVDSCommand(ManagingVmCommand.java:17)
        at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
        at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
        at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
        at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:398)

Comment 4 Sandro Bonazzola 2020-08-04 11:28:34 UTC
Marking as test blocker, it prevents OCP on oVirt deployment to work.

Comment 5 Sandro Bonazzola 2020-08-04 11:29:06 UTC
Also marking as regression since it's working with oVirt 4.4.0

Comment 6 Liran Rotenberg 2020-08-04 12:54:34 UTC
The NPE is a result of TSC. It introduced in 4.4 and didn't backport to 4.3.
The result is VDSM not reporting the TSC from the host to the engine for clusters < 4.4 and we should avoid setting TSC in that case.

As investigated, the failure regarding the USB is due missing the DELETE API requests to remove the graphic consoles of the VM.

Comment 7 Michal Skrivanek 2020-08-04 15:33:31 UTC
(In reply to Sandro Bonazzola from comment #5)
> Also marking as regression since it's working with oVirt 4.4.0

you shouldn't be using 4.3 cluster level. In fully up to date setup it's supposed to be 4.4 with 4.4 clusters

Comment 8 RHEL Program Management 2020-08-04 15:40:09 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 9 Roy Golan 2020-08-05 06:39:45 UTC
Michal, Liran, why is the engine creating graphic console if the VM is HP and should be headless? 
And by that rendering a VM which will always fail to start (I'm ignoring the NPE here, this is not the real problem)

Comment 10 Liran Rotenberg 2020-08-05 08:14:24 UTC
(In reply to Roy Golan from comment #9)
> Michal, Liran, why is the engine creating graphic console if the VM is HP
> and should be headless? 
> And by that rendering a VM which will always fail to start (I'm ignoring the
> NPE here, this is not the real problem)

It is the way HP VMs were designed for safety, yes it's non-trivial.
More can be read in:
BZ 1780936#c3
https://gerrit.ovirt.org/#/c/105929/
BZ 1406394

Comment 11 Arik 2020-08-05 12:17:16 UTC
Adding to comment 10 that not all HP VMs are necessarily headless

Comment 13 Arik 2020-08-06 15:10:46 UTC
Liran, please check if the exception that was mentioned in the description still happens - the VM should start with VNC in this case

Comment 14 Arik 2020-08-07 15:49:10 UTC
*** Bug 1867098 has been marked as a duplicate of this bug. ***

Comment 15 Guilherme Santos 2020-08-11 13:39:44 UTC
Related to this one: https://bugzilla.redhat.com/show_bug.cgi?id=1848922

Comment 18 Nisim Simsolo 2020-08-27 12:36:20 UTC
Verified:
ovirt-engine-4.4.2.3-0.6.el8ev
vdsm-4.40.26-1.el8ev.x86_64
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64

Verification scenario:
1. Create new VM with Q35 chipset and legacy BIOS.
2. Change VM to high performance VM from API. run VM.
3. Verify:
VM is running properly
Console can be opened for that VM, console USB selection is grayed out (cannot select USB device)
ssh VM and verify lsusb is not listing USB devices
           verify lspci is not listing USB controllers.
No errors in engine.log and vdsm.log
4. Repeat steps 1-3, this time for pc-i440fx and legacy BIOS.
5. Repeat steps 1-4, this time set high performance VM from WebAdmin.
step 3 note: when setting HP VM from WebAdmin, headless mode is also enabled, so console cannot be opened on such VM.
6. repeat steps 1-5, this time using 4.3 cluster.

Comment 19 Sandro Bonazzola 2020-09-18 07:12:47 UTC
This bugzilla is included in oVirt 4.4.2 release, published on September 17th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.