Bug 1566456

Summary: [PPC] Failed to start High performance VM with libvirtError: No free USB ports
Product: [oVirt] ovirt-engine Reporter: Polina <pagranat>
Component: BLL.VirtAssignee: Michal Skrivanek <michal.skrivanek>
Status: CLOSED DEFERRED QA Contact: Polina <pagranat>
Severity: high Docs Contact:
Priority: high    
Version: 4.2.2CC: bugs, pagranat, rbarry, sgratch
Target Milestone: ---Keywords: Automation, AutomationBlocker
Target Release: ---   
Hardware: ppc64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-01 14:48:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm and engine
none
logs none

Description Polina 2018-04-12 11:02:37 UTC
Created attachment 1420811 [details]
vdsm and engine

Description of problem:
Start High Performance VM in PPC env fails with libvirtError: No free USB ports

Version-Release number of selected component (if applicable): rhv-release-4.2.2-10-001.noarch

How reproducible:100%


Steps to Reproduce:
1.Edit VM / Optimized For/ High Performance
2.Check i/o threads
3.Pin VM NUMA node and VM CPU's on the same host NUMA nodes
2.Run VM

Actual results: failed
engine. 
2018-04-08 13:35:39,546+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-6) [] EVENT_ID: VM_DOWN_ERROR(119), VM golden_env_mixed_virtio_1_0 is down with error. Exit message: internal error: No free USB ports.

Expected results: VM must start

Additional info: vdsm and engine log attached

Comment 1 Michal Skrivanek 2018-04-13 04:55:36 UTC
Please also attach qemu.log

Comment 2 Polina 2018-04-23 14:16:22 UTC
Created attachment 1425674 [details]
logs

Comment 3 Polina 2018-04-23 14:16:42 UTC
added engine and vdsm libvirt logs. also edit window image . qemu.log is empty - VM failed with libvirt error 
2018-04-23 13:31:54.369+0000: 31390: error : virDomainUSBAddressAssign:2094 : internal error: No free USB ports (from libvirtd.log)

Comment 4 Ryan Barry 2018-08-15 21:07:57 UTC
Michal, Sharon -

I'm wading through the code, but I'm not sure why High Performance would be different here

The VM isn't getting a controller model defined, although it looks like there's a default for ppc64.

<controller index="0" model="none" type="usb" />

os.other_ppc64.devices.usb.controller.value = nec-xhci

Comment 5 Sharon Gratch 2018-08-16 11:12:23 UTC
(In reply to Ryan Barry from comment #4)
> Michal, Sharon -
> 
> I'm wading through the code, but I'm not sure why High Performance would be
> different here
> 
> The VM isn't getting a controller model defined, although it looks like
> there's a default for ppc64.
> 
> <controller index="0" model="none" type="usb" />
> 
> os.other_ppc64.devices.usb.controller.value = nec-xhci

High Performance VMs are different since only for them the USB controller is really disabled. That's why you see that no usb controller is created.

I see that in this case the VM runs with VNC console so need to check why USB is required here.

Comment 6 Sharon Gratch 2018-08-16 11:21:13 UTC
Usually usb is required in case of VNC for the tablet input device but we handled that already in the code and it doesn't seem like the case here. 
So need to check what is the exact problem.

Comment 7 Michal Skrivanek 2018-08-16 11:29:36 UTC
do you have the qemu log for this? It may not reproduce anymore too....

Comment 8 Polina 2018-08-19 13:48:43 UTC
Hi, from the same run the qemu log was empty (please see the comment 3).
I'll reproduce the scenario as soon as I have ppc environment built.

Comment 9 Polina 2018-08-20 15:28:59 UTC
Hi, I've reproduced the automation case in the ppc9 environment recently built (rhv-release-4.2.6-3-001.noarch). The VM start process fails immediately and the qemu log is not created. 

the bug was found while automation run. here are the manual way to reproduce:
1. System tab 
   Optimized for High Performance
   Total Virtual CPUs = 2

2. In Host tab 
   pin to host 1
   Don't allow migration
   Numa CPU Count = 2

3. Resource Allocation tab
   0#0_1#64

Close the Edit window , open again the 'Numa Pinning' in Host tab
pin vNuma to the nodes (Socket0, Socket1)


this matches to the automation test configuration:
Update existed vm golden_env_mixed_virtio_1_0 with {'placement_affinity': 'pinned', 'cpu_cores': 2, 'io_threads': 1, 'vcpu_pinning': [{'0': '0'}, {'1': '64'}], 'placement_hosts': ['host_mixed_1'], 'type': 'high_performance'}
Add NUMA node to VM with {'vm_name': 'golden_env_mixed_virtio_1_0', 'index': 0, 'pin_list': [0], 'host_name': 'host_mixed_1', 'memory': 512, 'cores': [0]}
Add NUMA node to VM with {'vm_name': 'golden_env_mixed_virtio_1_0', 'index': 1, 'pin_list': [8], 'host_name': 'host_mixed_1', 'memory': 512, 'cores': [1]}

Comment 10 Sharon Gratch 2018-08-21 15:38:15 UTC
(In reply to Polina from comment #9)
> Hi, I've reproduced the automation case in the ppc9 environment recently
> built (rhv-release-4.2.6-3-001.noarch). The VM start process fails
> immediately and the qemu log is not created. 
> 
Hi,

I verified that bug using a ppc host in my env. 
Polina's reproducing description mentioned in comment #9 didn't reproduce the problem and the VM started without any problem.

The bug was reproduced only after setting this HP VM to be non-headless (i.e. run the VM with VGA video device and with VNC graphics device).

Digging into it more deeper it seams that the VNC graphics device requires USB support although no mouse or tablet is defined (running with VGA without VNC didn't cause the error).

I searched in libvirt doc and didn't find something interesting.
Any ideas?

Comment 11 Ryan Barry 2018-08-21 15:47:08 UTC
Do we have documentated support for HP PPC VMs with VNC graphics? I'd lean towards removing this combination from the UI and confirming your findings with platform

Comment 12 Raz Tamir 2018-08-21 18:51:55 UTC
Probably worth to mention that this bug was found on IBM PPC and not HP PPC.

Comment 13 Ryan Barry 2018-08-21 20:53:08 UTC
Sorry, Raz. I'm using HP to refer to high performance here, not the vendor

Comment 14 Sharon Gratch 2018-08-21 21:05:27 UTC
(In reply to Raz Tamir from comment #12)
> Probably worth to mention that this bug was found on IBM PPC and not HP PPC.

Sorry for the confusion in my description, but when I write "HP VM" in comment #10, I mean "High Performance VM type" and not regarding the cpu arch (it is a shortcut that we frequently us, but in this context it is confusing so sorry).

Comment 15 Sharon Gratch 2018-08-21 21:30:47 UTC

(In reply to Ryan Barry from comment #11)
> Do we have documentated support for HP PPC VMs with VNC graphics? I'd lean
> towards removing this combination from the UI and confirming your findings
> with platform
But in that case you won't be able to run a High Performance VM with a graphic console. The default for High Performance is headless mode, but we do want to enable graphic console if user prefers or even just in "Run Once" for installing operations.

Maybe the problem is that since in engine we removed input devices in case of High Performance vms and ppc, then for VNC - libvirt added a mouse or a tablet with bus="usb" as a default? Need to investigate that.

Comment 16 Ryan Barry 2019-01-21 14:54:29 UTC
Re-targeting to 4.3.1 since it is missing a patch, an acked blocker flag, or both

Comment 17 Michal Skrivanek 2020-03-18 15:45:18 UTC
This bug didn't get any attention for a while, we didn't have the capacity to make any progress. If you deeply care about it or want to work on it please assign/target accordingly

Comment 18 Michal Skrivanek 2020-03-18 15:50:16 UTC
This bug didn't get any attention for a while, we didn't have the capacity to make any progress. If you deeply care about it or want to work on it please assign/target accordingly

Comment 19 Michal Skrivanek 2020-04-01 14:48:24 UTC
Closing old bug. Please reopen if still relevant/you want to work on it.