Bug 1402323

Summary: Unable to attach Racepoint Lancer G5 adapter(FC) as pass through devices to VMs on HP DL580 G8 rack server on Rhel7.3 OS.
Product: Red Hat Enterprise Linux 7 Reporter: basavaraja gummanur <basavaraja.gummanur>
Component: qemu-kvmAssignee: Alex Williamson <alex.williamson>
Status: CLOSED CANTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.3CC: chayang, juzhang, knoel, rbalakri, vinaykumar.laghavarapu, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-07 14:50:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Passthrough Scenario in HP rack server none

Description basavaraja gummanur 2016-12-07 09:11:03 UTC 
Created attachment 1228941 [details]
Passthrough Scenario in HP rack server

Description of problem:Unable to attach Racepoint Lancer G5 adapter as pass through devices to VMs on HP DL580 G8 rack server on Rhel7.3 OS. 

Apart from this we have tried on ML350 Gen9 and same behavior is seen.

This issue is not seen on IBM servers.We have refered the link to over come this issue. But still we see even after disable of shared memory this issue is seen.

https://h20565.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=7271259&docId=emr_na-c04781229&docLocale=en_US
 


Version-Release number of selected component (if applicable):RHEL 7.3


How reproducible:Always


Steps to Reproduce:
1. Load the latest firmware 11.2.160.6  to ILO using Virtual Drives -> folder
option. 
2. After successful installation of latest firmware version reboot the system.
3. Boot to OS & install KVM & configure a Virt Manager.Install RHEL 7.3 OS to
the VM.
4. Enable Intel_iommu=on before attach the lancer adapter as passthrough device
to VM.
5. Boot the OS and verified the OS is able to boot.
5. Attach the lancer G5 adapter as passthrough to the VM & boot it. The OS is
not booting and it pops the message as "Error starting domain: internal error:
qemu unexpectedly closed the monitor".

Note: This server is in UEFI mode & and has been updated to latest ROM.


Find the details logs below:

Enabled Intel_ommu=on on the grub file.

[root@dhcp-10-227-77-72 ~]# !cat
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.el7.x86_64 root=/dev/mapper/rhel01-root ro
crashkernel=auto rd.lvm.lv=rhel01/root rd.lvm.lv=rhel01/swap rhgb quiet
intel_iommu=on
[root@dhcp-10-227-77-72 ~]#

Detached the Lancer PCI device 
[root@dhcp-10-227-77-72 ~]# virsh nodedev-detach pci_0000_41_00_0
Device pci_0000_41_00_0 detached

[root@dhcp-10-227-77-72 ~]# virsh nodedev-detach pci_0000_41_00_1
Device pci_0000_41_00_1 detached

Actual results:

Not able to boot the VM when we attach the Lancer G5 adapter(FC) as pass through.

Expected results:

VM should be bootable after we attach the Lancer G5 adapter(FC)


Additional info:

Attached the /var/log/messages
/var/log/libvirt/qemu/genric.log
Dmesg
screen shot of error message.
Comment 2 Alex Williamson 2016-12-07 14:50:14 UTC 
I'm sorry you're having this problem, however as noted in dmesg, the device is encumbered by the system firmware with reserved memory ranges, which are incompatible with device assignment:

[  699.965401] vfio-pci 0000:41:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

For further information, please see the following white paper:

https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf

It's possible that the disabling of shared memory is not complete or requires additional steps or specific firmware versions.  HP should be able to help with these issues.  So long as you're seeing the above dmesg error, we cannot support device assignment for that configuration.  Thanks.