Bug 1148502
| Summary: | pci pass 1002:6749 (radeon) error: Unable to read from monitor: Connection reset by peer | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | lejeczek <peljasz> | ||||||||
| Component: | doc-Virtualization_Tuning_and_Optimization_Guide | Assignee: | Laura Novich <lnovich> | ||||||||
| Status: | CLOSED NOTABUG | QA Contact: | ecs-bugs | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 7.0 | CC: | hhuang, jen, jhradile, lnovich, rbalakri, rhel-docs, virt-maint | ||||||||
| Target Milestone: | rc | Keywords: | Documentation, Reopened | ||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2015-12-14 16:49:23 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
libvirt-1.1.1-29.el7_0.1.x86_64 libvirt-client-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-config-network-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-interface-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-lxc-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-network-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-qemu-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-secret-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-driver-storage-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-kvm-1.1.1-29.el7_0.1.x86_64 libvirt-daemon-lxc-1.1.1-29.el7_0.1.x86_64 libvirt-python-1.1.1-29.el7_0.1.x86_64 qemu-2.0.0-1.el7.1.x86_64 qemu-common-2.0.0-1.el7.1.x86_64 qemu-img-1.5.3-60.el7_0.7.x86_64 qemu-kvm-1.5.3-60.el7_0.7.x86_64 Sorry, NVIDIA K-series Quadro, GRID, and Tesla are the only GPUs currently supported for device assignment. The vfio error indicates that both the GPU function 42:00.0 and the audio function 42:00.1 are part of the same IOMMU group and both devices must therefore be bound to vfio-pci. The audio function does not need to be assigned to the guest, but minimally it must be unbound from the host drivers. You can do this with "virsh nodedev-detach pci_0000_42_00_1" prior to starting the VM. Or simply assign both functions and libvirt will handle it. Your kernel log also shows an out-of-memory (oom) condition, note that when using PCI device assignment all of guest memory must be pinned on the host. It is not possible to overcommit memory with a guest using PCI device assignment. Also, graphics drivers like the radeon driver are not as amenable to unbinding and re-binding devices as are typical NIC drivers. It's recommended to blacklist the graphics driver in the host or prevent it from binding to the GPU by invoking pci-stub to claim it first. This can be done using a boot option like "pci-stub.ids=1002:6749". Regardless of this advice, assignment of AMD GPUs is not currently supported and I don't know if it will work. I could swear I had it work (meanwhile have done a number of upgrades/migrations of a few boxes and cannot be 100% sure), but I still have a rhel 6.5 with euk and man-build qemu 1.2.1 where AMD Radeons work still perfectly fine, actually more success than with NVIDIA. but back to the problem at hand - I am really not overcommitting, system is 2 socket Opteron 6366 with 64GB of RAM. even if I stub the devices I get (another-dmesg attached). If having devices stubbed I do qemu this way: (this time trying different Radeon) qemu-kvm -name __.a.win2008 -rtc base=localtime -m 16g -cpu host -smp cores=16 -vga std -drive file=virt4-dzien-64bit.win2008.img,index=0,cache=none,if=virtio -vnc :06, -monitor tcp:127.0.0.1:5907,server,nowait,nodelay -net tap,name=tap0,vlan=0,ifname=tap0,script=no,downscript=no -net nic,macaddr=52:54:00:12:36:75,model=virtio -device pci-assign,host=0000:41:00.0,prefer_msi=on,multifunction=off it results in: No IOMMU found. Unable to assign device "(null)" qemu-kvm: -device pci-assign,host=0000:41:00.0,prefer_msi=on,multifunction=off: Device initialization failed. qemu-kvm: -device pci-assign,host=0000:41:00.0,prefer_msi=on,multifunction=off: Device 'kvm-pci-assign' could not be initialized If I don't stub the device but dettach it(both or secondary only) before starting the guest - the same (oomkiller-dmesg) Created attachment 943500 [details]
oom killer
Created attachment 943501 [details]
another dmesg
and abrt does not catch it at all :( (In reply to lejeczek from comment #5) > I could swear I had it work (meanwhile have done a number of > upgrades/migrations of a few boxes and cannot be 100% sure), but I still > have a rhel 6.5 with euk and man-build qemu 1.2.1 where AMD Radeons work > still perfectly fine, actually more success than with NVIDIA. > > but back to the problem at hand - I am really not overcommitting, system is > 2 socket Opteron 6366 with 64GB of RAM. > even if I stub the devices I get (another-dmesg attached). > If having devices stubbed I do qemu this way: (this time trying different > Radeon) > > qemu-kvm -name __.a.win2008 -rtc base=localtime -m 16g -cpu host -smp > cores=16 -vga std -drive > file=virt4-dzien-64bit.win2008.img,index=0,cache=none,if=virtio -vnc :06, > -monitor tcp:127.0.0.1:5907,server,nowait,nodelay -net > tap,name=tap0,vlan=0,ifname=tap0,script=no,downscript=no -net > nic,macaddr=52:54:00:12:36:75,model=virtio -device > pci-assign,host=0000:41:00.0,prefer_msi=on,multifunction=off > > it results in: > > No IOMMU found. Unable to assign device "(null)" > qemu-kvm: -device > pci-assign,host=0000:41:00.0,prefer_msi=on,multifunction=off: Device > initialization failed. > qemu-kvm: -device > pci-assign,host=0000:41:00.0,prefer_msi=on,multifunction=off: Device > 'kvm-pci-assign' could not be initialized > > If I don't stub the device but dettach it(both or secondary only) before > starting the guest - the same (oomkiller-dmesg) pci-assign is not supported on RHEL7. Your oom log shows that 6250*2M hugepages are allocated per node, that's 12.5G * 4 = 50G. You therefore only have 14G remaining and you're trying to start a 16G VM that isn't making use of hugepages. Result: oom yes, that indeed were hugepages the root of the problem. I've followed redhat's "Visualisation Tuning and Optimisation Guide" but now I see it would have been nice if this had been covered more thoroughly. Is there any howto on VM+hugepages which one could read? I don't know, I think we can close this bug report. regards Assigning this to the Virtualization Optimization Guide The section in question is here https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Memory-Tuning.html#sect-Virtualization_Tuning_Optimization_Guide-Memory-Huge_Pages What else is needed here Best Regards, Laura lejeczek writes in Comment 10: >yes, that indeed were hugepages the root of the problem. >I've followed redhat's "Visualisation Tuning and Optimisation Guide" but now I >see it would have been nice if this had been covered more thoroughly. >Is there any howto on VM+hugepages which one could read? > >I don't know, I think we can close this bug report. Originally I took this to mean that there might be an opportunity to improve the documentation, but I don't know what the original reporter (lejeczek) thinks would be helpful. In addition, the original reporter suggests that the BZ can be closed. So, I'm going to close this report. If there are specific suggestions for improving the documentation, feel free to reopen this BZ. it's been a while and I cannot remember what exactly in my mind was missing there, but I had a quick look now at the url and it looks good. |
Created attachment 943105 [details] kernel log Description of problem: cannot start a windows guest with [AMD/ATI] Turks GL [FirePro V4900] attached. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: error: internal error: early end of file from monitor: possible problem: qemu-kvm: -device vfio-pci,host=42:00.0,id=hostdev0,bus=pci.0,multifunction=off,addr=0x5: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. qemu-kvm: -device vfio-pci,host=42:00.0,id=hostdev0,bus=pci.0,multifunction=off,addr=0x5: vfio: failed to get group 17 qemu-kvm: -device vfio-pci,host=42:00.0,id=hostdev0,bus=pci.0,multifunction=off,addr=0x5: Device initialization failed. qemu-kvm: -device vfio-pci,host=42:00.0,id=hostdev0,bus=pci.0,multifunction=off,addr=0x5: Device 'vfio-pci' could not be initialized Expected results: Additional info: #virsh nodedev-dumpxml pci_0000_42_00_0 <device> <name>pci_0000_42_00_0</name> <path>/sys/devices/pci0000:40/0000:40:02.0/0000:42:00.0</path> <parent>pci_0000_40_02_0</parent> <driver> <name>radeon</name> </driver> <capability type='pci'> <domain>0</domain> <bus>66</bus> <slot>0</slot> <function>0</function> <product id='0x6749'>Turks GL [FirePro V4900]</product> <vendor id='0x1002'>Advanced Micro Devices, Inc. [AMD/ATI]</vendor> <iommuGroup number='17'> <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/> <address domain='0x0000' bus='0x42' slot='0x00' function='0x1'/> </iommuGroup> </capability> </device>