Bug 624790
Summary: | pass through fails with KVM using Neterion Inc's X3100 Series 10GbE PCIe I/O Virtualized Server Adapter in Multifunction mode. | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | sivakumar <sivakumar.subramani> | ||||||
Component: | qemu-kvm | Assignee: | Alex Williamson <alex.williamson> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
Severity: | urgent | Docs Contact: | |||||||
Priority: | high | ||||||||
Version: | 6.1 | CC: | amit.shah, berrange, ddutile, eblake, juzhang, llim, ltroan, mkenneth, ram.vepa, tburke, virt-maint, xen-maint | ||||||
Target Milestone: | beta | ||||||||
Target Release: | 6.1 | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | qemu-kvm-0.12.1.2-2.127.el6 | Doc Type: | Bug Fix | ||||||
Doc Text: |
PCI configuration space access is improved, enabling a broader set of PCI devices to be device-assigned to guest VMs.
|
Story Points: | --- | ||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2011-05-19 11:23:25 UTC | Type: | --- | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | 544366, 626511 | ||||||||
Bug Blocks: | 580566, 580954, 624824 | ||||||||
Attachments: |
|
Description
sivakumar
2010-08-17 19:22:19 UTC
This issue has been proposed when we are only considering blocker issues in the current Red Hat Enterprise Linux release. ** If you would still like this issue considered for the current release, ask your support representative to file as a blocker on your behalf. Otherwise ask that it be considered for the next Red Hat Enterprise Linux release. ** Please provide the libvirt XML used for the guest, and the /var/log/libvirt/qemu/$GUEST.log file. Also what version of the 'qemu-kvm' and 'libvirt' RPMs do you have installed ? Created attachment 439442 [details]
Guest XML file.
Created attachment 439443 [details]
Guest log file.
Attached the XML and log file for the guest OSes. I have used the rpm from redhat6.0 snapshot10. libvirt-0.8.1-21.el6.x86_64 qemu-kvm-0.12.1.2-2.108.el6.x86_64 The log shows this error:
> create_userspace_phys_mem: Invalid argument
> assigned_dev_iomem_map: Error: create new mapping failed
I don't know why its failing that, but it appears the problem is at the QEMU layer.
Does this still fail with the latest bits? See bug 626511 for RHEL5 bug. Sivakumar, I can't seem to configure multi-function mode (or even SR-IOV mode) on my test system: # ./vxge_configure fmode avail +------------------------------------------------------------------------------+ | Neterion X3100 Adapter Configuration Utility: Version # xt1.59 | | | | Copyright(c) 2002-2009 Neterion Inc. | +------------------------------------------------------------------------------+ Defined Function Modes are: Segmentation fault (core dumped) dmesg shows: vxge_configure[5721] general protection ip:829ab79 sp:ffe1134c error:0 in vxge_configure[8048000+381000] The adapter reports: eth2: Firmware version : 1.4.4 Date : 09/17/2009 Attempting to blindly set mode #s also segfaults. I'm using vxge_configure from here: ftp://ftp.s2io.com/pub/X3100-Drivers/FIRMWARE/REL_1.4.4_FW-1.59_Linux.tar.gz How can I switch modes? Provided Alex with an out of tree vxge driver with the capability to change the function mode, besides other configurations. Once the X3100 is reconfigured with the out of tree driver, the vxge driver shipping with RHEL 5.x and 6.x will work. Ram Where can I download qemu-kvm-0.12.1.2-2.127.el6 version to test it? Will I be able to install it on Redhat 6.0 / 5.6 direclty. (In reply to comment #17) > Where can I download qemu-kvm-0.12.1.2-2.127.el6 version to test it? Will I be > able to install it on Redhat 6.0 / 5.6 direclty. Your partner manager(ltroan?) should be able to get you that package for testing. Note the ".el6" at the end of the package name: implying it's *only* for rhel6, not rhel5. Siva, please test this RHEL6 based module and report rsults back here. I am getting dependencies error while installing qemu-kvm. Can try to install with force option? or will you try to get me the corresponding version of qemu-img rpm [root@Nemo siva]# rpm -ivh qemu-kvm-0.12.1.2-2.127.el6.x86_64.rpm error: Failed dependencies: qemu-img = 2:0.12.1.2-2.127.el6 is needed by qemu-kvm-2:0.12.1.2-2.127.el6.x86_64 [root@Nemo siva]# Siva, you should be ok using the --force install option. Larry, you might pass the qemu-img rpm as well just to make it cleaner. I used --force and --nodeps option to install the rpm. qemu-kvm-0.12.1.2-2.127.el6.x86_64.rpm fixes the problem. I am able to test pass-through in MF mode on X3100 with out any issue. How do customers get this update rpm for RHEL 6.0? Thanks, Ram Both tested on fixed version(qemu-kvm-0.12.1.2-2.129.el6) and on unfixed version(qemu-kvm-0.12.1.2-2.127.el6),both can pass-through nics to guest. can not produced this issue.the following is detailed info,any mistake,please correct me. 1.Please note,eth0: Multi Function Mode Enabled #dmesg | grep eth0 eth0: SERIAL NUMBER: SXC1021107 eth0: PART NUMBER: X3120DA0001 eth0: Neterion X3120 Dual-Port DA 10GbE Server Adapter eth0: MAC ADDR: 00:0c:fc:01:46:5c eth0: Link Width x4 eth0: Firmware version : 1.4.4 Date : 09/16/2009 eth0: Multi Function Mode Enabled eth0: 4 Vpath(s) opened eth0: Interrupt type MSI-X eth0: RTH steering enabled for TCP_IPV4 eth0: Tx multiqueue steering enabled eth0: Generic receive offload enabled eth0: Rx doorbell mode enabled eth0: VLAN tag stripping Enabled eth0: Ring blocks : 2 eth0: Fifo blocks : 14 eth0: MTU is 1500 2.Unbind etho form host #ethtool -i eth0 driver: vxge version: 2.0.8.20182-k firmware-version: 1.4.4 bus-info: 0000:03:00.0 #lspci -n | grep 03:00.0 03:00.0 0200: 17d5:5833 (rev 02) #echo "17d5 5833" >/sys/bus/pci/drivers/pci-stub/new_id #echo 0000:03:00.0 >/sys/bus/pci/devices/0000\:03\:00.0/driver/unbind #echo 0000:03:00.0 >/sys/bus/pci/drivers/pci-stub/bind 3.Boot guest with an device assignment 03:00.0 #/usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+x2apic -balloon none -no-kvm-pit-reinjection -rtc base=localtime,clock=host,driftfix=slew -usbdevice tablet -drive file=/root/rhel6.0_64.qcow2,if=none,format=qcow2,cache=none,werror=stop,rerror=stop,id=drive-ide0-0-0 -device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device rtl8139,netdev=hostnet0,mac=20:20:78:90:12:93 -boot c -monitor stdio -vnc :10 -device pci-assign,host=03:00.0,id=devicex3100 PCI region 4 at address 0xe4008000 has size 0x100, which is not a multiple of 4K. You might experience some performance hit due to that. QEMU 0.12.1 monitor - type 'help' for more information (qemu) Please note,in guest infos between fixed version and unfixed version. ------fixed version(qemu-kvm-0.12.1.2-2.129.el6)-- In guest,check pass-through device #lspci | grep Et 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20) 00:04.0 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet PCIe (rev 02) #ethtool -i eth3 driver: vxge version: 2.0.8.20182-k firmware-version: 1.4.4 bus-info: 0000:00:04.0 #ifconfig -a eth2 Link encap:Ethernet HWaddr 20:20:78:90:12:93 inet addr:10.66.72.69 Bcast:10.66.73.255 Mask:255.255.254.0 inet6 addr: fe80::2220:78ff:fe90:1293/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:899 errors:0 dropped:0 overruns:0 frame:0 TX packets:142 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:87573 (85.5 KiB) TX bytes:20963 (20.4 KiB) Interrupt:10 eth3 Link encap:Ethernet HWaddr 00:0C:FC:01:46:5C UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:11 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:18 errors:0 dropped:0 overruns:0 frame:0 TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1060 (1.0 KiB) TX bytes:1060 (1.0 KiB) virbr0 Link encap:Ethernet HWaddr 8E:D1:83:B1:AB:20 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:20 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:4046 (3.9 KiB) ------unfixed version(qemu-kvm-0.12.1.2-2.126.el6)-- In guest,check pass-through device #lspci | grep Et 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20) 00:04.0 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet PCIe (rev 02) #After issue "ifconfig -a" didn't find vxge device. ifconfig -a eth2 Link encap:Ethernet HWaddr 20:20:78:90:12:93 inet addr:10.66.72.69 Bcast:10.66.73.255 Mask:255.255.254.0 inet6 addr: fe80::2220:78ff:fe90:1293/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:149 errors:0 dropped:0 overruns:0 frame:0 TX packets:102 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:21651 (21.1 KiB) TX bytes:16443 (16.0 KiB) Interrupt:10 Base address:0x8000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:18 errors:0 dropped:0 overruns:0 frame:0 TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1060 (1.0 KiB) TX bytes:1060 (1.0 KiB) virbr0 Link encap:Ethernet HWaddr F2:62:00:CA:7F:9F inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:16 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:3718 (3.6 KiB) Would you please tell me how can I switch Neterion Inc's X3100 in SR-IOv mode? I tried comment11 Alex steps,download firmware from ftp://ftp.s2io.com/pub/X3100-Drivers/FIRMWARE/archive/,however,didn't find ./vxge_configure #ll total 19212 -rw-r--r--. 1 13110 juzhang 10598400 Aug 5 00:46 registerdb -rw-r--r--. 1 root root 4121570 Jan 19 08:41 REL_1.4.4_FW-1.59_Linux.tar.gz -rwxr-xr-x. 1 13110 juzhang 4620309 Aug 5 00:46 vxge_update -rw-r--r--. 1 root root 322949 Oct 10 21:35 X3100_2009_09_16_REV_1_4_4_uni.ncf thanks in advance! Best Regards, Junyi (In reply to comment #26) > Both tested on fixed version(qemu-kvm-0.12.1.2-2.129.el6) and on unfixed > version(qemu-kvm-0.12.1.2-2.127.el6),both can pass-through nics to guest. can > not produced this issue.the following is detailed info,any mistake,please > correct me. Can you double check your versions, this is fixed in qemu-kvm-0.12.1.2-2.127.el6, so it should work the same in both of the versions you list above. > Please note,in guest infos between fixed version and unfixed version. > > ------fixed version(qemu-kvm-0.12.1.2-2.129.el6)-- Here you're testing the fixed version and it seems to work. > In guest,check pass-through device > #lspci | grep Et > 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > RTL-8139/8139C/8139C+ (rev 20) > 00:04.0 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet > PCIe (rev 02) > #ethtool -i eth3 > driver: vxge > version: 2.0.8.20182-k > firmware-version: 1.4.4 > bus-info: 0000:00:04.0 > #ifconfig -a > eth2 Link encap:Ethernet HWaddr 20:20:78:90:12:93 > inet addr:10.66.72.69 Bcast:10.66.73.255 Mask:255.255.254.0 > inet6 addr: fe80::2220:78ff:fe90:1293/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:899 errors:0 dropped:0 overruns:0 frame:0 > TX packets:142 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:87573 (85.5 KiB) TX bytes:20963 (20.4 KiB) > Interrupt:10 > > eth3 Link encap:Ethernet HWaddr 00:0C:FC:01:46:5C > UP BROADCAST MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > Interrupt:11 > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:16436 Metric:1 > RX packets:18 errors:0 dropped:0 overruns:0 frame:0 > TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:1060 (1.0 KiB) TX bytes:1060 (1.0 KiB) > > virbr0 Link encap:Ethernet HWaddr 8E:D1:83:B1:AB:20 > inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:20 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:4046 (3.9 KiB) > > ------unfixed version(qemu-kvm-0.12.1.2-2.126.el6)-- Now you've switch to a version prior to the fix... > In guest,check pass-through device > #lspci | grep Et > 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > RTL-8139/8139C/8139C+ (rev 20) > 00:04.0 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet > PCIe (rev 02) > #After issue "ifconfig -a" didn't find vxge device. And this is expected because vxge won't claim the device in the guest prior to 127. > ifconfig -a > eth2 Link encap:Ethernet HWaddr 20:20:78:90:12:93 > inet addr:10.66.72.69 Bcast:10.66.73.255 Mask:255.255.254.0 > inet6 addr: fe80::2220:78ff:fe90:1293/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:149 errors:0 dropped:0 overruns:0 frame:0 > TX packets:102 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:21651 (21.1 KiB) TX bytes:16443 (16.0 KiB) > Interrupt:10 Base address:0x8000 > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:16436 Metric:1 > RX packets:18 errors:0 dropped:0 overruns:0 frame:0 > TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:1060 (1.0 KiB) TX bytes:1060 (1.0 KiB) > > virbr0 Link encap:Ethernet HWaddr F2:62:00:CA:7F:9F > inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:16 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:3718 (3.6 KiB) > > > Would you please tell me how can I switch Neterion Inc's X3100 in SR-IOv mode? I'll forward the necessary driver. (In reply to comment #27) > > Can you double check your versions, this is fixed in > qemu-kvm-0.12.1.2-2.127.el6, so it should work the same in both of the versions Double check qemu-kvm version,I tested on fixed version(qemu-kvm-0.12.1.2-2.129.el6) and unfixed version(qemu-kvm-0.12.1.2-2.126.el6). (In reply to comment #28) > (In reply to comment #27) > > > > Can you double check your versions, this is fixed in > > qemu-kvm-0.12.1.2-2.127.el6, so it should work the same in both of the versions > Double check qemu-kvm version,I tested on fixed > version(qemu-kvm-0.12.1.2-2.129.el6) and unfixed > version(qemu-kvm-0.12.1.2-2.126.el6). Your testing seems to confirm the following: - qemu-kvm-0.12.1.2-2.126.el6: device does NOT work in guest, unclaimed by vxge driver - qemu-kvm-0.12.1.2-2.129.el6: device is fully functional in guest So while we can't reproduce the guest failing to boot with the device assigned (which must have been fixed by some previous change), there is an obvious improvement in the support of this device with these fixes, correct? I think if we can confirm that the device is now functional in the guest, it's sufficient to verify this bug. Thanks. *** Bug 643901 has been marked as a duplicate of this bug. *** Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: KVM device assignment: Improved PCI configuration space support enabling device assignment to work with a broader assortment of devices, including the Exar X3100 series 10GbE cards. Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -KVM device assignment: Improved PCI configuration space support enabling device assignment to work with a broader assortment of devices, including the Exar X3100 series 10GbE cards.+PCI configuration space access is improved, enabling a broader set of PCI devices to be device-assigned to guest VMs. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0534.html |