| Summary: | can't hotplug pci device to pci-bridge (bus 1) in qemu monitor | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jun Li <juli> |
| Component: | qemu-kvm | Assignee: | Amos Kong <akong> |
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.0 | CC: | acathrow, ailan, akong, alex.williamson, chayang, hhuang, juli, juzhang, mst, qiguo, rhod, virt-maint |
| Target Milestone: | rc | Flags: | akong:
needinfo?
(mst) |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-12-27 02:59:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Lijun, It seems not a bug. Can you retest by enabling "multifunction"? device_add ....,multifunction=on (In reply to Amos Kong from comment #2) > Lijun, > > It seems not a bug. Can you retest by enabling "multifunction"? > > device_add ....,multifunction=on Hi Amos, Retest with "multifunction=on", steps as comment 0, got the same result as comment 0. <cli>: {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.4","id":"hostnet_VF8","bus":"bridge1","addr":"0x14","multifunction": "on"}} Actual results: Can not find this vf inside guest. qemu-kvm can find this device(info pci inside HMP). (qemu)info pci Bus 1, device 20, function 0: Ethernet controller: PCI device 8086:10ed BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. id "hostnet_VF8" --------- Best Regards, Jun Li Hi lijun, When we hotplug virtio-nic (multiple functions), one device contains 8 functions, all the functions should be added by 'device_add' command. You only hot-added one function (0), so only one device was listed. for j in `seq 1 7` 0;do (qemu) device_add virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on done You should also find out the pci IDs of VFs, then execute device_add cmds. device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on ... device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on (In reply to Amos Kong from comment #4) > Hi lijun, > > When we hotplug virtio-nic (multiple functions), one device contains 8 > functions, all the functions should be added by 'device_add' command. You > only hot-added one function (0), so only one device was listed. > > for j in `seq 1 7` 0;do > (qemu) device_add > virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on > done > > > You should also find out the pci IDs of VFs, then execute device_add cmds. > > device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on > device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on > ... > device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on Retest as the followings: {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}} --- (qemu) info pci Bus 1, device 20, function 1: Ethernet controller: PCI device 8086:10ed BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. id "hostnet_VF8" --- Guest can not find this 82599EB vf card. qemu monitor can find this card as above show. ------- I also test with {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8"}} Guest and qemu monitor can find and use this vf correctly. ---- Best Regards, Jun Li (In reply to Jun Li from comment #5) > (In reply to Amos Kong from comment #4) > > Hi lijun, > > > > When we hotplug virtio-nic (multiple functions), one device contains 8 > > functions, all the functions should be added by 'device_add' command. You > > only hot-added one function (0), so only one device was listed. > > > > for j in `seq 1 7` 0;do > > (qemu) device_add > > virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on > > done > > > > > > You should also find out the pci IDs of VFs, then execute device_add cmds. > > > > device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on > > device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on > > ... > > device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on > Retest as the followings: > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2", > "id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}} > --- > (qemu) info pci > Bus 1, device 20, function 1: > Ethernet controller: PCI device 8086:10ed > BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. > BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. > id "hostnet_VF8" > --- > Guest can not find this 82599EB vf card. qemu monitor can find this card as > above show. Can you find new vf insider guest by execute "lspci |grep Eth" ? and disabled interfaces in the output of # ifconfig -a ? > ------- > I also test with > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2", > "id":"hostnet_VF8"}} > Guest and qemu monitor can find and use this vf correctly. > ---- It's expected. (In reply to Amos Kong from comment #6) > (In reply to Jun Li from comment #5) > > (In reply to Amos Kong from comment #4) > > > Hi lijun, > > > > > > When we hotplug virtio-nic (multiple functions), one device contains 8 > > > functions, all the functions should be added by 'device_add' command. You > > > only hot-added one function (0), so only one device was listed. > > > > > > for j in `seq 1 7` 0;do > > > (qemu) device_add > > > virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on > > > done > > > > > > > > > You should also find out the pci IDs of VFs, then execute device_add cmds. > > > > > > device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on > > > device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on > > > ... > > > device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on > > Retest as the followings: > > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2", > > "id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}} > > --- > > (qemu) info pci > > Bus 1, device 20, function 1: > > Ethernet controller: PCI device 8086:10ed > > BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. > > BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. > > id "hostnet_VF8" > > --- > > Guest can not find this 82599EB vf card. qemu monitor can find this card as > > above show. > > Can you find new vf insider guest by execute "lspci |grep Eth" ? > and disabled interfaces in the output of # ifconfig -a ? when file this bug, guest is win2012R2-64. Retest it with RHEL-7.0 guest: --- {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}} --- # ifconfig -a eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.66.110.31 netmask 255.255.248.0 broadcast 10.66.111.255 inet6 fe80::1859:aff:fe4b:aa94 prefixlen 64 scopeid 0x20<link> ether 1a:59:0a:4b:aa:94 txqueuelen 1000 (Ethernet) RX packets 3377 bytes 260450 (254.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91 bytes 15812 (15.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 42 bytes 4460 (4.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 42 bytes 4460 (4.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 --- No 82599EB vf card. # lspci |grep Eth 01:1f.0 Ethernet controller: Red Hat, Inc Virtio network device --- Also no 82599 vf card. ----- # lsmod |grep ixgbe ixgbevf 50546 0 ixgbe 232897 0 ptp 18580 1 ixgbe mdio 13807 1 ixgbe dca 15130 1 ixgbe --- So guest has ixgbevf driver. > > > ------- > > I also test with > > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2", > > "id":"hostnet_VF8"}} > > Guest and qemu monitor can find and use this vf correctly. > > ---- > > It's expected. If we plug multiple function devices, function 0 should always be plugged if we want to make other function to be identified by guest.
For example:
plug a vf function to 0x12.1
plug a vf function to 0x12.2
(no pci device appears in guest : lspci)
plug a vf function to 0x12.0
(all three functions appear in guest)
# lspci|grep Eth
00:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
1) I tried to plug two VFs to 0x9.1, 0x9.0
(qemu) device_add vfio-pci,host=05:11.1,id=hostnet_VF001,addr=0x9.1,multifunction=on
(qemu) device_add vfio-pci,host=05:11.2,id=hostnet_VF000,addr=0x9.0,multifunction=on
2) check pci information in qemu monitor
(qemu) info pci
Bus 0, device 9, function 0:
Ethernet controller: PCI device 8086:10ed
BAR0: 64 bit memory at 0x80018000 [0x8001bfff].
BAR3: 64 bit memory at 0x8001c000 [0x8001ffff].
id "hostnet_VF000"
Bus 0, device 9, function 1:
Ethernet controller: PCI device 8086:10ed
BAR0: 64 bit memory at 0x80020000 [0x80023fff].
BAR3: 64 bit memory at 0x80024000 [0x80027fff].
id "hostnet_VF001"
3) # lspci|grep Eth
00:09.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:09.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
4) two net interfaces appear in ifconfig output
[root@dhcp-10-204 ~]# ethtool -i ens9f1
driver: ixgbevf
version: 2.7.12-k
firmware-version:
bus-info: 0000:00:09.1 <<<<<<
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@dhcp-10-204 ~]# ethtool -i ens9f0
driver: ixgbevf
version: 2.7.12-k
firmware-version:
bus-info: 0000:00:09.0 <<<<<
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
# qemu-kvm ... \ -device ,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x3 \ -device vfio-pci,host=05:11.1,id=vfiopci01,addr=0x13,bus=bridge1 <<< assigned pci dev to bus 1 by cmdline I tried to add a vfio-pci device to (bus 1, 0x13) in commandline, it can be identified in guest. 01:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) === The problem in Comment #0 is that it's fail to plug a pci device to pci-bridge bus(bus 1). I also tested with virtio-net-pci. # device_add virtio-net-pci,bus=pci.0,addr=0x11 can be identified by guest # device_add virtio-net-pci,bus=bridge1,addr=0x12 can only be found in "info pci" # qemu -device virtio-net-pci,bus=bridge1,addr=0x7 ... <<< assigned pci dev to bus 1 by cmdline can be identified by guest http://www.linux-kvm.org/page/PCITodo Hotplug for devices behind PCI bridges Issue: QEMU lacks support for device hotplug behind pci bridges. Plan: - each bus gets assigned a number 0-255 - generated ACPI code writes this number to a new BSEL register, then uses existing UP/DOWN registers to probe slot status; to eject, write number to BSEL register, then slot into existing EJ This is to address the ACPI spec requirement to avoid config cycle access to any bus except PCI roots. Note: ACPI doesn't support adding or removing bridges by hotplug. We should and prevent removal of bridges by hotplug, unless they were added by hotplug previously (and so, are not described by ACPI). Developer: Michael S. Tsirkin what's the status of this task? Seams dupl with bug #953008 , I will close it. If there's something wrong I made, please re-open it. *** This bug has been marked as a duplicate of bug 953008 *** |
Description of problem: Intel 82599NB vf can not be found inside guest after hot-plug with pci-bridge. BTW, Intel 82599NB vf can not be found and got IP inside guest after hot-plug with pci. Version-Release number of selected component (if applicable): 3.10.0-61.el7.x86_64 qemu-kvm-1.5.3-30.el7.x86_64 guest: win2012R2-64 How reproducible: 100% Steps to Reproduce: 1.Boot guest with following cmd: <cli>: # /usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew \ -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x3 \ -device virtio-scsi-pci,bus=bridge1,addr=0x5,id=scsi0 \ -drive file=/home/juli/win2012r2.raw,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=raw,werror=stop,rerror=stop,aio=native -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 \ -drive file=/home/juli/ISO/en_windows_server_2012_r2_x64_dvd_2707946.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-0 -device ide-drive,drive=drive-ide1-0-0,id=ide1-0-0,bus=ide.0,unit=0,bootindex=4 \ -drive file=/home/juli/virtio-win-prewhql-0.1-74.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-1 -device ide-drive,drive=drive-ide1-0-1,id=ide1-0-1,bus=ide.0,unit=1,bootindex=6 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -netdev tap,id=tap1,vhost=on,queues=4,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-juli -device virtio-net-pci,netdev=tap1,id=nic1,mq=on,vectors=17,mac=1a:59:0a:4b:aa:94,bus=bridge1,addr=0x1f 2.hot-plug a vfio-pci device via qmp. {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF6","bus":"bridge1","addr":"0x13"}} 3.check this vf inside guest and qemu-kvm. Actual results: Can not find this vf inside guest. qemu-kvm can find this device(info pci inside HMP). (qemu)info pci Bus 1, device 19, function 0: Ethernet controller: PCI device 8086:10ed BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe]. id "hostnet_VF6" Expected results: guest can find and use this vf correctly. Additional info: