RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1045827 - can't hotplug pci device to pci-bridge (bus 1) in qemu monitor
Summary: can't hotplug pci device to pci-bridge (bus 1) in qemu monitor
Keywords:
Status: CLOSED DUPLICATE of bug 953008
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.0
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: Amos Kong
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-22 10:42 UTC by Jun Li
Modified: 2023-09-14 01:55 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-27 02:59:39 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Jun Li 2013-12-22 10:42:20 UTC 
Description of problem:
Intel 82599NB vf can not be found inside guest after hot-plug with pci-bridge. 
BTW, Intel 82599NB vf can not be found and got IP inside guest after hot-plug with pci.

Version-Release number of selected component (if applicable):
3.10.0-61.el7.x86_64
qemu-kvm-1.5.3-30.el7.x86_64
guest:
win2012R2-64

How reproducible:
100%

Steps to Reproduce:
1.Boot guest with following cmd:
<cli>:
# /usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew \
-device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x3 \
-device virtio-scsi-pci,bus=bridge1,addr=0x5,id=scsi0 \
-drive file=/home/juli/win2012r2.raw,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=raw,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 \
-drive file=/home/juli/ISO/en_windows_server_2012_r2_x64_dvd_2707946.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-0 -device ide-drive,drive=drive-ide1-0-0,id=ide1-0-0,bus=ide.0,unit=0,bootindex=4 \
-drive file=/home/juli/virtio-win-prewhql-0.1-74.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-1 -device ide-drive,drive=drive-ide1-0-1,id=ide1-0-1,bus=ide.0,unit=1,bootindex=6 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -netdev tap,id=tap1,vhost=on,queues=4,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-juli -device virtio-net-pci,netdev=tap1,id=nic1,mq=on,vectors=17,mac=1a:59:0a:4b:aa:94,bus=bridge1,addr=0x1f
2.hot-plug a vfio-pci device via qmp.
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF6","bus":"bridge1","addr":"0x13"}}
3.check this vf inside guest and qemu-kvm.

Actual results:
Can not find this vf inside guest.
qemu-kvm can find this device(info pci inside HMP).
(qemu)info pci
  Bus  1, device  19, function 0:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      id "hostnet_VF6"

Expected results:
guest can find and use this vf correctly.

Additional info:
Comment 2 Amos Kong 2013-12-23 03:51:28 UTC 
Lijun, 

It seems not a bug.  Can you retest by enabling "multifunction"?

device_add ....,multifunction=on
Comment 3 Jun Li 2013-12-24 02:48:57 UTC 
(In reply to Amos Kong from comment #2)
> Lijun, 
> 
> It seems not a bug.  Can you retest by enabling "multifunction"?
> 
> device_add ....,multifunction=on

Hi Amos,
   Retest with "multifunction=on", steps as comment 0, got the same result as comment 0.
<cli>:
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.4","id":"hostnet_VF8","bus":"bridge1","addr":"0x14","multifunction": "on"}}
Actual results:
Can not find this vf inside guest.
qemu-kvm can find this device(info pci inside HMP).
(qemu)info pci
  Bus  1, device  20, function 0:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      id "hostnet_VF8"
---------
Best Regards,
Jun Li
Comment 4 Amos Kong 2013-12-24 04:47:36 UTC 
Hi lijun,

When we hotplug virtio-nic (multiple functions), one device contains 8 functions, all the functions should be added by 'device_add' command. You only hot-added one function (0), so only one device was listed.

for j in `seq 1 7` 0;do
(qemu) device_add virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
done


You should also find out the pci IDs of VFs, then execute device_add cmds.

device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
...
device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on
Comment 5 Jun Li 2013-12-26 03:03:54 UTC 
(In reply to Amos Kong from comment #4)
> Hi lijun,
> 
> When we hotplug virtio-nic (multiple functions), one device contains 8
> functions, all the functions should be added by 'device_add' command. You
> only hot-added one function (0), so only one device was listed.
> 
> for j in `seq 1 7` 0;do
> (qemu) device_add
> virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
> done
> 
> 
> You should also find out the pci IDs of VFs, then execute device_add cmds.
> 
> device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
> device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
> ...
> device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on
Retest as the followings:
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
---
(qemu) info pci
  Bus  1, device  20, function 1:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      id "hostnet_VF8"
---
Guest can not find this 82599EB vf card. qemu monitor can find this card as above show.
-------
I also test with
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8"}}
Guest and qemu monitor can find and use this vf correctly.
----

Best Regards,
Jun Li
Comment 6 Amos Kong 2013-12-26 05:08:13 UTC 
(In reply to Jun Li from comment #5)
> (In reply to Amos Kong from comment #4)
> > Hi lijun,
> > 
> > When we hotplug virtio-nic (multiple functions), one device contains 8
> > functions, all the functions should be added by 'device_add' command. You
> > only hot-added one function (0), so only one device was listed.
> > 
> > for j in `seq 1 7` 0;do
> > (qemu) device_add
> > virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
> > done
> > 
> > 
> > You should also find out the pci IDs of VFs, then execute device_add cmds.
> > 
> > device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
> > device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
> > ...
> > device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on
> Retest as the followings:
> {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> "id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
> ---
> (qemu) info pci
>   Bus  1, device  20, function 1:
>     Ethernet controller: PCI device 8086:10ed
>       BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
>       BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
>       id "hostnet_VF8"
> ---
> Guest can not find this 82599EB vf card. qemu monitor can find this card as
> above show.

Can you find new vf insider guest by execute "lspci |grep Eth" ?
and disabled interfaces in the output of # ifconfig -a ?

> -------
> I also test with
> {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> "id":"hostnet_VF8"}}
> Guest and qemu monitor can find and use this vf correctly.
> ----

It's expected.
Comment 7 Jun Li 2013-12-26 09:18:13 UTC 
(In reply to Amos Kong from comment #6)
> (In reply to Jun Li from comment #5)
> > (In reply to Amos Kong from comment #4)
> > > Hi lijun,
> > > 
> > > When we hotplug virtio-nic (multiple functions), one device contains 8
> > > functions, all the functions should be added by 'device_add' command. You
> > > only hot-added one function (0), so only one device was listed.
> > > 
> > > for j in `seq 1 7` 0;do
> > > (qemu) device_add
> > > virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
> > > done
> > > 
> > > 
> > > You should also find out the pci IDs of VFs, then execute device_add cmds.
> > > 
> > > device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
> > > device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
> > > ...
> > > device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on
> > Retest as the followings:
> > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> > "id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
> > ---
> > (qemu) info pci
> >   Bus  1, device  20, function 1:
> >     Ethernet controller: PCI device 8086:10ed
> >       BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
> >       BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
> >       id "hostnet_VF8"
> > ---
> > Guest can not find this 82599EB vf card. qemu monitor can find this card as
> > above show.
> 
> Can you find new vf insider guest by execute "lspci |grep Eth" ?
> and disabled interfaces in the output of # ifconfig -a ?
when file this bug, guest is win2012R2-64.
Retest it with RHEL-7.0 guest:
---
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
---
# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.110.31  netmask 255.255.248.0  broadcast 10.66.111.255
        inet6 fe80::1859:aff:fe4b:aa94  prefixlen 64  scopeid 0x20<link>
        ether 1a:59:0a:4b:aa:94  txqueuelen 1000  (Ethernet)
        RX packets 3377  bytes 260450 (254.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 91  bytes 15812 (15.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 42  bytes 4460 (4.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 42  bytes 4460 (4.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
---
No 82599EB vf card.
# lspci |grep Eth
01:1f.0 Ethernet controller: Red Hat, Inc Virtio network device
---
Also no 82599 vf card.
-----
# lsmod |grep ixgbe
ixgbevf                50546  0 
ixgbe                 232897  0 
ptp                    18580  1 ixgbe
mdio                   13807  1 ixgbe
dca                    15130  1 ixgbe
---
So guest has ixgbevf driver.
> 
> > -------
> > I also test with
> > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> > "id":"hostnet_VF8"}}
> > Guest and qemu monitor can find and use this vf correctly.
> > ----
> 
> It's expected.
Comment 8 Amos Kong 2013-12-27 01:00:02 UTC 
If we plug multiple function devices, function 0 should always be plugged if we want to make other function to be identified by guest.

For example:
  plug a vf function to 0x12.1
  plug a vf function to 0x12.2
 (no pci device appears in guest : lspci)
  plug a vf function to 0x12.0
 (all three functions appear in guest)

# lspci|grep Eth
00:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)


1) I tried to plug two VFs to 0x9.1, 0x9.0

(qemu) device_add vfio-pci,host=05:11.1,id=hostnet_VF001,addr=0x9.1,multifunction=on
(qemu) device_add vfio-pci,host=05:11.2,id=hostnet_VF000,addr=0x9.0,multifunction=on

2) check pci information in qemu monitor

(qemu) info pci
  Bus  0, device   9, function 0:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0x80018000 [0x8001bfff].
      BAR3: 64 bit memory at 0x8001c000 [0x8001ffff].
      id "hostnet_VF000"
  Bus  0, device   9, function 1:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0x80020000 [0x80023fff].
      BAR3: 64 bit memory at 0x80024000 [0x80027fff].
      id "hostnet_VF001"

3) # lspci|grep Eth
00:09.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:09.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)


4) two net interfaces appear in ifconfig output
[root@dhcp-10-204 ~]# ethtool -i ens9f1
driver: ixgbevf
version: 2.7.12-k
firmware-version: 
bus-info: 0000:00:09.1 <<<<<<
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@dhcp-10-204 ~]# ethtool -i ens9f0
driver: ixgbevf
version: 2.7.12-k
firmware-version: 
bus-info: 0000:00:09.0 <<<<<
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
Comment 9 Amos Kong 2013-12-27 02:47:59 UTC 
# qemu-kvm ... \
   -device ,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x3 \
   -device vfio-pci,host=05:11.1,id=vfiopci01,addr=0x13,bus=bridge1 <<< assigned pci dev to bus 1 by cmdline

I tried to add a vfio-pci device to (bus 1, 0x13) in commandline, it can be identified in guest.

01:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

===

The problem in Comment #0 is that it's fail to plug a pci device to pci-bridge bus(bus 1).

I also tested with virtio-net-pci.

# device_add virtio-net-pci,bus=pci.0,addr=0x11
  can be identified by guest

# device_add virtio-net-pci,bus=bridge1,addr=0x12
  can only be found in "info pci"

# qemu -device virtio-net-pci,bus=bridge1,addr=0x7 ...  <<< assigned pci dev to bus 1 by cmdline
  can be identified by guest
Comment 10 Amos Kong 2013-12-27 02:54:00 UTC 
http://www.linux-kvm.org/page/PCITodo

Hotplug for devices behind PCI bridges
 Issue: QEMU lacks support for device hotplug behind
 pci bridges.
  Plan:
   - each bus gets assigned a number 0-255
   - generated ACPI code writes this number
     to a new BSEL register, then uses existing
     UP/DOWN registers to probe slot status;
     to eject, write number to BSEL register,
     then slot into existing EJ
   This is to address the ACPI spec requirement to
   avoid config cycle access to any bus except PCI roots.
   Note: ACPI doesn't support adding or removing bridges by hotplug.
   We should and prevent removal of bridges by hotplug,
   unless they were added by hotplug previously
   (and so, are not described by ACPI).
 Developer: Michael S. Tsirkin

what's the status of this task?
Comment 11 Qian Guo 2013-12-27 02:59:01 UTC 
Seams dupl with bug #953008 , I will close it.

If there's something wrong I made, please re-open it.
Comment 12 Qian Guo 2013-12-27 02:59:39 UTC 

*** This bug has been marked as a duplicate of bug 953008 ***
Comment 13 Red Hat Bugzilla 2023-09-14 01:55:57 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.