Bug 1045827

Lijun, 

It seems not a bug.  Can you retest by enabling "multifunction"?

device_add ....,multifunction=on

(In reply to Amos Kong from comment #2)
> Lijun, 
> 
> It seems not a bug.  Can you retest by enabling "multifunction"?
> 
> device_add ....,multifunction=on

Hi Amos,
   Retest with "multifunction=on", steps as comment 0, got the same result as comment 0.
<cli>:
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.4","id":"hostnet_VF8","bus":"bridge1","addr":"0x14","multifunction": "on"}}
Actual results:
Can not find this vf inside guest.
qemu-kvm can find this device(info pci inside HMP).
(qemu)info pci
  Bus  1, device  20, function 0:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      id "hostnet_VF8"
---------
Best Regards,
Jun Li

Hi lijun,

When we hotplug virtio-nic (multiple functions), one device contains 8 functions, all the functions should be added by 'device_add' command. You only hot-added one function (0), so only one device was listed.

for j in `seq 1 7` 0;do
(qemu) device_add virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
done


You should also find out the pci IDs of VFs, then execute device_add cmds.

device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
...
device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on

(In reply to Amos Kong from comment #4)
> Hi lijun,
> 
> When we hotplug virtio-nic (multiple functions), one device contains 8
> functions, all the functions should be added by 'device_add' command. You
> only hot-added one function (0), so only one device was listed.
> 
> for j in `seq 1 7` 0;do
> (qemu) device_add
> virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
> done
> 
> 
> You should also find out the pci IDs of VFs, then execute device_add cmds.
> 
> device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
> device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
> ...
> device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on
Retest as the followings:
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
---
(qemu) info pci
  Bus  1, device  20, function 1:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
      id "hostnet_VF8"
---
Guest can not find this 82599EB vf card. qemu monitor can find this card as above show.
-------
I also test with
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8"}}
Guest and qemu monitor can find and use this vf correctly.
----

Best Regards,
Jun Li

(In reply to Jun Li from comment #5)
> (In reply to Amos Kong from comment #4)
> > Hi lijun,
> > 
> > When we hotplug virtio-nic (multiple functions), one device contains 8
> > functions, all the functions should be added by 'device_add' command. You
> > only hot-added one function (0), so only one device was listed.
> > 
> > for j in `seq 1 7` 0;do
> > (qemu) device_add
> > virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
> > done
> > 
> > 
> > You should also find out the pci IDs of VFs, then execute device_add cmds.
> > 
> > device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
> > device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
> > ...
> > device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on
> Retest as the followings:
> {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> "id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
> ---
> (qemu) info pci
>   Bus  1, device  20, function 1:
>     Ethernet controller: PCI device 8086:10ed
>       BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
>       BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
>       id "hostnet_VF8"
> ---
> Guest can not find this 82599EB vf card. qemu monitor can find this card as
> above show.

Can you find new vf insider guest by execute "lspci |grep Eth" ?
and disabled interfaces in the output of # ifconfig -a ?

> -------
> I also test with
> {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> "id":"hostnet_VF8"}}
> Guest and qemu monitor can find and use this vf correctly.
> ----

It's expected.

(In reply to Amos Kong from comment #6)
> (In reply to Jun Li from comment #5)
> > (In reply to Amos Kong from comment #4)
> > > Hi lijun,
> > > 
> > > When we hotplug virtio-nic (multiple functions), one device contains 8
> > > functions, all the functions should be added by 'device_add' command. You
> > > only hot-added one function (0), so only one device was listed.
> > > 
> > > for j in `seq 1 7` 0;do
> > > (qemu) device_add
> > > virtio-net-pci,netdev=dev5$j,id=nic5$j,addr=0x5.$j,multifunction=on
> > > done
> > > 
> > > 
> > > You should also find out the pci IDs of VFs, then execute device_add cmds.
> > > 
> > > device_add ...... host=05:10.*,...,addr=0x14.1,multifunction=on
> > > device_add ...... host=05:10.*,...,addr=0x14.2,multifunction=on
> > > ...
> > > device_add ...... host=05:10.4,...,addr=0x14.0,multifunction=on
> > Retest as the followings:
> > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> > "id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
> > ---
> > (qemu) info pci
> >   Bus  1, device  20, function 1:
> >     Ethernet controller: PCI device 8086:10ed
> >       BAR0: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
> >       BAR3: 64 bit memory at 0xffffffffffffffff [0x00003ffe].
> >       id "hostnet_VF8"
> > ---
> > Guest can not find this 82599EB vf card. qemu monitor can find this card as
> > above show.
> 
> Can you find new vf insider guest by execute "lspci |grep Eth" ?
> and disabled interfaces in the output of # ifconfig -a ?
when file this bug, guest is win2012R2-64.
Retest it with RHEL-7.0 guest:
---
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2","id":"hostnet_VF8","bus":"bridge1","addr":"0x14.1","multifunction": "on"}}
---
# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.110.31  netmask 255.255.248.0  broadcast 10.66.111.255
        inet6 fe80::1859:aff:fe4b:aa94  prefixlen 64  scopeid 0x20<link>
        ether 1a:59:0a:4b:aa:94  txqueuelen 1000  (Ethernet)
        RX packets 3377  bytes 260450 (254.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 91  bytes 15812 (15.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 42  bytes 4460 (4.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 42  bytes 4460 (4.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
---
No 82599EB vf card.
# lspci |grep Eth
01:1f.0 Ethernet controller: Red Hat, Inc Virtio network device
---
Also no 82599 vf card.
-----
# lsmod |grep ixgbe
ixgbevf                50546  0 
ixgbe                 232897  0 
ptp                    18580  1 ixgbe
mdio                   13807  1 ixgbe
dca                    15130  1 ixgbe
---
So guest has ixgbevf driver.
> 
> > -------
> > I also test with
> > {"execute":"device_add","arguments":{"driver":"vfio-pci","host":"05:10.2",
> > "id":"hostnet_VF8"}}
> > Guest and qemu monitor can find and use this vf correctly.
> > ----
> 
> It's expected.

If we plug multiple function devices, function 0 should always be plugged if we want to make other function to be identified by guest.

For example:
  plug a vf function to 0x12.1
  plug a vf function to 0x12.2
 (no pci device appears in guest : lspci)
  plug a vf function to 0x12.0
 (all three functions appear in guest)

# lspci|grep Eth
00:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)


1) I tried to plug two VFs to 0x9.1, 0x9.0

(qemu) device_add vfio-pci,host=05:11.1,id=hostnet_VF001,addr=0x9.1,multifunction=on
(qemu) device_add vfio-pci,host=05:11.2,id=hostnet_VF000,addr=0x9.0,multifunction=on

2) check pci information in qemu monitor

(qemu) info pci
  Bus  0, device   9, function 0:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0x80018000 [0x8001bfff].
      BAR3: 64 bit memory at 0x8001c000 [0x8001ffff].
      id "hostnet_VF000"
  Bus  0, device   9, function 1:
    Ethernet controller: PCI device 8086:10ed
      BAR0: 64 bit memory at 0x80020000 [0x80023fff].
      BAR3: 64 bit memory at 0x80024000 [0x80027fff].
      id "hostnet_VF001"

3) # lspci|grep Eth
00:09.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:09.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)


4) two net interfaces appear in ifconfig output
[root@dhcp-10-204 ~]# ethtool -i ens9f1
driver: ixgbevf
version: 2.7.12-k
firmware-version: 
bus-info: 0000:00:09.1 <<<<<<
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@dhcp-10-204 ~]# ethtool -i ens9f0
driver: ixgbevf
version: 2.7.12-k
firmware-version: 
bus-info: 0000:00:09.0 <<<<<
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

# qemu-kvm ... \
   -device ,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x3 \
   -device vfio-pci,host=05:11.1,id=vfiopci01,addr=0x13,bus=bridge1 <<< assigned pci dev to bus 1 by cmdline

I tried to add a vfio-pci device to (bus 1, 0x13) in commandline, it can be identified in guest.

01:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

===

The problem in Comment #0 is that it's fail to plug a pci device to pci-bridge bus(bus 1).

I also tested with virtio-net-pci.

# device_add virtio-net-pci,bus=pci.0,addr=0x11
  can be identified by guest

# device_add virtio-net-pci,bus=bridge1,addr=0x12
  can only be found in "info pci"

# qemu -device virtio-net-pci,bus=bridge1,addr=0x7 ...  <<< assigned pci dev to bus 1 by cmdline
  can be identified by guest

http://www.linux-kvm.org/page/PCITodo

Hotplug for devices behind PCI bridges
 Issue: QEMU lacks support for device hotplug behind
 pci bridges.
  Plan:
   - each bus gets assigned a number 0-255
   - generated ACPI code writes this number
     to a new BSEL register, then uses existing
     UP/DOWN registers to probe slot status;
     to eject, write number to BSEL register,
     then slot into existing EJ
   This is to address the ACPI spec requirement to
   avoid config cycle access to any bus except PCI roots.
   Note: ACPI doesn't support adding or removing bridges by hotplug.
   We should and prevent removal of bridges by hotplug,
   unless they were added by hotplug previously
   (and so, are not described by ACPI).
 Developer: Michael S. Tsirkin

what's the status of this task?

Seams dupl with bug #953008 , I will close it.

If there's something wrong I made, please re-open it.

*** This bug has been marked as a duplicate of bug 953008 ***

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days