Bug 678368

*** Bug 691328 has been marked as a duplicate of this bug. ***

*** Bug 968375 has been marked as a duplicate of this bug. ***

Verify this bug:
Version-Release number of selected component (if applicable):
qemu-kvm-1.5.3-30.el7.x86_64
3.10.0-61.el7.x86_64
---
Steps as followings:
<cli>:
1, boot guest with following command:
# gdb --args /usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/win2012r2.raw,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=raw,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -drive file=/home/juli/ISO/en_windows_server_2012_r2_x64_dvd_2707946.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-0 -device ide-drive,drive=drive-ide1-0-0,id=ide1-0-0,bus=ide.0,unit=0,bootindex=4 -drive file=/home/juli/virtio-win-prewhql-0.1-74.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-1 -device ide-drive,drive=drive-ide1-0-1,id=ide1-0-1,bus=ide.0,unit=1,bootindex=6 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -netdev tap,id=tap1,vhost=on,queues=4,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-juli -device virtio-net-pci,netdev=tap1,id=nic1,mq=on,vectors=17,mac=1a:59:0a:4b:aa:94
-----
2, run add vf with the following script.
# cat add-vf.sh 
#! /bin/sh
i=0
while ((i < 9)); do
    for ((j=0;j<7;j=j+2)); do
        echo "device_add vfio-pci,host=05:1$i.$j,id=vf_$i$j" | nc -U /tmp/monitor1
        sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        #((j=j+2))
    done
    ((i=i+1))
done
---
3, Check vfs in guest.

Actual results:
(qemu) device_add vfio-pci,host=05:16.2,id=vf_62
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:16.4,id=vf_64
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:16.6,id=vf_66
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.0,id=vf_70
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.2,id=vf_72
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.4,id=vf_74
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.6,id=vf_76
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.0,id=vf_80
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.2,id=vf_82
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.4,id=vf_84
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.6,id=vf_86
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
-----
So can only hot-plug 26 vfs. Check inside win2012r2 guest only has 26 82599 vfs. 
BTW, when hot-plug vfs more than 26, qemu-kvm will give error, but works well. No core.

Base on above test, this bug has been verified.

Hi Alex,
   For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices not 32. Could you give some explanations ? Thank you.

Best Regards,
Jun Li

(In reply to Jun Li from comment #12)
> Hi Alex,
>    For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices
> not 32. Could you give some explanations ? Thank you.

You're hitting the PCI slot limit, not the VFIO device limit.  You therefore either need to assign some devices statically (cold-plug) and make use of multifunction support or you need to add a PCI-to-PCI bridge to the VM to get more slots on a secondary bus.

(In reply to Alex Williamson from comment #13)
> (In reply to Jun Li from comment #12)
> > Hi Alex,
> >    For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices
> > not 32. Could you give some explanations ? Thank you.
> 
> You're hitting the PCI slot limit, not the VFIO device limit.  You therefore
> either need to assign some devices statically (cold-plug) and make use of
> multifunction support or you need to add a PCI-to-PCI bridge to the VM to
> get more slots on a secondary bus.

boot guest with 36 vfs:
1, generate boot rhel7 guest commands:
# cat generate-boot-rhel7-cli.sh 
#! /bin/sh
i=0
num=1
n=1;
echo "#! /bin/sh" > boot_a_rhel7.sh
echo "/usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/RHEL-Server-7.0-64.qcow2_v3,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=qcow2,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x8 -device pci-bridge,bus=pci.0,id=bridge2,chassis_nr=1,addr=0x9 \\" >> boot_a_rhel7.sh
while ((i<9)); do
    for ((j=0;j<7;j=j+2)); do
        if ((i>5)); then
            num=2;
        fi
        if ((n==32)); then
            n=1;
        fi
        n=`printf "%x" $n`
        if ((i==8 && j==6)); then
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n"  >> boot_a_rhel7.sh
        else 
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n \\"  >> boot_a_rhel7.sh
        fi
        #echo $i$j
        #sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        n=$((0x$n))
        ((n=n+1))
    done
    ((i=i+1))
done
chmod +x boot_a_rhel7.sh
---
2, boot guest with "boot_a_rhel7.sh".
# ./boot_a_rhel7.sh 
QEMU 1.5.3 monitor - type 'help' for more information
(qemu) qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: vfio: Maximum supported vfio devices (32) already attached

qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: Device initialization failed.
qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: Device 'vfio-pci' could not be initialized
---
As the comment 9, for RHEL 7.0 w/ VFIO the intended limit is 32 devices.
===========
Boot guest with 32 vfs:
1, generate boot rhel7 guest commands:
# cat generate-boot-rhel7-cli.sh 
#! /bin/sh
i=0
num=1
n=1;
echo "#! /bin/sh" > boot_a_rhel7.sh
echo "/usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/RHEL-Server-7.0-64.qcow2_v3,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=qcow2,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x8 -device pci-bridge,bus=pci.0,id=bridge2,chassis_nr=1,addr=0x9 \\" >> boot_a_rhel7.sh
while ((i<8)); do
    for ((j=0;j<7;j=j+2)); do
        if ((i>5)); then
            num=2;
        fi
        if ((n==32)); then
            n=1;
        fi
        n=`printf "%x" $n`
        if ((i==7 && j==6)); then
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n"  >> boot_a_rhel7.sh
        else 
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n \\"  >> boot_a_rhel7.sh
        fi
        #echo $i$j
        #sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        n=$((0x$n))
        ((n=n+1))
    done
    ((i=i+1))
done
chmod +x boot_a_rhel7.sh
-----
2, Boot guest with "boot_a_rhel7.sh"
# ./boot_a_rhel7.sh
---
3, check vfs inside guest and qemu monitor.

Actual results:
inside guest and qemu monitor can find these 32 vfs. vfs inside guest can obtain IP and ping other host.
==========
Based on above test, this bug has been fixed.

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.