Bug 678368 - RFE: Support more than 8 assigned devices
RFE: Support more than 8 assigned devices
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.0
x86_64 Linux
low Severity medium
: rc
: 7.0
Assigned To: Bandan Das
Virtualization Bugs
: FutureFeature, Triaged
: 691328 968375 (view as bug list)
Depends On:
Blocks: 817995
  Show dependency treegraph
 
Reported: 2011-02-17 12:34 EST by Alex Williamson
Modified: 2014-06-17 23:12 EDT (History)
24 users (show)

See Also:
Fixed In Version: qemu-kvm-1.5.3-26.el7
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 670787
Environment:
Last Closed: 2014-06-13 07:59:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 2 Alex Williamson 2011-03-28 09:43:23 EDT
*** Bug 691328 has been marked as a duplicate of this bug. ***
Comment 8 Alex Williamson 2013-06-07 12:51:56 EDT
*** Bug 968375 has been marked as a duplicate of this bug. ***
Comment 12 Jun Li 2013-12-20 03:15:57 EST
Verify this bug:
Version-Release number of selected component (if applicable):
qemu-kvm-1.5.3-30.el7.x86_64
3.10.0-61.el7.x86_64
---
Steps as followings:
<cli>:
1, boot guest with following command:
# gdb --args /usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/win2012r2.raw,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=raw,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -drive file=/home/juli/ISO/en_windows_server_2012_r2_x64_dvd_2707946.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-0 -device ide-drive,drive=drive-ide1-0-0,id=ide1-0-0,bus=ide.0,unit=0,bootindex=4 -drive file=/home/juli/virtio-win-prewhql-0.1-74.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-1 -device ide-drive,drive=drive-ide1-0-1,id=ide1-0-1,bus=ide.0,unit=1,bootindex=6 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -netdev tap,id=tap1,vhost=on,queues=4,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-juli -device virtio-net-pci,netdev=tap1,id=nic1,mq=on,vectors=17,mac=1a:59:0a:4b:aa:94
-----
2, run add vf with the following script.
# cat add-vf.sh 
#! /bin/sh
i=0
while ((i < 9)); do
    for ((j=0;j<7;j=j+2)); do
        echo "device_add vfio-pci,host=05:1$i.$j,id=vf_$i$j" | nc -U /tmp/monitor1
        sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        #((j=j+2))
    done
    ((i=i+1))
done
---
3, Check vfs in guest.

Actual results:
(qemu) device_add vfio-pci,host=05:16.2,id=vf_62
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:16.4,id=vf_64
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:16.6,id=vf_66
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.0,id=vf_70
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.2,id=vf_72
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.4,id=vf_74
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.6,id=vf_76
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.0,id=vf_80
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.2,id=vf_82
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.4,id=vf_84
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.6,id=vf_86
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
-----
So can only hot-plug 26 vfs. Check inside win2012r2 guest only has 26 82599 vfs. 
BTW, when hot-plug vfs more than 26, qemu-kvm will give error, but works well. No core.

Base on above test, this bug has been verified.

Hi Alex,
   For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices not 32. Could you give some explanations ? Thank you.

Best Regards,
Jun Li
Comment 13 Alex Williamson 2013-12-20 09:09:52 EST
(In reply to Jun Li from comment #12)
> Hi Alex,
>    For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices
> not 32. Could you give some explanations ? Thank you.

You're hitting the PCI slot limit, not the VFIO device limit.  You therefore either need to assign some devices statically (cold-plug) and make use of multifunction support or you need to add a PCI-to-PCI bridge to the VM to get more slots on a secondary bus.
Comment 14 Jun Li 2013-12-27 23:36:39 EST
(In reply to Alex Williamson from comment #13)
> (In reply to Jun Li from comment #12)
> > Hi Alex,
> >    For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices
> > not 32. Could you give some explanations ? Thank you.
> 
> You're hitting the PCI slot limit, not the VFIO device limit.  You therefore
> either need to assign some devices statically (cold-plug) and make use of
> multifunction support or you need to add a PCI-to-PCI bridge to the VM to
> get more slots on a secondary bus.

boot guest with 36 vfs:
1, generate boot rhel7 guest commands:
# cat generate-boot-rhel7-cli.sh 
#! /bin/sh
i=0
num=1
n=1;
echo "#! /bin/sh" > boot_a_rhel7.sh
echo "/usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/RHEL-Server-7.0-64.qcow2_v3,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=qcow2,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x8 -device pci-bridge,bus=pci.0,id=bridge2,chassis_nr=1,addr=0x9 \\" >> boot_a_rhel7.sh
while ((i<9)); do
    for ((j=0;j<7;j=j+2)); do
        if ((i>5)); then
            num=2;
        fi
        if ((n==32)); then
            n=1;
        fi
        n=`printf "%x" $n`
        if ((i==8 && j==6)); then
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n"  >> boot_a_rhel7.sh
        else 
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n \\"  >> boot_a_rhel7.sh
        fi
        #echo $i$j
        #sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        n=$((0x$n))
        ((n=n+1))
    done
    ((i=i+1))
done
chmod +x boot_a_rhel7.sh
---
2, boot guest with "boot_a_rhel7.sh".
# ./boot_a_rhel7.sh 
QEMU 1.5.3 monitor - type 'help' for more information
(qemu) qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: vfio: Maximum supported vfio devices (32) already attached

qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: Device initialization failed.
qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: Device 'vfio-pci' could not be initialized
---
As the comment 9, for RHEL 7.0 w/ VFIO the intended limit is 32 devices.
===========
Boot guest with 32 vfs:
1, generate boot rhel7 guest commands:
# cat generate-boot-rhel7-cli.sh 
#! /bin/sh
i=0
num=1
n=1;
echo "#! /bin/sh" > boot_a_rhel7.sh
echo "/usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/RHEL-Server-7.0-64.qcow2_v3,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=qcow2,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x8 -device pci-bridge,bus=pci.0,id=bridge2,chassis_nr=1,addr=0x9 \\" >> boot_a_rhel7.sh
while ((i<8)); do
    for ((j=0;j<7;j=j+2)); do
        if ((i>5)); then
            num=2;
        fi
        if ((n==32)); then
            n=1;
        fi
        n=`printf "%x" $n`
        if ((i==7 && j==6)); then
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n"  >> boot_a_rhel7.sh
        else 
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n \\"  >> boot_a_rhel7.sh
        fi
        #echo $i$j
        #sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        n=$((0x$n))
        ((n=n+1))
    done
    ((i=i+1))
done
chmod +x boot_a_rhel7.sh
-----
2, Boot guest with "boot_a_rhel7.sh"
# ./boot_a_rhel7.sh
---
3, check vfs inside guest and qemu monitor.

Actual results:
inside guest and qemu monitor can find these 32 vfs. vfs inside guest can obtain IP and ping other host.
==========
Based on above test, this bug has been fixed.
Comment 16 Ludek Smid 2014-06-13 07:59:26 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.