RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 678368 - RFE: Support more than 8 assigned devices
Summary: RFE: Support more than 8 assigned devices
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.0
Hardware: x86_64
OS: Linux
low
medium
Target Milestone: rc
: 7.0
Assignee: Bandan Das
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 691328 968375 (view as bug list)
Depends On:
Blocks: 817995
TreeView+ depends on / blocked
 
Reported: 2011-02-17 17:34 UTC by Alex Williamson
Modified: 2014-06-18 03:12 UTC (History)
24 users (show)

Fixed In Version: qemu-kvm-1.5.3-26.el7
Doc Type: Enhancement
Doc Text:
Clone Of: 670787
Environment:
Last Closed: 2014-06-13 11:59:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Comment 2 Alex Williamson 2011-03-28 13:43:23 UTC 
*** Bug 691328 has been marked as a duplicate of this bug. ***
Comment 8 Alex Williamson 2013-06-07 16:51:56 UTC 
*** Bug 968375 has been marked as a duplicate of this bug. ***
Comment 12 Jun Li 2013-12-20 08:15:57 UTC 
Verify this bug:
Version-Release number of selected component (if applicable):
qemu-kvm-1.5.3-30.el7.x86_64
3.10.0-61.el7.x86_64
---
Steps as followings:
<cli>:
1, boot guest with following command:
# gdb --args /usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/win2012r2.raw,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=raw,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -drive file=/home/juli/ISO/en_windows_server_2012_r2_x64_dvd_2707946.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-0 -device ide-drive,drive=drive-ide1-0-0,id=ide1-0-0,bus=ide.0,unit=0,bootindex=4 -drive file=/home/juli/virtio-win-prewhql-0.1-74.iso,if=none,media=cdrom,format=raw,aio=native,id=drive-ide1-0-1 -device ide-drive,drive=drive-ide1-0-1,id=ide1-0-1,bus=ide.0,unit=1,bootindex=6 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -netdev tap,id=tap1,vhost=on,queues=4,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,ifname=tap-juli -device virtio-net-pci,netdev=tap1,id=nic1,mq=on,vectors=17,mac=1a:59:0a:4b:aa:94
-----
2, run add vf with the following script.
# cat add-vf.sh 
#! /bin/sh
i=0
while ((i < 9)); do
    for ((j=0;j<7;j=j+2)); do
        echo "device_add vfio-pci,host=05:1$i.$j,id=vf_$i$j" | nc -U /tmp/monitor1
        sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        #((j=j+2))
    done
    ((i=i+1))
done
---
3, Check vfs in guest.

Actual results:
(qemu) device_add vfio-pci,host=05:16.2,id=vf_62
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:16.4,id=vf_64
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:16.6,id=vf_66
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.0,id=vf_70
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.2,id=vf_72
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.4,id=vf_74
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:17.6,id=vf_76
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.0,id=vf_80
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.2,id=vf_82
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.4,id=vf_84
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
(qemu) QEMU 1.5.3 monitor - type 'help' for more information
(qemu) device_add vfio-pci,host=05:18.6,id=vf_86
PCI: no slot/function available for vfio-pci, all in use
Device initialization failed.
Device 'vfio-pci' could not be initialized
-----
So can only hot-plug 26 vfs. Check inside win2012r2 guest only has 26 82599 vfs. 
BTW, when hot-plug vfs more than 26, qemu-kvm will give error, but works well. No core.

Base on above test, this bug has been verified.

Hi Alex,
   For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices not 32. Could you give some explanations ? Thank you.

Best Regards,
Jun Li
Comment 13 Alex Williamson 2013-12-20 14:09:52 UTC 
(In reply to Jun Li from comment #12)
> Hi Alex,
>    For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices
> not 32. Could you give some explanations ? Thank you.

You're hitting the PCI slot limit, not the VFIO device limit.  You therefore either need to assign some devices statically (cold-plug) and make use of multifunction support or you need to add a PCI-to-PCI bridge to the VM to get more slots on a secondary bus.
Comment 14 Jun Li 2013-12-28 04:36:39 UTC 
(In reply to Alex Williamson from comment #13)
> (In reply to Jun Li from comment #12)
> > Hi Alex,
> >    For RHEL 7.0 w/ VFIO , based on the above test, the limit is 26 devices
> > not 32. Could you give some explanations ? Thank you.
> 
> You're hitting the PCI slot limit, not the VFIO device limit.  You therefore
> either need to assign some devices statically (cold-plug) and make use of
> multifunction support or you need to add a PCI-to-PCI bridge to the VM to
> get more slots on a secondary bus.

boot guest with 36 vfs:
1, generate boot rhel7 guest commands:
# cat generate-boot-rhel7-cli.sh 
#! /bin/sh
i=0
num=1
n=1;
echo "#! /bin/sh" > boot_a_rhel7.sh
echo "/usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/RHEL-Server-7.0-64.qcow2_v3,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=qcow2,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x8 -device pci-bridge,bus=pci.0,id=bridge2,chassis_nr=1,addr=0x9 \\" >> boot_a_rhel7.sh
while ((i<9)); do
    for ((j=0;j<7;j=j+2)); do
        if ((i>5)); then
            num=2;
        fi
        if ((n==32)); then
            n=1;
        fi
        n=`printf "%x" $n`
        if ((i==8 && j==6)); then
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n"  >> boot_a_rhel7.sh
        else 
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n \\"  >> boot_a_rhel7.sh
        fi
        #echo $i$j
        #sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        n=$((0x$n))
        ((n=n+1))
    done
    ((i=i+1))
done
chmod +x boot_a_rhel7.sh
---
2, boot guest with "boot_a_rhel7.sh".
# ./boot_a_rhel7.sh 
QEMU 1.5.3 monitor - type 'help' for more information
(qemu) qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: vfio: Maximum supported vfio devices (32) already attached

qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: Device initialization failed.
qemu-kvm: -device vfio-pci,host=05:18.0,id=vf_80,bus=bridge2,addr=0x2: Device 'vfio-pci' could not be initialized
---
As the comment 9, for RHEL 7.0 w/ VFIO the intended limit is 32 devices.
===========
Boot guest with 32 vfs:
1, generate boot rhel7 guest commands:
# cat generate-boot-rhel7-cli.sh 
#! /bin/sh
i=0
num=1
n=1;
echo "#! /bin/sh" > boot_a_rhel7.sh
echo "/usr/libexec/qemu-kvm -S -M pc-i440fx-rhel7.0.0 -cpu SandyBridge,hv_spinlocks=0x1fff,hv_relaxed,hv_vapic -enable-kvm -m 2G -smp 2,sockets=2,cores=1,threads=1 -name juli -uuid 355a2475-4e03-4cdd-bf7b-5d6a59edaa68 -rtc base=localtime,clock=host,driftfix=slew -device virtio-scsi-pci,bus=pci.0,addr=0x5,id=scsi0 -drive file=/home/juli/RHEL-Server-7.0-64.qcow2_v3,if=none,id=drive-scsi0-0-0,media=disk,cache=none,format=qcow2,werror=stop,rerror=stop,aio=native  -device scsi-hd,drive=drive-scsi0-0-0,bus=scsi0.0,scsi-id=0,lun=0,id=juli,bootindex=0 -device virtio-balloon-pci,id=ballooning -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on,reboot-timeout=-1,strict=on -qmp tcp:0:4499,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :8 -spice port=5961,disable-ticketing  -vga qxl -global qxl-vga.revision=3 -monitor stdio -monitor tcp:0:7766,server,nowait -monitor unix:/tmp/monitor1,server,nowait -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1,addr=0x8 -device pci-bridge,bus=pci.0,id=bridge2,chassis_nr=1,addr=0x9 \\" >> boot_a_rhel7.sh
while ((i<8)); do
    for ((j=0;j<7;j=j+2)); do
        if ((i>5)); then
            num=2;
        fi
        if ((n==32)); then
            n=1;
        fi
        n=`printf "%x" $n`
        if ((i==7 && j==6)); then
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n"  >> boot_a_rhel7.sh
        else 
            echo "-device vfio-pci,host=05:1$i.$j,id=vf_$i$j,bus=bridge$num,addr=0x$n \\"  >> boot_a_rhel7.sh
        fi
        #echo $i$j
        #sleep 8
        #echo "device_add vfio-pci,host=06:1$i.$j,id=vf_$i$j"
        n=$((0x$n))
        ((n=n+1))
    done
    ((i=i+1))
done
chmod +x boot_a_rhel7.sh
-----
2, Boot guest with "boot_a_rhel7.sh"
# ./boot_a_rhel7.sh
---
3, check vfs inside guest and qemu monitor.

Actual results:
inside guest and qemu monitor can find these 32 vfs. vfs inside guest can obtain IP and ping other host.
==========
Based on above test, this bug has been fixed.
Comment 16 Ludek Smid 2014-06-13 11:59:26 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.
Note You need to log in before you can comment on or make changes to this bug.