Bug 1034072
| Summary: | Domain OS can't boot up from specified disk when existing more than 8 multifunction "virtio-blk-pci" disks in that domain. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Hu Jianwei <jiahu> | ||||||||
| Component: | seabios | Assignee: | Gerd Hoffmann <kraxel> | ||||||||
| Status: | CLOSED CANTFIX | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | medium | ||||||||||
| Version: | 7.0 | CC: | ajia, dyuan, flang, hhuang, honzhang, juzhang, kraxel, mzhan, pbonzini, rhod, virt-bugs, virt-docs, virt-maint, xuhan, yruseva | ||||||||
| Target Milestone: | rc | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Known Issue | |||||||||
| Doc Text: |
The SeaBIOS application runs in real mode for compatibility with BIOS interfaces. This limits the amount of memory available. As a consequence, SeaBIOS is only able to handle a limited number of disks. Currently, the supported number of disks is:
virtio-scsi — 64
virtio-blk — 4
ahci/sata — 24 (4 controllers with all 6 ports connected)
usb-storage — 4
As a workaround for this problem, when attaching a large number of disks to your virtual machine, make sure that your system disk has a small pci slot number, so SeaBIOS sees it first when scanning the pci bus. It is also recommended to use the virtio-scsi device instead of virtio-blk as the per-disk memory overhead is smaller.
|
Story Points: | --- | ||||||||
| Clone Of: | |||||||||||
| : | 1040714 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2014-01-19 10:27:15 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1040714 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 828565 [details]
r7_multi.xml
Created attachment 828566 [details]
r7_without_multi.xml
Created attachment 828567 [details]
NO_bootable.png
Most likely seabios runs out of memory (real mode address space). Not sure we can do much about it, other than switching to OVMF. Workaround: use virtio-scsi instead, needs less memory per disk as all disks share one virtio queue for requests. So, all we can do about this is documenting the limits. Can you please test how many disks seabios can manage for (a) virtio-blk (b) virtio-scsi (all disks on one controller) (c) ahci Test procedere: start guest with boot menu enabled, hit F12, verify all configured disks are listed in the boot menu. thank you. (In reply to Gerd Hoffmann from comment #6) > So, all we can do about this is documenting the limits. > > Can you please test how many disks seabios can manage for > > (a) virtio-blk > (b) virtio-scsi (all disks on one controller) > (c) ahci > > Test procedere: start guest with boot menu enabled, hit F12, > verify all configured disks are listed in the boot menu. > > thank you. Hi Flang, Would you please do some test and update in the bz? Best Regards, Junyi (In reply to Gerd Hoffmann from comment #6) > So, all we can do about this is documenting the limits. > > Can you please test how many disks seabios can manage for > > (a) virtio-blk > (b) virtio-scsi (all disks on one controller) > (c) ahci > > Test procedere: start guest with boot menu enabled, hit F12, > verify all configured disks are listed in the boot menu. > > thank you. Test this bug as follow version: Host # uname -r 3.10.0-55.el7.x86_64 # rpm -q qemu-kvm qemu-kvm-1.5.3-20.el7.x86_64 Results:Max num of seabios *:not test _________________________________________________________________________________________________________________________________________ | Device | virtio-scsi | virtio-blk | ide | sata | virtio-net | rtl8139 | e1000 | usb3.0 | |______________|_______________________|__________________|___________|________|______________|_____________|_______________|___________| | | | | | | | | | | | Max num | 126 | 7 | 4 | 6 | 22 | 22 | 22 | 18 | |one controller| | | | | | | | | |______________|_______________________|__________________|___________|________|______________|_____________|_______________|___________| | | | | | | | | | | | Max num | * | * | * | 93 | * | * | * | * | |mul controller| | | | | | | | | |______________|_______________________|__________________|___________|________|______________|_____________|_______________|___________| Test as follow version:
Host
# uname -r
3.10.0-55.el7.x86_64
# rpm -q qemu-kvm
qemu-kvm-1.5.3-20.el7.x86_64
Steps:
1.Boot guest("-nodefaults ") with the test devices(and without any other devices).
2.Press "F12"
3.See the boot menu
Results:
Max num of seabios boot menu :
Results:
________________________________________________________________________________
| Device | virtio-scsi | virtio-blk | ide | sata |
|______________|___________________|__________________|___________|________|
| | | | | |
| Max num | 126 | 7 | 4 | 6 |
|one controller| | | | |
|______________|___________________|__________________|___________|________|
| | | | | |
| Max num | * | * | * | 93 |
|mul controller| | | | |
|______________|___________________|__________________|___________|________|
________________________________________________________________________
| Device | virtio-net | rtl8139 | e1000 | usb3.0 |
|______________|_____________|_____________|_______________|___________|
| | | | | |
| Max num | 22 | 22 | 22 | 18 |
|one controller| | | | |
|______________|_____________|_____________|_______________|___________|
| | | | | |
| Max num | | | | |
|mul controller| * | * | * | * |
|______________|_____________|_____________|_______________|___________|
Test seabios version: seabios-1.7.2.2-4.el7.x86_64 Thanks. So lets apply apply some save margin so small changes in the seabios size don't render the documented limits invalid, and also to leave some room for multiple kinds of interfaces at the same time.
How about these limits:
virtio-scsi: 64
virtio-blk: 4
ide: 4 (not sure this is needed, you can't have more ide devices anyway ...)
ahci/sata: 48 (8 controllers with all 6 ports hooked up).
usb-storage: 4 (should be enough for all practical purposes, usb isn't
the interface you want use for lots of disks anyway).
Paolo? What do you think?
Ronen? How we are going to document this? Known issue? Release Notes?
Looks great for boot-menu purposes, but I'm sure some customers are looking for more than this number of disks. I've definitely seen bug reports going up to vdj or more. Could we specify that boot disks should be given lower PCI addresses if the need arises? Does that require disabling parallel scanning of hardware (and how much boot speed does parallel scanning really buy)? (In reply to Paolo Bonzini from comment #12) > Looks great for boot-menu purposes, but I'm sure some customers are looking > for more than this number of disks. I've definitely seen bug reports going > up to vdj or more. Sure, that is (seabios) bootable disks only. > Could we specify that boot disks should be given lower PCI addresses if the > need arises? Probably a good idea to mention it. > Does that require disabling parallel scanning of hardware (and > how much boot speed does parallel scanning really buy)? The speedup effect is probably small on virtual hardware which tends to respond pretty fast. I've never actually measured it though. Also the detection order doesn't diverge from scanning order much. Again on virtual hardware, probably also due to hardware responding fast. I'd tend to stay with the upstream default (parallel scanning enabled) due to better test coverage. *** Bug 1066788 has been marked as a duplicate of this bug. *** |
Description of problem: Domain OS can't boot up from specified disk when existing more than 8 multifunction "virtio-blk-pci" disks in that domain. Version-Release number of selected component (if applicable): libvirt-1.1.1-12.el7.x86_64 kernel-3.10.0-48.el7.x86_64 qemu-kvm-1.5.3-19.el7.x86_64 seabios-bin-1.7.2.2-4.el7.noarch seabios-1.7.2.2-4.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Define two domains with/without multifunction pci devices(disks). Refer to attachments(r7_multi.xml and r7_without_multi.xml) [root@ibm-x3650m3-03 216353]# virsh define r7_multi.xml Domain r7_multi defined from r7_multi.xml [root@ibm-x3650m3-03 216353]# virsh define r7_without_multi.xml Domain r7_without_multi defined from r7_without_multi.xml [root@ibm-x3650m3-03 216353]# virsh list --all Id Name State ---------------------------------------------------- - r7_multi shut off - r7_without_multi shut off [root@ibm-x3650m3-03 216353]# virsh start r7_multi Domain r7_multi started [root@ibm-x3650m3-03 216353]# virsh start r7_without_multi Domain r7_without_multi started 2. On multifunction enabled pci devices,check the disk status from libvirt side and qemu-kvm side. The boot order was set in "<boot order='1'/>" and passed it to qemu-kvm command line "bootindex=1". [root@ibm-x3650m3-03 216353]# virsh dumpxml r7_multi | grep disk -aA5 <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/var/lib/libvirt/images/r7.img'/> <target dev='vda' bus='virtio'/> <boot order='1'/> <=========set boot order=1 <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-a.img'/> <target dev='sdaa' bus='virtio'/> <alias name='virtio-disk26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-b.img'/> <target dev='sdab' bus='virtio'/> <alias name='virtio-disk27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-c.img'/> <target dev='sdac' bus='virtio'/> <alias name='virtio-disk28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-d.img'/> <target dev='sdad' bus='virtio'/> <alias name='virtio-disk29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-e.img'/> <target dev='sdae' bus='virtio'/> <alias name='virtio-disk30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-f.img'/> <target dev='sdaf' bus='virtio'/> <alias name='virtio-disk31'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-g.img'/> <target dev='sdag' bus='virtio'/> <alias name='virtio-disk32'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-h.img'/> <target dev='sdah' bus='virtio'/> <alias name='virtio-disk33'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-i.img'/> <target dev='sdai' bus='virtio'/> <alias name='virtio-disk34'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-j.img'/> <target dev='sdaj' bus='virtio'/> <alias name='virtio-disk35'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-k.img'/> <target dev='sdak' bus='virtio'/> <alias name='virtio-disk36'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2' multifunction='on'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> [root@ibm-x3650m3-03 216353]# ps aux | grep r7_multi | sed "s/-device/\n-device/g" | grep -v grep qemu 24446 1.8 0.1 1651448 40776 ? Sl 16:05 0:01 /usr/libexec/qemu-kvm -name r7_multi -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 417fdbc8-491f-40c1-bef6-0bc0053db8cb -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7_multi.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/r7.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/libvirt/images/disks/usb-a-a.img,if=none,id=drive-virtio-disk26,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3,drive=drive-virtio-disk26,id=virtio-disk26 -drive file=/var/lib/libvirt/images/disks/usb-a-b.img,if=none,id=drive-virtio-disk27,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3.0x1,drive=drive-virtio-disk27,id=virtio-disk27 -drive file=/var/lib/libvirt/images/disks/usb-a-c.img,if=none,id=drive-virtio-disk28,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3.0x2,drive=drive-virtio-disk28,id=virtio-disk28 -drive file=/var/lib/libvirt/images/disks/usb-a-d.img,if=none,id=drive-virtio-disk29,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3.0x3,drive=drive-virtio-disk29,id=virtio-disk29 -drive file=/var/lib/libvirt/images/disks/usb-a-e.img,if=none,id=drive-virtio-disk30,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3.0x4,drive=drive-virtio-disk30,id=virtio-disk30 -drive file=/var/lib/libvirt/images/disks/usb-a-f.img,if=none,id=drive-virtio-disk31,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3.0x5,drive=drive-virtio-disk31,id=virtio-disk31 -drive file=/var/lib/libvirt/images/disks/usb-a-g.img,if=none,id=drive-virtio-disk32,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3.0x6,drive=drive-virtio-disk32,id=virtio-disk32 -drive file=/var/lib/libvirt/images/disks/usb-a-h.img,if=none,id=drive-virtio-disk33,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x3.0x7,drive=drive-virtio-disk33,id=virtio-disk33 -drive file=/var/lib/libvirt/images/disks/usb-a-i.img,if=none,id=drive-virtio-disk34,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x4,drive=drive-virtio-disk34,id=virtio-disk34 -drive file=/var/lib/libvirt/images/disks/usb-a-j.img,if=none,id=drive-virtio-disk35,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x4.0x1,drive=drive-virtio-disk35,id=virtio-disk35 -drive file=/var/lib/libvirt/images/disks/usb-a-k.img,if=none,id=drive-virtio-disk36,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x4.0x2,drive=drive-virtio-disk36,id=virtio-disk36 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5902,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 [root@ibm-x3650m3-03 216353]# [root@ibm-x3650m3-03 216353]# ps aux | grep r7_multi | sed "s/-device/\n-device/g" | grep -v grep| grep "multifunction=on"| wc -l 11 [root@ibm-x3650m3-03 216353]# 3. Check the domain status, booted failed, see attachments(NO_bootable.png) 4. On no multifunction pci device, we can boot up the domain OS normally, login to the guest, we can find all attached disks. [root@ibm-x3650m3-03 216353]# [root@ibm-x3650m3-03 216353]# [root@ibm-x3650m3-03 216353]# [root@ibm-x3650m3-03 216353]# virsh dumpxml r7_232| grep disk -aA5 <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/var/lib/libvirt/images/r7.img'/> <target dev='vda' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-a.img'/> <target dev='sdaa' bus='virtio'/> <boot order='2'/> <alias name='virtio-disk26'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-b.img'/> <target dev='sdab' bus='virtio'/> <boot order='3'/> <alias name='virtio-disk27'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-c.img'/> <target dev='sdac' bus='virtio'/> <boot order='4'/> <alias name='virtio-disk28'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-d.img'/> <target dev='sdad' bus='virtio'/> <alias name='virtio-disk29'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-e.img'/> <target dev='sdae' bus='virtio'/> <alias name='virtio-disk30'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-f.img'/> <target dev='sdaf' bus='virtio'/> <alias name='virtio-disk31'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-h.img'/> <target dev='sdah' bus='virtio'/> <alias name='virtio-disk33'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/disks/usb-a-i.img'/> <target dev='sdai' bus='virtio'/> <alias name='virtio-disk34'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> [root@ibm-x3650m3-03 216353]# [root@ibm-x3650m3-03 216353]# ps aux | grep r7_232 | sed "s/-device/\n-device/g" | grep -v grep qemu 24052 26.0 0.8 5073248 285736 ? Sl 16:01 0:15 /usr/libexec/qemu-kvm -name r7_232 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a5c90ce1-158e-4728-ba88-8e250055fe88 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7_232.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/r7.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/libvirt/images/disks/usb-a-a.img,if=none,id=drive-virtio-disk26,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk26,id=virtio-disk26,bootindex=2 -drive file=/var/lib/libvirt/images/disks/usb-a-b.img,if=none,id=drive-virtio-disk27,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4.0x1,drive=drive-virtio-disk27,id=virtio-disk27,bootindex=3 -drive file=/var/lib/libvirt/images/disks/usb-a-c.img,if=none,id=drive-virtio-disk28,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xb,drive=drive-virtio-disk28,id=virtio-disk28,bootindex=4 -drive file=/var/lib/libvirt/images/disks/usb-a-d.img,if=none,id=drive-virtio-disk29,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk29,id=virtio-disk29 -drive file=/var/lib/libvirt/images/disks/usb-a-e.img,if=none,id=drive-virtio-disk30,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk30,id=virtio-disk30 -drive file=/var/lib/libvirt/images/disks/usb-a-f.img,if=none,id=drive-virtio-disk31,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk31,id=virtio-disk31 -drive file=/var/lib/libvirt/images/disks/usb-a-h.img,if=none,id=drive-virtio-disk33,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xd,drive=drive-virtio-disk33,id=virtio-disk33 -drive file=/var/lib/libvirt/images/disks/usb-a-i.img,if=none,id=drive-virtio-disk34,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xc,drive=drive-virtio-disk34,id=virtio-disk34 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5901,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 [root@ibm-x3650m3-03 216353]# ps aux | grep r7_without_multi | sed "s/-device/\n-device/g" | grep -v grep| grep virtio-blk-pci|wc -l 9 [root@ibm-x3650m3-03 216353]# Actual results: See steps 2-3, the domain OS can't boot up normally, seabios just tell me no bootable device, but I have set bootindex=1 to my available disk. Expected results: Domain should boot up normally, like the step 4.