Bug 810471 - boot fails while starting guest with sockets>62 and cores=1 and threads=1 option and 10 virtio disks
boot fails while starting guest with sockets>62 and cores=1 and threads=1 opt...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.3
x86_64 Linux
high Severity medium
: rc
: ---
Assigned To: Gleb Natapov
Virtualization Bugs
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-06 05:57 EDT by FuXiangChun
Modified: 2013-12-08 19:56 EST (History)
19 users (show)

See Also:
Fixed In Version: seabios-0.6.1.2-19.el6
Doc Type: Bug Fix
Doc Text:
No documentation needed
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-20 07:46:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
no bootable device (5.67 KB, image/png)
2012-04-06 06:01 EDT, FuXiangChun
no flags Details

  None (edit)
Description FuXiangChun 2012-04-06 05:57:34 EDT
Description of problem:
If the value of cores and threads is not 1, then guest can boot successfully.
If I reduce the number of virto disks,then can add the same amount of sockets values.

for example:
fail scenario
1. -smp 63,sockets=63,cores=1,threads=1 and multiple virtio disks(10) 

success scenario
2. -smp 63,sockets=9,cores=1,threads=7 and multiple virtio disks(10)

3. -smp 63,sockets=63,cores=1,threads=1 and multiple virtio disks(9)

Version-Release number of selected component (if applicable):
# rpm -qa|grep qemu
qemu-kvm-0.12.1.2-2.267.el6.x86_64
# uname -r
2.6.32-259.el6.x86_64

# rpm -qa|grep seabios
seabios-0.6.1.2-12.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1./usr/libexec/qemu-kvm -M rhel6.3.0  -enable-kvm -m 4G -smp 63,sockets=63,cores=1,threads=1 -name redsap-vm1 -uuid de037220-61e0-6ade-65ad-ea175e98adb9 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/redsap-vm1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=c,menu=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/rhel6.3-64.raw,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=native -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/mnt/d1,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/mnt/d2,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/mnt/d3,if=none,id=drive-virtio-disk3,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3 -drive file=/mnt/d4,if=none,id=drive-virtio-disk4,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4 -drive file=/mnt/d5,if=none,id=drive-virtio-disk5,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk5,id=virtio-disk5 -drive file=/mnt/d6,if=none,id=drive-virtio-disk6,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xb,drive=drive-virtio-disk6,id=virtio-disk6 -drive file=/mnt/d7,if=none,id=drive-virtio-disk7,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xc,drive=drive-virtio-disk7,id=virtio-disk7 -drive file=/mnt/d8,if=none,id=drive-virtio-disk8,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xd,drive=drive-virtio-disk8,id=virtio-disk8 -drive file=/mnt/d9,if=none,id=drive-virtio-disk9,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xe,drive=drive-virtio-disk9,id=virtio-disk9 -drive file=/mnt/d10,if=none,id=drive-virtio-disk10,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xf,drive=drive-virtio-disk10,id=virtio-disk10 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc :0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

2.
3.
  
Actual results:
boot fail(no bootable device) 

Expected results:
guest boot successfully

Additional info:

host info:
cpu:AMD Opteron(tm) Processor 6172, 48 cores
memory: 512G
Comment 1 FuXiangChun 2012-04-06 06:01:21 EDT
Created attachment 575691 [details]
no bootable  device
Comment 3 Gleb Natapov 2012-04-08 08:43:00 EDT
So this is just a variation of BZ06797. BIOS runs out of memory and cannot initialize boot disk. Amount of CPU matters since MPC table size depends on it. Bios create an entry for each socket, so number of sockets is really what matters here.
Comment 4 Ademar Reis 2012-04-10 18:05:52 EDT
(In reply to comment #3)
> So this is just a variation of BZ06797. BIOS runs out of memory and cannot
> initialize boot disk. Amount of CPU matters since MPC table size depends on it.
> Bios create an entry for each socket, so number of sockets is really what
> matters here.

Gleb: you mean Bug 706797, right? If that's the case, the bug was closed as NOTABUG, but this one is devel_ack'ed, so I assume it's indeed valid and I'll keep it AS IS.
Comment 5 Gleb Natapov 2012-04-11 04:42:56 EDT
(In reply to comment #4)
> (In reply to comment #3)
> > So this is just a variation of BZ06797. BIOS runs out of memory and cannot
> > initialize boot disk. Amount of CPU matters since MPC table size depends on it.
> > Bios create an entry for each socket, so number of sockets is really what
> > matters here.
> 
> Gleb: you mean Bug 706797, right? If that's the case, the bug was closed as
Yes, I do. Sorry.

> NOTABUG, but this one is devel_ack'ed, so I assume it's indeed valid and I'll
> keep it AS IS.

Why devel_ack is relevant? However sets devel_ack does it without investigating the problem usually, so it mostly meaningless before developer actually looks at the bug. I am sure that one was devel_ack'ed too before been closed as NOTABUG.

But I see that large number of virtio disks causes troubles in real life scenarios, so I will try to optimize that use case. I will try to implement the hack that I descried in Bug 706797 [1] and see how bad it is. But that would be the fix for 706797, not this bug. There is nothing that can be done for system with to many sockets. Of course if virtio disk will not consume a lot of low memory, more memory will be available for MPC tables. Hope this will push max amount of sockets we can support high enough for us to not care.



[1] Another fix would be to have only one vq for all virtio devices and
    reinitialize it on each disk access. This is hackish and may slow down BIOS
    access to virtio disk though.
Comment 6 Dor Laor 2012-04-11 08:38:50 EDT
I'll close the bug since there is a work around which is actually the desired path (using less sockets and more cores per sockets)
Comment 7 Gleb Natapov 2012-04-11 09:05:36 EDT
(In reply to comment #6)
> I'll close the bug since there is a work around which is actually the desired
> path (using less sockets and more cores per sockets)

That's good because actually moving to one vq for all vritio disks will not help this BZ. The reason is that vqs are allocated not from the same memory pool as MPC tables. There are small, per disk structure that is allocated from the same pool and in the case of this BZ this small allocations start to fail. I am looking into moving this allocation out of this memory pool, but the assumption goes pretty dip into the code.
Comment 8 Gleb Natapov 2012-04-15 09:46:08 EDT
This problem is more or less solved on upstream Seabios. The memory allocation that fails is from f-segment pool. Upstream Seabios relocates 32bit init code from f-segment to high memory before execution and adds freed memory to f-segment pool. In the end the pool contains ~20K bytes instead of 2K (of course it depend on how much memory 16bit code occupies in f-segment. 16bit code has to be there and cannot be relocated). Backporting all the patches that do reallocation is not trivial.
Comment 10 Dor Laor 2012-04-22 07:31:59 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
No documentation needed
Comment 11 daiwei 2012-04-26 11:42:26 EDT
Reproduced this issue with seabios-0.6.1.2-12.el6.x86_64

1.Boot guest with the same qemu-kvm command line in Comment 0

Boot failed, VNC console prompts "no bootable device" .


Verified this issue with seabios-0.6.1.2-19.el6.x86_64

1.Boot guest with the same qemu-kvm command line in Comment 0

Guest normal start, also test with " -smp 160 " can boot guest successfully.

So, this bug has been fixed.
Comment 14 errata-xmlrpc 2012-06-20 07:46:12 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0746.html

Note You need to log in before you can comment on or make changes to this bug.