810471 – boot fails while starting guest with sockets>62 and cores=1 and threads=1 option and 10 virtio disks

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 810471 - boot fails while starting guest with sockets>62 and cores=1 and threads=1 option and 10 virtio disks

Summary: boot fails while starting guest with sockets>62 and cores=1 and threads=1 opt...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	6.3
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Gleb Natapov
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-04-06 09:57 UTC by FuXiangChun
Modified:	2013-12-09 00:56 UTC (History)
CC List:	19 users (show)
Fixed In Version:	seabios-0.6.1.2-19.el6
Doc Type:	Bug Fix
Doc Text:	No documentation needed
Clone Of:
Environment:
Last Closed:	2012-06-20 11:46:12 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
no bootable device (5.67 KB, image/png) 2012-04-06 10:01 UTC, FuXiangChun	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2012:0746	0	normal	SHIPPED_LIVE	qemu-kvm bug fix and enhancement update	2012-06-19 19:31:48 UTC
Red Hat Product Errata	RHBA-2012:0802	0	normal	SHIPPED_LIVE	seabios bug fix and enhancement update	2012-06-19 19:51:36 UTC

Description FuXiangChun 2012-04-06 09:57:34 UTC

Description of problem:
If the value of cores and threads is not 1, then guest can boot successfully.
If I reduce the number of virto disks,then can add the same amount of sockets values.

for example:
fail scenario
1. -smp 63,sockets=63,cores=1,threads=1 and multiple virtio disks(10) 

success scenario
2. -smp 63,sockets=9,cores=1,threads=7 and multiple virtio disks(10)

3. -smp 63,sockets=63,cores=1,threads=1 and multiple virtio disks(9)

Version-Release number of selected component (if applicable):
# rpm -qa|grep qemu
qemu-kvm-0.12.1.2-2.267.el6.x86_64
# uname -r
2.6.32-259.el6.x86_64

# rpm -qa|grep seabios
seabios-0.6.1.2-12.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1./usr/libexec/qemu-kvm -M rhel6.3.0  -enable-kvm -m 4G -smp 63,sockets=63,cores=1,threads=1 -name redsap-vm1 -uuid de037220-61e0-6ade-65ad-ea175e98adb9 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/redsap-vm1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot order=c,menu=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/mnt/rhel6.3-64.raw,if=none,id=drive-ide0-0-0,format=raw,cache=none,aio=native -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/mnt/d1,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/mnt/d2,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/mnt/d3,if=none,id=drive-virtio-disk3,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3 -drive file=/mnt/d4,if=none,id=drive-virtio-disk4,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk4,id=virtio-disk4 -drive file=/mnt/d5,if=none,id=drive-virtio-disk5,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk5,id=virtio-disk5 -drive file=/mnt/d6,if=none,id=drive-virtio-disk6,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xb,drive=drive-virtio-disk6,id=virtio-disk6 -drive file=/mnt/d7,if=none,id=drive-virtio-disk7,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xc,drive=drive-virtio-disk7,id=virtio-disk7 -drive file=/mnt/d8,if=none,id=drive-virtio-disk8,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xd,drive=drive-virtio-disk8,id=virtio-disk8 -drive file=/mnt/d9,if=none,id=drive-virtio-disk9,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xe,drive=drive-virtio-disk9,id=virtio-disk9 -drive file=/mnt/d10,if=none,id=drive-virtio-disk10,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xf,drive=drive-virtio-disk10,id=virtio-disk10 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc :0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

2.
3.
  
Actual results:
boot fail(no bootable device) 

Expected results:
guest boot successfully

Additional info:

host info:
cpu:AMD Opteron(tm) Processor 6172, 48 cores
memory: 512G

Comment 1 FuXiangChun 2012-04-06 10:01:21 UTC

Created attachment 575691 [details]
no bootable  device

Comment 3 Gleb Natapov 2012-04-08 12:43:00 UTC

So this is just a variation of BZ06797. BIOS runs out of memory and cannot initialize boot disk. Amount of CPU matters since MPC table size depends on it. Bios create an entry for each socket, so number of sockets is really what matters here.

Comment 4 Ademar Reis 2012-04-10 22:05:52 UTC

(In reply to comment #3)
> So this is just a variation of BZ06797. BIOS runs out of memory and cannot
> initialize boot disk. Amount of CPU matters since MPC table size depends on it.
> Bios create an entry for each socket, so number of sockets is really what
> matters here.

Gleb: you mean Bug 706797, right? If that's the case, the bug was closed as NOTABUG, but this one is devel_ack'ed, so I assume it's indeed valid and I'll keep it AS IS.

Comment 5 Gleb Natapov 2012-04-11 08:42:56 UTC

(In reply to comment #4)
> (In reply to comment #3)
> > So this is just a variation of BZ06797. BIOS runs out of memory and cannot
> > initialize boot disk. Amount of CPU matters since MPC table size depends on it.
> > Bios create an entry for each socket, so number of sockets is really what
> > matters here.
> 
> Gleb: you mean Bug 706797, right? If that's the case, the bug was closed as
Yes, I do. Sorry.

> NOTABUG, but this one is devel_ack'ed, so I assume it's indeed valid and I'll
> keep it AS IS.

Why devel_ack is relevant? However sets devel_ack does it without investigating the problem usually, so it mostly meaningless before developer actually looks at the bug. I am sure that one was devel_ack'ed too before been closed as NOTABUG.

But I see that large number of virtio disks causes troubles in real life scenarios, so I will try to optimize that use case. I will try to implement the hack that I descried in Bug 706797 [1] and see how bad it is. But that would be the fix for 706797, not this bug. There is nothing that can be done for system with to many sockets. Of course if virtio disk will not consume a lot of low memory, more memory will be available for MPC tables. Hope this will push max amount of sockets we can support high enough for us to not care.



[1] Another fix would be to have only one vq for all virtio devices and
    reinitialize it on each disk access. This is hackish and may slow down BIOS
    access to virtio disk though.

Comment 6 Dor Laor 2012-04-11 12:38:50 UTC

I'll close the bug since there is a work around which is actually the desired path (using less sockets and more cores per sockets)

Comment 7 Gleb Natapov 2012-04-11 13:05:36 UTC

(In reply to comment #6)
> I'll close the bug since there is a work around which is actually the desired
> path (using less sockets and more cores per sockets)

That's good because actually moving to one vq for all vritio disks will not help this BZ. The reason is that vqs are allocated not from the same memory pool as MPC tables. There are small, per disk structure that is allocated from the same pool and in the case of this BZ this small allocations start to fail. I am looking into moving this allocation out of this memory pool, but the assumption goes pretty dip into the code.

Comment 8 Gleb Natapov 2012-04-15 13:46:08 UTC

This problem is more or less solved on upstream Seabios. The memory allocation that fails is from f-segment pool. Upstream Seabios relocates 32bit init code from f-segment to high memory before execution and adds freed memory to f-segment pool. In the end the pool contains ~20K bytes instead of 2K (of course it depend on how much memory 16bit code occupies in f-segment. 16bit code has to be there and cannot be relocated). Backporting all the patches that do reallocation is not trivial.

Comment 10 Dor Laor 2012-04-22 11:31:59 UTC

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
No documentation needed

Comment 11 daiwei 2012-04-26 15:42:26 UTC

Reproduced this issue with seabios-0.6.1.2-12.el6.x86_64

1.Boot guest with the same qemu-kvm command line in Comment 0

Boot failed, VNC console prompts "no bootable device" .


Verified this issue with seabios-0.6.1.2-19.el6.x86_64

1.Boot guest with the same qemu-kvm command line in Comment 0

Guest normal start, also test with " -smp 160 " can boot guest successfully.

So, this bug has been fixed.

Comment 14 errata-xmlrpc 2012-06-20 11:46:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0746.html

Note You need to log in before you can comment on or make changes to this bug.