Bug 1010882 - kvm: backport "Improve create VCPU parameter"
kvm: backport "Improve create VCPU parameter"
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.5
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Andrew Jones
Virtualization Bugs
:
: 1090861 (view as bug list)
Depends On: 868744 1010881
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-23 05:03 EDT by Andrew Jones
Modified: 2014-10-14 01:25 EDT (History)
11 users (show)

See Also:
Fixed In Version: kernel-2.6.32-465.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1010881
Environment:
Last Closed: 2014-10-14 01:25:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andrew Jones 2013-09-23 05:03:35 EDT
+++ This bug was initially created as a clone of Bug #1010881 +++

Backport 972fc544b6034 "kvm: warn if num cpus is greater than num recommended" from uq/master in order to help enforce the recommended max vcpu limit.
Comment 1 Andrew Jones 2014-02-24 09:52:12 EST
Unsetting devel-ack for now, this may not be necessary.
Comment 2 Andrew Jones 2014-04-15 12:05:15 EDT
The qemu RHEL machine models should have had 160 instead of 255 specified for their max_cpus in order to match KVM_MAX_VCPUS in the rhel6 kernel, e.g.

diff --git a/hw/pc.c b/hw/pc.c
index 1ebcefa15b79a..05f71c41206d9 100644
--- a/hw/pc.c
+++ b/hw/pc.c
@@ -1789,7 +1789,7 @@ static QEMUMachine pc_machine_rhel650 = {
        .alias = "pc",
        .desc = "RHEL 6.5.0 PC",
        .init = pc_init_rhel650,
-       .max_cpus = 255,
+       .max_cpus = 160,
        .is_default = 1,
 };

but, at this point it's not a good idea to change them, as it could change a behavior that a user has grown to expect. We don't ever want qemu-kvm to [presumably] successfully request more than 160 vcpus though, as that could potentially lead to trouble in the kvm module. We can guard against that a different way though, than how it was proposed when this bug was opened. We just need to move this bug to the kernel and backport 338c7dbadd "KVM: Improve create VCPU parameter". I'll do that now.
Comment 3 RHEL Product and Program Management 2014-04-15 12:11:16 EDT
This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.
Comment 4 Rafael Aquini 2014-04-18 11:31:32 EDT
Patch(es) available on kernel-2.6.32-459.el6
Comment 6 Qunfang Zhang 2014-04-24 05:59:59 EDT
Hi, Andrew

When I boot up a guest with 160 vcpu on a host with 160 p-cpu, I hit the following issue: 

(1) Boot up the guest with:

"-smp 160,sockets=2,cores=40,threads=2" or "-smp 160,sockets=4,cores=20,threads=2"

Fail to boot up guest and prompt:

kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.


(2) Boot up the guest with:

smp 160,sockets=160,cores=1,threads=1

Guest could boot up successfully. 

So, this should be an issue? 


Thanks,
Qunfang
Comment 7 Andrew Jones 2014-04-24 06:13:12 EDT
(In reply to Qunfang Zhang from comment #6)
> (1) Boot up the guest with:
> 
> "-smp 160,sockets=2,cores=40,threads=2" or "-smp
> 160,sockets=4,cores=20,threads=2"
> 
> Fail to boot up guest and prompt:
> 
> kvm_create_vcpu: Invalid argument
> Failed to create vCPU. Check the -smp parameter.

Confirmed with qzhang that this used to work with the -431 kernel, so likely the patch for this bug is involved. But, this patch can't be wrong, as it simply restricts vcpu_id to the range 0-159 (160 possible vcpus). qemu is likely doing something funny when the topology is more complicated. We should open a new bug to qemu to investigate this.
Comment 8 Qunfang Zhang 2014-04-24 06:22:55 EDT
Thanks for Andrew's explanation and Bug 1090861 is filed to track the issue.
Comment 10 Andrew Jones 2014-04-29 10:52:20 EDT
*** Bug 1090861 has been marked as a duplicate of this bug. ***
Comment 11 Andrew Jones 2014-04-29 10:54:39 EDT
It looks like I messed up. KVM_CREATE_VCPU needs to support vcpu_ids up to 255, even if we only support up to 160 vcpus, because the vcpu_id should match the apic_id, and topologies with greater than one socket can have gaps in the apic_id space.
Comment 12 Rafael Aquini 2014-05-14 08:34:41 EDT
Patch(es) available on kernel-2.6.32-465.el6
Comment 14 Qunfang Zhang 2014-07-10 00:53:13 EDT
Reproduced on kernel-2.6.32-431.el6.x86_64:

  /usr/libexec/qemu-kvm -cpu SandyBridge -M rhel6.5.0 -enable-kvm -m 4096 -smp 161 -name rhel6.4-64 -uuid 9a0e67ec-f286-d8e7-0548-0c1c9ec93009 -nodefconfig -nodefaults -monitor stdio -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/home/rhel6.6.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:51:8a,bus=pci.0,addr=0x3 -chardev socket,id=charserial0,path=/tmp/isa-serial,server,nowait -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc :10 -vga std  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6  -drive if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -qmp tcp:0:5555,server,nowait -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0

kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.
Aborted (core dumped)

Verified on kernel-2.6.32-489.el6.x86_64(RHEL6.6 max guest cpu supported is 240):

(1) Boot up guest with "-smp 161": Boot up successfully, cpu number is correct inside guest. 
(2) Boot up guest with "-smp 240": Boot up successfully, cpu number is correct inside guest. 
(3) Test comment 6 "-smp 160,sockets=2,cores=40,threads=2" or "-smp 160,sockets=4,cores=20,threads=2": Boot up successfully, cpu number is correct inside guest. 
(4) Boot up guest with "-smp 241" and "-smp 255":
Could not boot up guest, prompt: "Invalid number of CPUs"

Based on above, this issue is fixed.
Comment 15 Qunfang Zhang 2014-07-10 00:57:51 EDT
Hi, Andrew

There's still a question I want to confirm with you. In rhel7 bug 1010881, guest could boot up even the guest vcpu is larger than the supported number, and qemu gives some warning. While in rhel6.6, guest could not boot up if the guest cpu is larger than supported number (240). Is this expected? 

Thanks,
Qunfang
Comment 16 Andrew Jones 2014-07-10 07:51:40 EDT
(In reply to Qunfang Zhang from comment #15)
> Hi, Andrew
> 
> There's still a question I want to confirm with you. In rhel7 bug 1010881,
> guest could boot up even the guest vcpu is larger than the supported number,
> and qemu gives some warning. While in rhel6.6, guest could not boot up if
> the guest cpu is larger than supported number (240). Is this expected? 

rhel7 should also fail now, not just output a warning. See bug 998708.
Comment 17 Jun Li 2014-09-03 03:04:28 EDT
Retest this bz:

Version of components:
# rpm -qa|grep qemu && uname -r
qemu-kvm-0.12.1.2-2.442.el6.x86_64
qemu-img-0.12.1.2-2.442.el6.x86_64
qemu-guest-agent-0.12.1.2-2.442.el6.x86_64
gpxe-roms-qemu-0.9.7-6.12.el6.noarch
2.6.32-500.el6.x86_64

CLI:
# /usr/libexec/qemu-kvm -m 2G -drive file=/home/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,cache=none,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -boot menu=on  -monitor stdio -vga qxl -spice port=5931,disable-ticketing -drive file=/home/my-data-disk.qcow2,if=none,format=qcow2,id=drive-data-disk,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk -smp 161

Results:
When boot guest with "-smp 161", can not boot up and will got following core dump:
kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffe89da2700 (LWP 4080)]
0x00007ffff483c915 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff483c915 in raise () from /lib64/libc.so.6
#1  0x00007ffff483e0f5 in abort () from /lib64/libc.so.6
#2  0x00007ffff7dda35b in ?? ()
#3  0x00007ffff76e89d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff48f2ccd in clone () from /lib64/libc.so.6
==============================

QE also test with "-smp <= 160", qemu-kvm and guest all work well.
When "-smp >160", qemu-kvm will abort, and will got the same error message with "-smp 161". 

FYI, Bug 864242 - [HP HPS 6.7 FEAT]: Increase KVM VCPU limit to 240
Comment 18 Jun Li 2014-09-03 03:22:32 EDT
Hi Andrew,

  Could you help to check comment #17. When boot guest with "-smp 161", qemu-kvm will "abort". QE think it should quit with normal error tip is ok, but now it will "abort". Do you think it's a issue? QE suspect this bz can not be modified completely. If not, could you give some explanations. Thx.


Best Regards,
Jun Li
Comment 19 Jun Li 2014-09-03 04:43:43 EDT
Retest:
1, "-smp 161" on host kernel 2.6.32-497.el6.x86_64 and qemu-kvm-0.12.1.2-2.442.el6.x86_64.
CLI:
# gdb --args /usr/libexec/qemu-kvm -m 2G -drive file=/home/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,cache=none,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -boot menu=on  -monitor stdio -vga qxl -spice port=5931,disable-ticketing -drive file=/home/my-data-disk.qcow2,if=none,format=qcow2,id=drive-data-disk,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk -smp 241

Results:
Guest can boot up successfully. qemu-kvm works well, too.


2, "-smp 161" on host kernel 2.6.32-498.el6.x86_64 and qemu-kvm-0.12.1.2-2.442.el6.x86_64.
CLI:
# gdb --args /usr/libexec/qemu-kvm -m 2G -drive file=/home/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,cache=none,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -boot menu=on  -monitor stdio -vga qxl -spice port=5931,disable-ticketing -drive file=/home/my-data-disk.qcow2,if=none,format=qcow2,id=drive-data-disk,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk -smp 241

Results:
kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffe89da2700 (LWP 13940)]
0x00007ffff483c915 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff483c915 in raise () from /lib64/libc.so.6
#1  0x00007ffff483e0f5 in abort () from /lib64/libc.so.6
#2  0x00007ffff7dda35b in ?? ()
#3  0x00007ffff76e89d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff48f2ccd in clone () from /lib64/libc.so.6
===================

As patch reverted on kernel-2.6.32-498.el6(ref: bz 864242 #c20), so for version of kernel >=  kernel-2.6.32-498.el6, should not support "-smp > 160".

But there still exist an issue:
When boot with "-smp 161", qemu-kvm should give a friendly hint. Such as(ref bz1010881 #c4):
Warning: Number of SMP cpus requested (255) exceeds the recommended cpus supported by KVM (160)

Any issue, free to update it in comments. Thx.

Best Regards,
Jun Li
Comment 20 Andrew Jones 2014-09-03 10:42:13 EDT
This is a kernel bug, and the "kvm_create_vcpu: Invalid argument" is a direct result of the kernel returning EINVAL. That error is actually even generated from code that was present before the patch for this bug was committed. IOW, none of the concerns in comment 17 are relevant to this bug. Those concerns belong in a qemu-kvm component bug, which may or may not already exist (I can't remember, and didn't try searching). However, I think there's a decent chance we'd just close a change like that for rhel6 qemu as wont-fix anyway.

So, based on the "retest (2)" of comment 19, I'd say this bug is verified.
Comment 21 Qunfang Zhang 2014-09-03 22:48:22 EDT
Hi, Andrew

However when we verified this bug in comment 14 on kernel-2.6.32-489.el6.x86_64, we boot up a guest with 241 vcpu (the limitation is 240 at that moment), qemu-kvm will neither core dump or aborted, it will fail to boot up and give a prompt "Invalid number of CPUs".  Isn't this the expected result for this bug?  Do you mean there should be a qemu-kvm bug tracker to fix such issue? 

Thanks,
Qunfang
Comment 22 Andrew Jones 2014-09-04 11:10:50 EDT
(In reply to Qunfang Zhang from comment #21)
> Hi, Andrew
> 
> However when we verified this bug in comment 14 on
> kernel-2.6.32-489.el6.x86_64, we boot up a guest with 241 vcpu (the
> limitation is 240 at that moment), qemu-kvm will neither core dump or
> aborted, it will fail to boot up and give a prompt "Invalid number of CPUs".
> Isn't this the expected result for this bug?  Do you mean there should be a
> qemu-kvm bug tracker to fix such issue? 
> 
> Thanks,
> Qunfang

My point in comment 20 is that this is a kernel bug, so you should test the kernel. Theoretically, you don't even need qemu to test the kernel, although for KVM patches there aren't much better alternatives for your kvm userspace to build tests on. qemu aborting with "kvm_create_vcpu: Invalid argument" vs. exiting with "Invalid number of CPUs" is an issue with qemu behavior, which is not relevant to the verification of this bug, and it deserves its own qemu-kvm component bug.

Now, to momentarily discuss qemu in a kernel bug, I don't see how you got different behavior from qemu when testing 241 on a kernel that supports 240 vs. 161 on a kernel that supports 160. I would expect these results

kernel version    #vcpus        result
-------------------------------------------------------
<= -475           <= 160        guest boots
<= -475            > 160        get "kvm_create_vcpu: Invalid argument"
[-476, -497]      <= 240        guest boots
[-476, -497]       > 240        get "kvm_create_vcpu: Invalid argument"
>= -498           <= 160        guest boots
>= -498            > 160        get "kvm_create_vcpu: Invalid argument"

Please recheck, and then open a qemu-kvm bug if something is not to your expectations.

BTW, the only way to verify the patch that was committed for this bug is to use > 255 vcpus, or a topology that creates a vcpu id (apic id) > 255, however in that case you'll still get EINVAL, so it'd be ambiguous with the results of testing > 160. IOW, this BZ really just needs sanity testing to be verified.
Comment 23 errata-xmlrpc 2014-10-14 01:25:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-1392.html

Note You need to log in before you can comment on or make changes to this bug.