Bug 1010882

Unsetting devel-ack for now, this may not be necessary.

The qemu RHEL machine models should have had 160 instead of 255 specified for their max_cpus in order to match KVM_MAX_VCPUS in the rhel6 kernel, e.g.

diff --git a/hw/pc.c b/hw/pc.c
index 1ebcefa15b79a..05f71c41206d9 100644
--- a/hw/pc.c
+++ b/hw/pc.c
@@ -1789,7 +1789,7 @@ static QEMUMachine pc_machine_rhel650 = {
        .alias = "pc",
        .desc = "RHEL 6.5.0 PC",
        .init = pc_init_rhel650,
-       .max_cpus = 255,
+       .max_cpus = 160,
        .is_default = 1,
 };

but, at this point it's not a good idea to change them, as it could change a behavior that a user has grown to expect. We don't ever want qemu-kvm to [presumably] successfully request more than 160 vcpus though, as that could potentially lead to trouble in the kvm module. We can guard against that a different way though, than how it was proposed when this bug was opened. We just need to move this bug to the kernel and backport 338c7dbadd "KVM: Improve create VCPU parameter". I'll do that now.

This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.

Patch(es) available on kernel-2.6.32-459.el6

Hi, Andrew

When I boot up a guest with 160 vcpu on a host with 160 p-cpu, I hit the following issue: 

(1) Boot up the guest with:

"-smp 160,sockets=2,cores=40,threads=2" or "-smp 160,sockets=4,cores=20,threads=2"

Fail to boot up guest and prompt:

kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.


(2) Boot up the guest with:

smp 160,sockets=160,cores=1,threads=1

Guest could boot up successfully. 

So, this should be an issue? 


Thanks,
Qunfang

(In reply to Qunfang Zhang from comment #6)
> (1) Boot up the guest with:
> 
> "-smp 160,sockets=2,cores=40,threads=2" or "-smp
> 160,sockets=4,cores=20,threads=2"
> 
> Fail to boot up guest and prompt:
> 
> kvm_create_vcpu: Invalid argument
> Failed to create vCPU. Check the -smp parameter.

Confirmed with qzhang that this used to work with the -431 kernel, so likely the patch for this bug is involved. But, this patch can't be wrong, as it simply restricts vcpu_id to the range 0-159 (160 possible vcpus). qemu is likely doing something funny when the topology is more complicated. We should open a new bug to qemu to investigate this.

Thanks for Andrew's explanation and Bug 1090861 is filed to track the issue.

*** Bug 1090861 has been marked as a duplicate of this bug. ***

It looks like I messed up. KVM_CREATE_VCPU needs to support vcpu_ids up to 255, even if we only support up to 160 vcpus, because the vcpu_id should match the apic_id, and topologies with greater than one socket can have gaps in the apic_id space.

Patch(es) available on kernel-2.6.32-465.el6

Reproduced on kernel-2.6.32-431.el6.x86_64:

  /usr/libexec/qemu-kvm -cpu SandyBridge -M rhel6.5.0 -enable-kvm -m 4096 -smp 161 -name rhel6.4-64 -uuid 9a0e67ec-f286-d8e7-0548-0c1c9ec93009 -nodefconfig -nodefaults -monitor stdio -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/home/rhel6.6.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:51:8a,bus=pci.0,addr=0x3 -chardev socket,id=charserial0,path=/tmp/isa-serial,server,nowait -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc :10 -vga std  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6  -drive if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -qmp tcp:0:5555,server,nowait -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0

kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.
Aborted (core dumped)

Verified on kernel-2.6.32-489.el6.x86_64(RHEL6.6 max guest cpu supported is 240):

(1) Boot up guest with "-smp 161": Boot up successfully, cpu number is correct inside guest. 
(2) Boot up guest with "-smp 240": Boot up successfully, cpu number is correct inside guest. 
(3) Test comment 6 "-smp 160,sockets=2,cores=40,threads=2" or "-smp 160,sockets=4,cores=20,threads=2": Boot up successfully, cpu number is correct inside guest. 
(4) Boot up guest with "-smp 241" and "-smp 255":
Could not boot up guest, prompt: "Invalid number of CPUs"

Based on above, this issue is fixed.

Hi, Andrew

There's still a question I want to confirm with you. In rhel7 bug 1010881, guest could boot up even the guest vcpu is larger than the supported number, and qemu gives some warning. While in rhel6.6, guest could not boot up if the guest cpu is larger than supported number (240). Is this expected? 

Thanks,
Qunfang

(In reply to Qunfang Zhang from comment #15)
> Hi, Andrew
> 
> There's still a question I want to confirm with you. In rhel7 bug 1010881,
> guest could boot up even the guest vcpu is larger than the supported number,
> and qemu gives some warning. While in rhel6.6, guest could not boot up if
> the guest cpu is larger than supported number (240). Is this expected? 

rhel7 should also fail now, not just output a warning. See bug 998708.

Retest this bz:

Version of components:
# rpm -qa|grep qemu && uname -r
qemu-kvm-0.12.1.2-2.442.el6.x86_64
qemu-img-0.12.1.2-2.442.el6.x86_64
qemu-guest-agent-0.12.1.2-2.442.el6.x86_64
gpxe-roms-qemu-0.9.7-6.12.el6.noarch
2.6.32-500.el6.x86_64

CLI:
# /usr/libexec/qemu-kvm -m 2G -drive file=/home/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,cache=none,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -boot menu=on  -monitor stdio -vga qxl -spice port=5931,disable-ticketing -drive file=/home/my-data-disk.qcow2,if=none,format=qcow2,id=drive-data-disk,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk -smp 161

Results:
When boot guest with "-smp 161", can not boot up and will got following core dump:
kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffe89da2700 (LWP 4080)]
0x00007ffff483c915 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff483c915 in raise () from /lib64/libc.so.6
#1  0x00007ffff483e0f5 in abort () from /lib64/libc.so.6
#2  0x00007ffff7dda35b in ?? ()
#3  0x00007ffff76e89d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff48f2ccd in clone () from /lib64/libc.so.6
==============================

QE also test with "-smp <= 160", qemu-kvm and guest all work well.
When "-smp >160", qemu-kvm will abort, and will got the same error message with "-smp 161". 

FYI, Bug 864242 - [HP HPS 6.7 FEAT]: Increase KVM VCPU limit to 240

Hi Andrew,

  Could you help to check comment #17. When boot guest with "-smp 161", qemu-kvm will "abort". QE think it should quit with normal error tip is ok, but now it will "abort". Do you think it's a issue? QE suspect this bz can not be modified completely. If not, could you give some explanations. Thx.


Best Regards,
Jun Li

Retest:
1, "-smp 161" on host kernel 2.6.32-497.el6.x86_64 and qemu-kvm-0.12.1.2-2.442.el6.x86_64.
CLI:
# gdb --args /usr/libexec/qemu-kvm -m 2G -drive file=/home/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,cache=none,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -boot menu=on  -monitor stdio -vga qxl -spice port=5931,disable-ticketing -drive file=/home/my-data-disk.qcow2,if=none,format=qcow2,id=drive-data-disk,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk -smp 241

Results:
Guest can boot up successfully. qemu-kvm works well, too.


2, "-smp 161" on host kernel 2.6.32-498.el6.x86_64 and qemu-kvm-0.12.1.2-2.442.el6.x86_64.
CLI:
# gdb --args /usr/libexec/qemu-kvm -m 2G -drive file=/home/RHEL-Server-6.6-64-virtio.raw,if=none,id=img,cache=none,snapshot=on -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -boot menu=on  -monitor stdio -vga qxl -spice port=5931,disable-ticketing -drive file=/home/my-data-disk.qcow2,if=none,format=qcow2,id=drive-data-disk,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-data-disk,id=data-disk -smp 241

Results:
kvm_create_vcpu: Invalid argument
Failed to create vCPU. Check the -smp parameter.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffe89da2700 (LWP 13940)]
0x00007ffff483c915 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff483c915 in raise () from /lib64/libc.so.6
#1  0x00007ffff483e0f5 in abort () from /lib64/libc.so.6
#2  0x00007ffff7dda35b in ?? ()
#3  0x00007ffff76e89d1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff48f2ccd in clone () from /lib64/libc.so.6
===================

As patch reverted on kernel-2.6.32-498.el6(ref: bz 864242 #c20), so for version of kernel >=  kernel-2.6.32-498.el6, should not support "-smp > 160".

But there still exist an issue:
When boot with "-smp 161", qemu-kvm should give a friendly hint. Such as(ref bz1010881 #c4):
Warning: Number of SMP cpus requested (255) exceeds the recommended cpus supported by KVM (160)

Any issue, free to update it in comments. Thx.

Best Regards,
Jun Li

This is a kernel bug, and the "kvm_create_vcpu: Invalid argument" is a direct result of the kernel returning EINVAL. That error is actually even generated from code that was present before the patch for this bug was committed. IOW, none of the concerns in comment 17 are relevant to this bug. Those concerns belong in a qemu-kvm component bug, which may or may not already exist (I can't remember, and didn't try searching). However, I think there's a decent chance we'd just close a change like that for rhel6 qemu as wont-fix anyway.

So, based on the "retest (2)" of comment 19, I'd say this bug is verified.

Hi, Andrew

However when we verified this bug in comment 14 on kernel-2.6.32-489.el6.x86_64, we boot up a guest with 241 vcpu (the limitation is 240 at that moment), qemu-kvm will neither core dump or aborted, it will fail to boot up and give a prompt "Invalid number of CPUs".  Isn't this the expected result for this bug?  Do you mean there should be a qemu-kvm bug tracker to fix such issue? 

Thanks,
Qunfang

(In reply to Qunfang Zhang from comment #21)
> Hi, Andrew
> 
> However when we verified this bug in comment 14 on
> kernel-2.6.32-489.el6.x86_64, we boot up a guest with 241 vcpu (the
> limitation is 240 at that moment), qemu-kvm will neither core dump or
> aborted, it will fail to boot up and give a prompt "Invalid number of CPUs".
> Isn't this the expected result for this bug?  Do you mean there should be a
> qemu-kvm bug tracker to fix such issue? 
> 
> Thanks,
> Qunfang

My point in comment 20 is that this is a kernel bug, so you should test the kernel. Theoretically, you don't even need qemu to test the kernel, although for KVM patches there aren't much better alternatives for your kvm userspace to build tests on. qemu aborting with "kvm_create_vcpu: Invalid argument" vs. exiting with "Invalid number of CPUs" is an issue with qemu behavior, which is not relevant to the verification of this bug, and it deserves its own qemu-kvm component bug.

Now, to momentarily discuss qemu in a kernel bug, I don't see how you got different behavior from qemu when testing 241 on a kernel that supports 240 vs. 161 on a kernel that supports 160. I would expect these results

kernel version    #vcpus        result
-------------------------------------------------------
<= -475           <= 160        guest boots
<= -475            > 160        get "kvm_create_vcpu: Invalid argument"
[-476, -497]      <= 240        guest boots
[-476, -497]       > 240        get "kvm_create_vcpu: Invalid argument"
>= -498           <= 160        guest boots
>= -498            > 160        get "kvm_create_vcpu: Invalid argument"

Please recheck, and then open a qemu-kvm bug if something is not to your expectations.

BTW, the only way to verify the patch that was committed for this bug is to use > 255 vcpus, or a topology that creates a vcpu id (apic id) > 255, however in that case you'll still get EINVAL, so it'd be ambiguous with the results of testing > 160. IOW, this BZ really just needs sanity testing to be verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-1392.html