Bug 1099599

Another hit with:

        <vcpu>12</vcpu>

        <cpu match="exact">
                <model>SandyBridge</model>
                <topology cores="6" sockets="2" threads="1"/>
        </cpu>

Message:
 libvirtError: internal error got wrong number of vCPU pids from QEMU monitor. got 10, wanted 12


Version:
 libvirt 0.10.2-29.el6_5.7

Reproduced with:
 libvirt-0.10.2-36.el6.x86_64

For the <topology cores="3" sockets="2" threads="1"/> we start qemu-kvm as 

/usr/libexec/qemu-kvm -name
 cputest -S -M rhel6.5.0 -enable-kvm -m 256 -realtime mlock=off -smp 6,sockets=2,cores=3,threads=1 -uuid 4f7dae53-3960-e816-ebb0-aa9af5569187 -nodefconfig -nodefaults -charde
v socket,id=charmonitor,path=/var/lib/libvirt/qemu/cputest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb
-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -k en-us -vga std -device virtio-balloon-pc
i,id=balloon0,bus=pci.0,addr=0x4

The query-cpus command then returns the following output:

{
    "return": [
        {
            "enabled-in-acpi": true,
            "current": true,
            "CPU": 0,
            "pc": 4294967280,
            "halted": false,
            "thread_id": 18606
        },
        {
            "enabled-in-acpi": true,
            "current": false,
            "CPU": 1,
            "pc": 4294967280,
            "halted": true,
            "thread_id": 18607
        },
        {
            "enabled-in-acpi": true,
            "current": false,
            "CPU": 2,
            "pc": 4294967280,
            "halted": true,
            "thread_id": 18608
        },
        {
            "enabled-in-acpi": true,
            "current": false,
            "CPU": 3,
            "pc": 4294967280,
            "halted": true,
            "thread_id": 18609
        },
        {
            "enabled-in-acpi": true,
            "current": false,
            "CPU": 4,
            "pc": 4294967280,
            "halted": true,
            "thread_id": 18610
        },
        {
            "enabled-in-acpi": false,
            "current": false,
            "CPU": 5,
            "pc": 4294967280,
            "halted": true,
            "thread_id": 18611
        }
    ],
    "id": "libvirt-5"
}

This is then filtered by libvirt according to the enabled-in-acpi flag. Unfortunately CPU id 5 is disabled for some reason and thus libvirt's check fails afterwards.

Laszlo,
could you please have a look as you've worked on the enabled-in-acpi field?

Moving to qemu-kvm to investigate.

For the "-smp 6,sockets=2,cores=3,threads=1" topology, do_info_cpus() invokes acpi_query_processor() with the following APIC IDs: 0, 1, 2, 4, 5, 6.

do_info_cpus() fetches these APIC IDs by traversing the list of VCPUs and grabbing the "cpuid_apic_id" field. (I verified this with gdb.)

So, these values all fall into the very first byte of the ACPI online/offline bitmap that acpi_query_processor() reads. Let's see how the first byte looks:

(gdb) print pm_state->gpe->cpus_sts[0]
$1 = 63 '?'

Sigh. 63 decimal is 0011_1111 binary; the set bits are 0, 1, 2, 3, 4, 5.

So we have a mismatch here -- both the "cpus_sts" bitmap and the CPUX86State.cpuid_apic_id field should be indexed by / fall into the APIC ID domain. We either have wrong APIC IDs here (and they should be contiguous), or cpu_sts is incorrectly indexed by VCPU indices.

OK, the problem is in the initialization of "cpus_sts". This is a long-standing bug that hasn't been exposed before.

enable_processor() and disable_processor() (== hot(un)plug) are both invoked with APIC IDs: qemu_system_cpu_hot_add() calculates the APIC ID first with apic_id_for_cpu() -- storing it in CPUX86State.cpuid_apic_id too for new VCPUs -- and then passes that APIC ID to enable_processor() / disable_processor().

I modeled acpi_query_processor() after enable_processor() and disable_processor(), similarly taking an APIC ID as parameter.

However, the initialization of the "cpus_sts" bitmap is incorrect, we just never noticed:

void piix4_acpi_system_hot_add_init(PCIBus *bus, const char *cpu_model)
{
    int i = 0, cpus = smp_cpus;
    struct gpe_regs *gpe = &pm_state->gpe;

    while (cpus > 0) {
        gpe->cpus_sts[i++] = (cpus < 8) ? (1 << cpus) - 1 : 0xff;
        cpus -= 8;
    }
    /* ... */
}

This loop does the following: it populates the bitmap densely, from low addresses up. In the last iteration, if there are fewer than 8 CPUs, the

  (1 << cpus) - 1

expression creates a byte where the least significant "cpus" bits are set, and other bits are clear -- the power-of-two value (the result of the left-shif) has one bit set only, and by subtracting 1, the "borrow" will set all low bits to 1, until the original 1-bit is reached (and then that one is cleared).

So, this initialization loop is wrong, and has been since

commit 3685bf4bcc8bf21de4648b9c48b4cc5fa588df95
Author: Eduardo Habkost <ehabkost>
Date:   Mon Oct 29 18:52:33 2012 +0100

    CPU hotplug: use apic_id_for_cpu()

This commit correctly changed the *domain* of enable_processor() and disable_processor() from VCPU index to APIC ID -- however it missed to update the initialization of "cpus_sts", in piix4_acpi_system_hot_add_init().

Eduardo, do you agree with this analysis?

I'll prepare a small fixup and a Z-Stream brew build for the libvirt guys and the customers to test.

Thanks

Created attachment 897931 [details]
proposed fix (RHEL-6.6, RHEL-6.5.z)

verified with the same topology as above, cpus_sts[0] is now 119 decimal == 0111_0111 binary (bits 0, 1, 2, 4, 5, 6)

libvirt-python-0.10.2-29.el6_5.7.x86_64
libvirt-0.10.2-29.el6_5.7.x86_64
libvirt-client-0.10.2-29.el6_5.7.x86_64

reproduced on above version,

Ack

Error starting domain: internal error got wrong number of vCPU pids from QEMU monitor. got 5, wanted 6

Fix included in qemu-kvm-0.12.1.2-2.427.el6

Fix included in qemu-kvm-0.12.1.2-2.415.el6_5.10

Revert accidental changes described in Comment 18. They were meant for bz 1100575, not this one.

Restore status change (continuation of revert in Comment 19). The state as set by Comment 16 is now restored.

CPU created correctly

# ps aux | grep -i cpu --color
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
qemu     31225 23.3  0.1 4945532 43868 ?       Sl   14:28   0:01 /usr/libexec/qemu-kvm -name cpu -S -M rhel6.6.0 -enable-kvm -m 4096 -realtime mlock=off -smp 6,sockets=2,cores=3,threads=1 -uuid ab7939ad-9056-8931-7bf2-fd7883745bed -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/cpu.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:8f:a3:df,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
root     31260  0.0  0.0 103264   952 pts/0    S+   14:28   0:00 grep -i cpu --color
[root@dell-per415-02 ~]# rpm -q qemu-kvm-rhev libvirt
qemu-kvm-rhev-0.12.1.2-2.429.el6.x86_64
libvirt-0.10.2-29.el6.x86_64


# virsh dumpxml cpu | head -30
<domain type='kvm' id='2'>
  <name>cpu</name>
  <uuid>ab7939ad-9056-8931-7bf2-fd7883745bed</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu>
    <topology sockets='2' cores='3' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/test.img'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>


Set to verify

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1490.html