Bug 1200685

Created attachment 1000288 [details]
host cpu info

As qemu-kvm-rhev-2.1.2-23.el7 didn't hit this issue.  So set this bug is as regression bug,  If I am wrong, please remove regression from keywords. 

Another, 7.1 works well.

sorry, correct it comment 3,  7.1/7.1 guest.

FuXiangChun,

Please always attach full console/demesg logs from guest/host to BZ.

Also
could you please reproduce issue, leave it in hang state and provide access to host where it happened.

Created attachment 1000749 [details]
guest full console logs

Created attachment 1000750 [details]
host full dmesg

Yanhui Ma,

Could you try to reproduce bug with RHEL6.6 guest kernel?

(In reply to Igor Mammedov from comment #10)
> Yanhui Ma,
> 
> Could you try to reproduce bug with RHEL6.6 guest kernel?

I have reproduced bug with RHEL6.6 guest kernel.The guest hang is not 100% reproduced, call trace and only 4 cpus successfully hotpluged  can be 100% reproduced. Another I have attached the RHEL6.6 guest full console logs.
You can access the host to see it according to comment 9.

Created attachment 1001271 [details]
RHEL6.6 guest full console logs

So here goes my analysis:

 1. I wasn't able to reproduce bug locally (probably due to lack of effort)
    but it's reproducible reliably on host in comment 9

 2. issue has nothing to do with cpu-hotplug, regular boot is affected as well when CPU count 5 and we have 2 NUMA nodes, here is striped down reproducer:

 qemu-kvm -m 4G -smp 5,sockets=1,cores=4,threads=1,maxcpus=8 -numa node,nodeid=0 -numa node,nodeid=1 -drive file=/home/rhel66-64-virtio.qcow2,if=virtio

 3. CPU[0] hangs in smp_call_function_many() waiting on execution of call on CPU[4], but CPU[4] loops in update_sd_lb_stats() due to incorrectly initialized
sched_groups looping on condition "while (sg != sd->groups)" since the last 'sg->next' instead of pointing to head 'sd->groups' points to itself.

crash> bt -a
PID: 1      TASK: ffff88007f307500  CPU: 0   COMMAND: "swapper"
    [exception RIP: smp_call_function_many+482]
[...]
 #0 [ffff88007f309ce8] smp_call_function at ffffffff810c7c22
 #1 [ffff88007f309cf8] on_each_cpu at ffffffff8108e1fd
 #2 [ffff88007f309d28] do_tune_cpucache at ffffffff8118186b
 #3 [ffff88007f309d98] enable_cpucache at ffffffff81181d36
 #4 [ffff88007f309dc8] setup_cpu_cache at ffffffff8150fcc2
 #5 [ffff88007f309e08] kmem_cache_create at ffffffff81182232
 #6 [ffff88007f309eb8] shmem_init at ffffffff81c4f27e
 #7 [ffff88007f309ed8] kernel_init at ffffffff81c29ef6
 #8 [ffff88007f309f48] kernel_thread at ffffffff8100c1ca
[...]

PID: 27     TASK: ffff88013f28eae0  CPU: 4   COMMAND: "events/4"
[...]
 #0 [ffff88013f295a58] update_sd_lb_stats at ffffffff8106fc8d
 #1 [ffff88013f295b38] find_busiest_group at ffffffff8106fe7a
 #2 [ffff88013f295c18] load_balance_newidle at ffffffff81070a57
 #3 [ffff88013f295d08] idle_balance at ffffffff8107e38e
 #4 [ffff88013f295d88] schedule at ffffffff815304e0
 #5 [ffff88013f295e38] worker_thread at ffffffff810aa640
 #6 [ffff88013f295ee8] kthread at ffffffff810af4d0
 #7 [ffff88013f295f48] kernel_thread at ffffffff8100c1ca

 4. Issue is caused by commit:
     dd0247e0 pc: acpi: mark all possible CPUs as enabled in SRAT
    which makes kernel actually use QEMU supplied numa mapping in SRAT for vCPUs, before that commit guest kernel was discarding CPU related SRAT info.

 5. Problem is that QEMU by default distributes vCPUs among NUMA nodes in round-robin order, which leads to insane topology where VCPU threads from one socket(package) end up in different NUMA nodes.

Setting cpus to node mapping manually with sane topology (i.e. threads from the same socket are on the same node), makes bug go away:
 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7

 Yanhui Ma,

1. Could you test if problem affects RHEL7, RHEL5 and Windows guests?

2. Also after #1 try following scratch build with fix:
https://brewweb.devel.redhat.com/taskinfo?taskID=8861393

to reproduce on Intel host following options have to be added to reproducer from comment 13:
 -cpu Opteron_G3,vendor=AuthenticAMD

Upstream fix posted:
https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg04008.html

Tested with different Windows versions, it's 90% reproducible with WS2012x64,
also seen it with WS2012R2x64 but only once.

Windows fails to boot going into reboot cycle with C4 error.

(In reply to Igor Mammedov from comment #14)
>  Yanhui Ma,
> 
> 1. Could you test if problem affects RHEL7, RHEL5 and Windows guests?
> 
Steps to Reproduce:
1. boot a HELR7.1/RHEL5.11/win-server-2008r2 guest with following qemu command line:

 /usr/libexec/qemu-kvm -M pc-i440fx-rhel7.1.0 -m 4G -cpu Opteron_G3,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time -smp 5,sockets=1,cores=4,threads=1,maxcpus=8 -numa node,nodeid=0 -numa node,nodeid=1 \

-monitor stdio -vga qxl -spice port=5900,disable-ticketing  \

-drive file=/home/rhel59-64-virtio.qcow2,if=none,id=drive-data-disk1,cache=none,format=qcow2,aio=threads,werror=stop,rerror=stop -device ide-drive,drive=drive-data-disk1,id=data-disk1 \

-netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,id=virtio-net-pci0,mac=00:24:21:7f:b6:11,bus=pci.0,addr=0x9 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -usb -device usb-tablet,id=input0


results:

RHEL5.11 and RHEL6.7 hit the issue, RHEL7.1 and win-server-2008r2 does not hit the issue.

> 2. Also after #1 try following scratch build with fix:
> https://brewweb.devel.redhat.com/taskinfo?taskID=8861393


test again with the above fixed build, RHEL5.11,RHEL6.7,RHEL7.1 and win-server-2008r2 can not hit the issue.

Fixed upstream in 2.3

fb43b73 pc: fix default VCPU to NUMA node mapping
57924bc numa: introduce machine callback for VCPU to node mapping

Please retest when qemu-kvm-rhev is rebased to 2.3 version.

Reproduce this issue.

host info:
qemu-kvm-rhev-2.2.0-8.el7.x86_64
3.10.0-230.el7.x86_64

Steps to Reproduce:
1. boot a RHEL6.7/win2012r2x64 guest with following qemu command line:

 /usr/libexec/qemu-kvm -M pc-i440fx-rhel7.1.0 -m 4G -cpu Opteron_G3,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_time -smp 5,sockets=1,cores=4,threads=1,maxcpus=8 -numa node,nodeid=0 -numa node,nodeid=1 \

-monitor stdio -vga qxl -spice port=5900,disable-ticketing  \

-drive file=/home/RHEL-Server-6.7-64-virtio.qcow2,if=none,id=drive-data-disk1,cache=none,format=qcow2,aio=threads,werror=stop,rerror=stop -device ide-drive,drive=drive-data-disk1,id=data-disk1 \

-netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,id=virtio-net-pci0,mac=00:24:21:7f:b6:11,bus=pci.0,addr=0x9 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -usb -device usb-tablet,id=input0

results:
rhel6.7 guest hit call trace info during boot,see attachment, win2012r2x64 guest does not hit the issue.

#########################################################

Verified this issue.
host info:
3.10.0-297.el7.x86_64
qemu-kvm-rhev-2.3.0-12.el7.x86_64

steps is the same as above

Result:
qemu prints out following info:
cpu topology: error: sockets (1) * cores (4) * threads (1) < smp_cpus (5)

hi, Igor
Is it the expected result? I remember guest can boot up successfully with scratch build in comment 14. If the qemu error is the expected result, then the bug has been fixed.

Created attachment 1056469 [details]
call trace info

(In reply to Yanhui Ma from comment #23)
> Reproduce this issue.
[...]
> Result:
> qemu prints out following info:
> cpu topology: error: sockets (1) * cores (4) * threads (1) < smp_cpus (5)
> 
> hi, Igor
> Is it the expected result? I remember guest can boot up successfully with
> scratch build in comment 14. If the qemu error is the expected result, then
> the bug has been fixed.

bug was fixed upstream by commit
 fb43b73b "pc: fix default VCPU to NUMA node mapping"
and error message was introduced upstream by 
 ec2cbbdd8 "vl: Don't silently change topology when all -smp options were set"
after 2.2 release which scratch build is based on.

So yes, I'd say exit with error is expected result.

(In reply to Igor Mammedov from comment #25)
> (In reply to Yanhui Ma from comment #23)
> > Reproduce this issue.
> [...]
> > Result:
> > qemu prints out following info:
> > cpu topology: error: sockets (1) * cores (4) * threads (1) < smp_cpus (5)
> > 
> > hi, Igor
> > Is it the expected result? I remember guest can boot up successfully with
> > scratch build in comment 14. If the qemu error is the expected result, then
> > the bug has been fixed.
> 
> bug was fixed upstream by commit
>  fb43b73b "pc: fix default VCPU to NUMA node mapping"
> and error message was introduced upstream by 
>  ec2cbbdd8 "vl: Don't silently change topology when all -smp options were
> set"
> after 2.2 release which scratch build is based on.
> 
> So yes, I'd say exit with error is expected result.

Thanks, if so, I think the bug was fixed.

According to comment23-comment26, set this issue as verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html