Bug 756903 - guest kernel panic when cpu threads in same physical cpu are assigned to different numa node
Summary: guest kernel panic when cpu threads in same physical cpu are assigned to diff...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.2
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Lenny Szubowicz
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: 833130 1066286 1269638
TreeView+ depends on / blocked
 
Reported: 2011-11-25 03:25 UTC by Suqin Huang
Modified: 2015-10-08 18:13 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1066286 (view as bug list)
Environment:
Last Closed: 2015-10-08 18:13:50 UTC
Target Upstream Version:


Attachments (Terms of Use)
kernel panic (6.96 KB, text/plain)
2011-11-25 03:29 UTC, Suqin Huang
no flags Details
console output of 2.6.32-287.el6 crash (12.96 KB, text/plain)
2012-08-07 19:17 UTC, Eduardo Habkost
no flags Details

Description Suqin Huang 2011-11-25 03:25:53 UTC
Description of problem:

kernel panic when boot guest with -numa node -numa node

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.209.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. boot guest with -numa node -numa node

/usr/libexec/qemu-kvm -monitor stdio  -chardev socket,id=serial_shuang,path=/tmp/serial-shuang,server,nowait -device isa-serial,chardev=serial_shuang -drive file=/home/RHEL-Server-6.2-64-virtio.qcow2,index=0,if=none,id=drive-virtio-disk1,media=disk,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idZui6UN,mac=9a:91:70:45:db:97,id=ndev00idZui6UN,bus=pci.0,addr=0x3 -netdev tap,id=idZui6UN,vhost=on,script=/home/Auto/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch -m 2048 -smp 4,cores=2,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off    -no-kvm-pit-reinjection  -M rhel6.2.0 -usb -device usb-tablet -enable-kvm -numa node -numa node

2.
3.
  
Actual results:


Expected results:


Additional info:
1. guest

rhel6.2-x86_64

Comment 1 Suqin Huang 2011-11-25 03:29:37 UTC
Created attachment 536138 [details]
kernel panic

Comment 2 Dor Laor 2011-11-27 12:07:37 UTC
Can you please try to use an accurate numa topology setup (that the host have)?
Apparently, with -node settings qemu provides numa settings that are wrong and the guest crashes (divides by zero) because of them.

It's low priority but nice to fix.

Comment 6 RHEL Program Management 2012-07-10 06:52:07 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 7 RHEL Program Management 2012-07-11 02:01:44 UTC
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.

Comment 9 Eduardo Habkost 2012-08-07 19:16:28 UTC
It's actually a guest bug, but probably low priority because it is triggered by an unusual configuration.

"-numa node -numa node" generates a round-robin VCPU configuration (VCPU 0 on node 0, VCPU 1 on node 1, VCPU 2 on node 0, VCPU 3 on node 1). I can reproduce it very easily if I use "-smp 4,threads=2 -numa node -numa node". I couldn't reproduce it using "-smp 4,cores=2 -numa node -numa node", even using the upstream QEMU.

Moving to kernel and attaching console output of a more recent kernel version. Reproduced using Fedora 17 qemu-kvm (qemu-kvm-1.0-18.fc17.x86_64). Probably it's low priority as having CPU threads on separate NUMA nodes is a very unusual situation.

The following command-line triggers a guest crash:
/usr/bin/qemu-kvm -enable-kvm -M pc -smp 4,cores=1,threads=2,sockets=2 -m 2048 -hda ~/system/vmachines/rhel63.img -net nic,model=rtl8139 -net user  -numa node -numa node -serial file:/tmp/crash

The following boots properly:
/usr/bin/qemu-kvm -enable-kvm -M pc -smp 4,cores=2,threads=1,sockets=2 -m 2048 -hda ~/system/vmachines/rhel63.img -net nic,model=rtl8139 -net user  -numa node -numa node

Guest is running kernel 2.6.32-287.el6. Crash is at:
(gdb) l *0xffffffff8105bcb1
0xffffffff8105bcb1 is in select_task_rq_fair (kernel/sched_fair.c:2173).
2168
2169                            avg_load += load;
2170                    }
2171
2172                    /* Adjust by relative CPU power of the group */
2173                    avg_load = (avg_load * SCHED_LOAD_SCALE) / group->cpu_power;
2174
2175                    if (local_group) {
2176                            this_load = avg_load;
2177                            this = group;
(gdb)

Comment 10 Eduardo Habkost 2012-08-07 19:17:17 UTC
Created attachment 602835 [details]
console output of 2.6.32-287.el6 crash


Note You need to log in before you can comment on or make changes to this bug.