Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 756903

Summary: guest kernel panic when cpu threads in same physical cpu are assigned to different numa node
Product: Red Hat Enterprise Linux 6 Reporter: Suqin Huang <shuang>
Component: kernelAssignee: Lenny Szubowicz <lszubowi>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2CC: areis, jshortt, juzhang, michen, mkenneth, qzhang, rbalakri, rpacheco, tburke, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1066286 (view as bug list) Environment:
Last Closed: 2015-10-08 18:13:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 833130, 1066286, 1269638    
Attachments:
Description Flags
kernel panic
none
console output of 2.6.32-287.el6 crash none

Description Suqin Huang 2011-11-25 03:25:53 UTC
Description of problem:

kernel panic when boot guest with -numa node -numa node

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.209.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. boot guest with -numa node -numa node

/usr/libexec/qemu-kvm -monitor stdio  -chardev socket,id=serial_shuang,path=/tmp/serial-shuang,server,nowait -device isa-serial,chardev=serial_shuang -drive file=/home/RHEL-Server-6.2-64-virtio.qcow2,index=0,if=none,id=drive-virtio-disk1,media=disk,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idZui6UN,mac=9a:91:70:45:db:97,id=ndev00idZui6UN,bus=pci.0,addr=0x3 -netdev tap,id=idZui6UN,vhost=on,script=/home/Auto/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch -m 2048 -smp 4,cores=2,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off    -no-kvm-pit-reinjection  -M rhel6.2.0 -usb -device usb-tablet -enable-kvm -numa node -numa node

2.
3.
  
Actual results:


Expected results:


Additional info:
1. guest

rhel6.2-x86_64

Comment 1 Suqin Huang 2011-11-25 03:29:37 UTC
Created attachment 536138 [details]
kernel panic

Comment 2 Dor Laor 2011-11-27 12:07:37 UTC
Can you please try to use an accurate numa topology setup (that the host have)?
Apparently, with -node settings qemu provides numa settings that are wrong and the guest crashes (divides by zero) because of them.

It's low priority but nice to fix.

Comment 6 RHEL Program Management 2012-07-10 06:52:07 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 7 RHEL Program Management 2012-07-11 02:01:44 UTC
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.

Comment 9 Eduardo Habkost 2012-08-07 19:16:28 UTC
It's actually a guest bug, but probably low priority because it is triggered by an unusual configuration.

"-numa node -numa node" generates a round-robin VCPU configuration (VCPU 0 on node 0, VCPU 1 on node 1, VCPU 2 on node 0, VCPU 3 on node 1). I can reproduce it very easily if I use "-smp 4,threads=2 -numa node -numa node". I couldn't reproduce it using "-smp 4,cores=2 -numa node -numa node", even using the upstream QEMU.

Moving to kernel and attaching console output of a more recent kernel version. Reproduced using Fedora 17 qemu-kvm (qemu-kvm-1.0-18.fc17.x86_64). Probably it's low priority as having CPU threads on separate NUMA nodes is a very unusual situation.

The following command-line triggers a guest crash:
/usr/bin/qemu-kvm -enable-kvm -M pc -smp 4,cores=1,threads=2,sockets=2 -m 2048 -hda ~/system/vmachines/rhel63.img -net nic,model=rtl8139 -net user  -numa node -numa node -serial file:/tmp/crash

The following boots properly:
/usr/bin/qemu-kvm -enable-kvm -M pc -smp 4,cores=2,threads=1,sockets=2 -m 2048 -hda ~/system/vmachines/rhel63.img -net nic,model=rtl8139 -net user  -numa node -numa node

Guest is running kernel 2.6.32-287.el6. Crash is at:
(gdb) l *0xffffffff8105bcb1
0xffffffff8105bcb1 is in select_task_rq_fair (kernel/sched_fair.c:2173).
2168
2169                            avg_load += load;
2170                    }
2171
2172                    /* Adjust by relative CPU power of the group */
2173                    avg_load = (avg_load * SCHED_LOAD_SCALE) / group->cpu_power;
2174
2175                    if (local_group) {
2176                            this_load = avg_load;
2177                            this = group;
(gdb)

Comment 10 Eduardo Habkost 2012-08-07 19:17:17 UTC
Created attachment 602835 [details]
console output of 2.6.32-287.el6 crash