RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 865380 - Kernel oops/crash when running perf on a SandyBridge host
Summary: Kernel oops/crash when running perf on a SandyBridge host
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.4
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: rc
: ---
Assignee: Jiri Olsa
QA Contact: Zhang Kexin
URL:
Whiteboard: virt
: 869216 871329 890962 (view as bug list)
Depends On:
Blocks: 1300182
TreeView+ depends on / blocked
 
Reported: 2012-10-11 10:03 UTC by Qunfang Zhang
Modified: 2019-07-12 07:39 UTC (History)
20 users (show)

Fixed In Version: kernel-2.6.32-338.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-02-21 06:49:57 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Host crash log (59.08 KB, text/plain)
2012-10-11 10:06 UTC, Qunfang Zhang 		no flags Details
dmesg of guest with -cpu SandyBridge (23.27 KB, text/plain)
2012-10-16 05:51 UTC, Qunfang Zhang 		no flags Details
host crash log when running perf on host without guest running (61.35 KB, text/plain)
2012-10-16 05:51 UTC, Qunfang Zhang 		no flags Details
kernel module source (1.29 KB, text/plain)
2012-10-23 11:23 UTC, Gleb Natapov 		no flags Details
fix (661 bytes, patch)
2012-10-23 16:05 UTC, Jiri Olsa 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0496 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 6 kernel update 2013-02-20 21:40:54 UTC

Description Qunfang Zhang 2012-10-11 10:03:22 UTC 
Description of problem:
Boot a guest on a SandyBridge host, host always crash. In the crash log, there's many pmu related message. I'm not sure whether this is related to the bug Bug 852083 in kernel side. I will try more to narrow down the problem and update here.
Hit this issue in kernel-328, and did not found it in kernel-323 or earlier kernel-315.

Version-Release number of selected component (if applicable):
kernel-2.6.32-328.el6.x86_64
qemu-kvm-0.12.1.2-2.320.el6.x86_64

How reproducible:
Sometimes

Steps to Reproduce:
1. Boot a guest on a SandyBridge host:
# /usr/libexec/qemu-kvm -M rhel6.3.0 -cpu SandyBridge -m 2048 -smp 2,sockets=2,cores=1,threads=1 -enable-kvm -name rhel6.4-64 -uuid feebc8fd-f8b0-4e75-abc3-e63fcdb67170 -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 -k en-us -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=input0 -drive file=/home/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=disk0 -device virtio-scsi-pci,id=disk0 -device scsi-hd,drive=disk0,scsi-id=0,lun=0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:23:AE:7A:6E:10,bus=pci.0,addr=0x4  -monitor stdio -qmp tcp:0:6666,server,nowait -boot c -bios /usr/share/seabios/bios-pm.bin -chardev socket,path=/tmp/isa-serial,server,nowait,id=isa1 -device isa-serial,chardev=isa1,id=isa-serial1 -drive if=none,id=drive-fdc0-0-0,readonly=on,format=raw -global isa-fdc.driveA=drive-fdc0-0-0 -vnc :10


2. Wait guest boot up or reboot it after finish boot.

3.
  
Actual results:
Host crash

Expected results:
Host works well

Additional info:

------------[ cut here ]------------
WARNING: at arch/x86/kernel/cpu/perf_event.c:1352 x86_pmu_stop+0x138/0x150() (Not tainted)
Hardware name: HP Compaq 8200 Elite MT PC
Modules linked in: autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm uinput sg microcode serio_raw i2c_i801 iTCO_wdt iTCO_vendor_support e1000e snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif ahci wmi radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 2947, comm: qemu-kvm Not tainted 2.6.32-328.el6.x86_64 #1
Call Trace:
 [<ffffffff8106b927>] ? warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b97a>] ? warn_slowpath_null+0x1a/0x20
 [<ffffffff8101be38>] ? x86_pmu_stop+0x138/0x150
 [<ffffffff810138f3>] ? native_sched_clock+0x13/0x80
 [<ffffffff81012c89>] ? sched_clock+0x9/0x10
 [<ffffffff8101ca3d>] ? x86_pmu_enable+0x9d/0x280
 [<ffffffff8110ea8b>] ? perf_pmu_enable+0x2b/0x40
 [<ffffffff811114f7>] ? perf_event_context_sched_in+0xa7/0xe0
 [<ffffffff811119d4>] ? __perf_event_task_sched_in+0x64/0x70
 [<ffffffff81057ddd>] ? finish_task_switch+0xad/0xe0
 [<ffffffff815109c0>] ? thread_return+0x4e/0x76e
 [<ffffffff8112bf07>] ? __alloc_pages_nodemask+0x87/0x8d0
 [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8106231a>] ? __cond_resched+0x2a/0x40
 [<ffffffff81511380>] ? _cond_resched+0x30/0x40
 [<ffffffffa038a354>] ? kvm_resched+0x24/0x30 [kvm]
 [<ffffffffa03a0d93>] ? kvm_arch_vcpu_ioctl_run+0xda3/0x1060 [kvm]
 [<ffffffffa0389172>] ? kvm_vcpu_ioctl+0x522/0x670 [kvm]
 [<ffffffff8101cab4>] ? x86_pmu_enable+0x114/0x280
 [<ffffffff81111519>] ? perf_event_context_sched_in+0xc9/0xe0
 [<ffffffff81195e72>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff8119633a>] ? do_vfs_ioctl+0x3aa/0x580
 [<ffffffff81196591>] ? sys_ioctl+0x81/0xa0
 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
---[ end trace 7dc20f705fb7ccca ]---
Comment 1 Qunfang Zhang 2012-10-11 10:04:55 UTC 
Host cpu info:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
stepping	: 7
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
bogomips	: 6784.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:
Comment 2 Qunfang Zhang 2012-10-11 10:06:28 UTC 
Created attachment 625475 [details]
Host crash log
Comment 5 Qunfang Zhang 2012-10-11 11:01:02 UTC 
Retest on the SandyBridge host on kernel-2.6.32-328.el6.x86_64:

1. "-M rhel6.4.0 -cpu SandyBridge" for more than 10 times. ==> Passed.
2. "-M rhel6.3.0"(defualt -cpu ) ==> Reproduced at the third time.
3. "-M rhel6.3.0 -cpu SandyBridge" ==> Reproduce at the third time.

Re-test on an old Conroe host on kernel-2.6.32-328.el6.x86_64:
1. "-M rhel6.3.0" ==> Passed after 20 times attempts.
Comment 6 Gleb Natapov 2012-10-12 10:59:09 UTC 
Can you attach guest dmesg with -cpu SandyBridge? Also can you make sure that the crahs does not happen when you run "perf record -e cycles firefox" (without any guest running on the machine).
Comment 7 Qunfang Zhang 2012-10-16 05:46:31 UTC 
(In reply to comment #6)
> Can you attach guest dmesg with -cpu SandyBridge? Also can you make sure
> that the crahs does not happen when you run "perf record -e cycles firefox"
> (without any guest running on the machine).

Hi, Gleb
Guest dmesg with -cpu SandyBridge will be upload. 
The crash also happens when I run "perf record -e cycles firefox" (without any guest running). Attachment will be upload as well.
Comment 8 Qunfang Zhang 2012-10-16 05:51:07 UTC 
Created attachment 627958 [details]
dmesg of guest with -cpu SandyBridge
Comment 9 Qunfang Zhang 2012-10-16 05:51:56 UTC 
Created attachment 627959 [details]
host crash log when running perf on host without guest running
Comment 10 Gleb Natapov 2012-10-16 07:39:54 UTC 
(In reply to comment #7)
> The crash also happens when I run "perf record -e cycles firefox" (without
> any guest running). Attachment will be upload as well.

Thanks you.

This is KVM unrelated problem. KVM creates PMU counter just like perf does. Assigning back to kernel.
Comment 11 Don Zickus 2012-10-16 15:05:51 UTC 
(In reply to comment #9)
> Created attachment 627959 [details]
> host crash log when running perf on host without guest running

Hi Qunfang,

Show stack trace shows guests running.  The panic itself happens within qemu-kvm.  Can you re-run your test without qemu running?

Thanks,
Don
Comment 13 Don Zickus 2012-10-16 19:41:28 UTC 
Hi Qunfang,

You don't happen to have a test setup do you?  I tried setting up a guest and failed miserably on 6.4.

I ran a 'perf record -e cycles grep -ri blah /*' on the host Sandybridge box with the -328.el6 kernel successfully.  Was trying to see if qemu caused issues.

Cheers,
Don
Comment 14 Qunfang Zhang 2012-10-17 09:21:03 UTC 
(In reply to comment #11)
> (In reply to comment #9)
> > Created attachment 627959 [details]
> > host crash log when running perf on host without guest running
> 
> Hi Qunfang,
> 
> Show stack trace shows guests running.  The panic itself happens within
> qemu-kvm.  Can you re-run your test without qemu running?
> 
> Thanks,
> Don
Hi, Don
I remember I test without guest running. Currently the host is using by someone else. I will re-run later after get the host.  Thanks.
Comment 15 Ademar Reis 2012-10-17 13:39:38 UTC 
(In reply to comment #13)
> Hi Qunfang,
> 
> You don't happen to have a test setup do you?  I tried setting up a guest
> and failed miserably on 6.4.
> 

You mean you need the setup for starting a guest, just to have qemu running? If that's the case, you should be able to use virt-manager. Install qemu-kvm, libvirt-daemon and then virt-manager, start the libvirt service and then invocate virt-manager. The rest should be fairly simple.
Comment 16 Don Zickus 2012-10-17 15:01:14 UTC 
Hi Ademar,

I tried, there are various 6.4 bugs that were in my way.  So I gave up.  Dave Allan helped my through some.  Also I was trying to do this remotely and virt-manager is not working over my ssh connection.  I am not sure why port forwarding is not working.

I will try running the virt install QE test to setup me up for 6.3 and then migrate over to the 6.4 qemu tools.

Cheers,
Don
Comment 17 Don Zickus 2012-10-18 20:52:16 UTC 
Hi,

So I finally had help to setup a RHEV guest.  However, it uses qemu-kvm-rhev and friends instead of qemu-kvm.  As a result I haven't been able to duplicate this problem.

I installed RHEL-6.3, added some rhev pkgs and then installed a -328.el6 kernel on the host. Used RHEV to install a 6.3 guest.  Rebooted the guest multiple times, no warning.

Another co-worker only sees this problem on RHEV with a -324.el6 kernel.  I can't duplicate that either.

Kinda stuck.  Is there a machine I can play with to investigate the issue more?

Cheers,
Don
Comment 23 Gleb Natapov 2012-10-23 11:23:43 UTC 
Created attachment 632035 [details]
kernel module source

I was able to reproduce the crash (or may be different one but related) without any guest running by using attached module. Compile it and load in a loop like this:

while true; do insmod perfev.ko; done

After a couple of iteration kernel crashes with:

create event ffffffffa0b69000
counter=539 enabled=4027 running=4027
counter=10000743 enabled=10156430 running=3381
release event ffffffffa0b69000
exit ffffffffa0b69000
BUG: unable to handle kernel paging request at ffffffffa0b69000
IP: [<ffffffffa0b69000>] 0xffffffffa0b69000
PGD 1a87067 PUD 1a8b063 PMD 46e29c067 PTE 0
Oops: 0010 [#1] SMP
last sysfs file: /sys/devices/virtual/misc/autofs/uevent
CPU 13
Modules linked in: netconsole configfs autofs4 fuse nfsd exportfs nfs nfs_acl auth_rpcgss fscache lockd sunrpc kvm_intel kvm ipv6 ext2 dm_crypt snd_pcsp snd_pcm snd_page_alloc cdc_ether i2c_i801 i7core_edac snd_timer usbnet iTCO_wdt serio_raw i2c_core shpchp mii edac_core ioatdma dca iTCO_vendor_support snd soundcore ext3 mbcache jbd btrfs(T) libcrc32c lzo_compress lzo_decompress zlib_deflate dm_mod sr_mod sd_mod crc_t10dif cdrom usb_storage mptsas mptscsih mptbase bnx2 scsi_transport_sas [last unloaded: scsi_wait_scan]


Pid: 3625, comm: bash Tainted: G        W  ---------------  T 2.6.32 #10 IBM IBM System x -[7870B3G]-/49Y5178
RIP: 0010:[<ffffffffa0b69000>]  [<ffffffffa0b69000>] 0xffffffffa0b69000
RSP: 0018:ffff880287547cd0  EFLAGS: 00010086
RAX: ffffffffa0b69000 RBX: ffff88046e153c00 RCX: ffff880287547f58
RDX: ffff880287547dd8 RSI: 0000000000000001 RDI: ffff88046e153c00
RBP: ffff880287547d68 R08: ffff880287547f58 R09: ffff880287547dd8
R10: ffff88046ee3be00 R11: 0000000000000007 R12: 0000000000000001
R13: 0000000000000000 R14: ffff880287547f58 R15: 0000000000000000
FS:  00007f979819a700(0000) GS:ffff880287540000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffa0b69000 CR3: 000000046ed7f000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 3625, threadinfo ffff88046ee3a000, task ffff88046eae0aa0)
Stack:
 ffffffff81112540 0000000000000000 0000000000000000 0000000000000000
<d> 0000000000000000 0000000000000000 0000000000000000 0000000000000000
<d> 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 <NMI>
 [<ffffffff81112540>] ? __perf_event_overflow+0xb0/0x2a0
 [<ffffffff81112b54>] perf_event_overflow+0x14/0x20
 [<ffffffff8101eb26>] intel_pmu_handle_irq+0x336/0x550
 [<ffffffff8150e976>] ? kprobe_exceptions_notify+0x16/0x450
 [<ffffffff8150d4d9>] perf_event_nmi_handler+0x39/0xb0
 [<ffffffff8150efc6>] notifier_call_chain+0x56/0x80
 [<ffffffff8150f02a>] atomic_notifier_call_chain+0x1a/0x20
 [<ffffffff81098bde>] notify_die+0x2e/0x30
 [<ffffffff8150cc5b>] do_nmi+0x1bb/0x340
 [<ffffffff8150c510>] nmi+0x20/0x30
 [<ffffffff8127edf8>] ? strnlen_user+0x78/0x90
 <<EOE>>
 [<ffffffff811847df>] copy_strings+0x7f/0x240
 [<ffffffff811854e2>] do_execve+0x1e2/0x2c0
 [<ffffffff810095ca>] sys_execve+0x4a/0x80
 [<ffffffff8100b48a>] stub_execve+0x6a/0xc0
Code:  Bad RIP value.
RIP  [<ffffffffa0b69000>] 0xffffffffa0b69000
 RSP <ffff880287547cd0>
CR2: ffffffffa0b69000
---[ end trace a252b2f0a2ddb53e ]---

So it looks like even callback is called after event is released by perf_event_release_kernel() and module is unloaded.

Kernel 279 runs this while insmod loop without any problem.
Comment 24 Jiri Olsa 2012-10-23 16:05:05 UTC 
Created attachment 632179 [details]
fix

fixies the leftover from:

[kernel] perf: Change and simplify ctx::is_active semantics
commit c103845d8ab9c98a97dee342ac86a496937a7a26
Author: Jiri Olsa <jolsa>
Date:   Fri Oct 5 13:54:54 2012 -0400

it fixies the issue in my tests
Comment 31 Michael S. Tsirkin 2012-10-31 09:53:17 UTC 
*** Bug 869216 has been marked as a duplicate of this bug. ***
Comment 33 Jarod Wilson 2012-11-01 20:25:12 UTC 
Patch(es) available on kernel-2.6.32-338.el6
Comment 37 Shaolong Hu 2012-11-15 03:03:05 UTC 
*** Bug 871329 has been marked as a duplicate of this bug. ***
Comment 40 errata-xmlrpc 2013-02-21 06:49:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0496.html
Comment 41 Gleb Natapov 2013-05-07 08:31:10 UTC 
*** Bug 890962 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.