Bug 795329

Summary: perf record ends with segfault when profiling kvm event
Product: Red Hat Enterprise Linux 6 Reporter: Xiaoqing Wei <xwei>
Component: kernelAssignee: Jiri Olsa <jolsa>
kernel sub component: Perf QA Contact: Guangze Bai <gbai>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: medium CC: acme, czhang, juzhang, michen, shuang, tpnoonan, yshao
Version: 6.3   
Target Milestone: rc   
Target Release: 6.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 811156 903559 (view as bug list) Environment:
Last Closed: 2012-08-29 10:58:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 632768, 811156, 903559    
Attachments:
Description Flags
gdb thread apply all bt full
none
core dump itself. none

Description Xiaoqing Wei 2012-02-20 09:26:30 UTC 
Description of problem:
perf code dumps when profiling kvm event

perf kvm --host --guest record -p $qemu-kvm_pid
Version-Release number of selected component (if applicable):

kernel-2.6.32-232.el6.x86_64
perf-2.6.32-232.el6.x86_64
qemu-kvm-0.12.1.2-2.231.el6.x86_64
How reproducible:

4 / 5
Steps to Reproduce:
1. boot a qemu-kvm guest, 
2. perf kvm --host --guest record -p $qemu-kvm_pid
3. ctrl+c to end the profiling 
  
Actual results:
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.029 MB perf.data.kvm (~1267 samples) ]
Segmentation fault (core dumped)
echo $?
139
Expected results:
should exit without core dump
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.034 MB perf.data.kvm (~1472 samples) ]

# echo $?
130

Additional info:

Tried above test on Fedora16 
kernel-3.2.6-3.fc16.x86_64,qemu-kvm-0.15.1-4.fc16.x86_64 
 didn't core dump.
Comment 1 Xiaoqing Wei 2012-02-20 09:27:25 UTC 
Created attachment 564353 [details]
gdb thread apply all bt full
Comment 2 Xiaoqing Wei 2012-02-20 09:29:53 UTC 
Created attachment 564355 [details]
core dump itself.

detail "gdb thread apply all bt full" is attached, but for who would like to have quick glance:


#0  machine__findnew_thread (self=0x0, pid=5952) at util/thread.c:75
#0  machine__findnew_thread (self=0x0, pid=5952) at util/thread.c:75
#1  0x000000000042d899 in build_id__mark_dso_hit (tool=<value optimized out>, event=<value optimized out>, sample=<value optimized out>, evsel=<value optimized out>, machine=0x0) at util/build-id.c:27
#2  0x000000000045444c in perf_session__process_event (session=0x29f2070, event=0x7ffedfe0fde0, tool=0x6ac0c0, file_offset=48608) at util/session.c:907
#3  0x000000000045506f in __perf_session__process_events (session=0x29f2070, data_offset=<value optimized out>, data_size=<value optimized out>, file_size=303056, tool=0x6ac0c0) at util/session.c:1136
#4  0x0000000000417f76 in process_buildids (status=<value optimized out>, arg=0x6ab100) at builtin-record.c:308
#5  perf_record__exit (status=<value optimized out>, arg=0x6ab100) at builtin-record.c:321
#6  0x00000039fe035d8a in __run_exit_handlers (status=0) at exit.c:64
#7  exit (status=0) at exit.c:100
#8  0x000000000040faa2 in handle_internal_command (argc=6, argv=0x7ffff07787e0) at perf.c:344
#9  run_argv (argc=6, argv=0x7ffff07787e0) at perf.c:388
#10 main (argc=6, argv=0x7ffff07787e0) at perf.c:486
Comment 4 Jiri Olsa 2012-04-02 08:48:48 UTC 
[jolsa@krava perf]$ pgrep qemu-kvm
3536
[jolsa@krava perf]$ sudo ./perf kvm --host --guest record -p 3536
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.017 MB perf.data.kvm (~739 samples) ]

cannot reproduce on the 259 build.

There were some perf tool backports since build 232. Please
retest and let me know if thats still an issue.
Comment 5 Xiaoqing Wei 2012-04-10 07:50:24 UTC 
#uname -r
2.6.32-259.el6.x86_64
# perf kvm --host --guest record -p 3401
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.034 MB perf.data.kvm (~1467 samples) ]
Segmentation fault (core dumped)
# echo $?
139

perf-2.6.32-259.el6.x86_64
Comment 6 Jiri Olsa 2012-04-12 10:04:46 UTC 
(In reply to comment #5)
> #uname -r
> 2.6.32-259.el6.x86_64
> # perf kvm --host --guest record -p 3401
> ^C[ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.034 MB perf.data.kvm (~1467 samples) ]
> Segmentation fault (core dumped)
> # echo $?
> 139
> 
> perf-2.6.32-259.el6.x86_64

ok, reproduced finally ;)

upstream is not failing because of following commit:

perf top: Don't process samples with no valid machine object
commit 0c095715b388d19d7a0b7e8eaceeceb018f5b3d1
Author: Joerg Roedel <joerg.roedel>
Date:   Fri Feb 10 18:05:04 2012 +0100

but that just workaround the real issue, which is that once you specify
--guest you also need to specify one of:
        --guestmount <directory>
        --guestvmlinux <file>
        --guestkallsyms <file>
        --guestmodules <file>

otherwise guest OS mappings are not recorded -> samples with invalid
machines are processed (during buildid record processing):
   - in RHEL6 segfault happens
   - in upstream special 'unprocessable sample' counter is incremented

working on fix for upstream
Comment 7 Jiri Olsa 2012-04-12 12:32:31 UTC 
fix sent upstream:
https://lkml.org/lkml/2012/4/12/156
Comment 10 Timothy Noonan 2012-05-17 15:54:17 UTC 
this defect blocked verification of ibm feature 67210 - 632768 [6.3 FEAT] add guest profiling support to perf tool
Comment 11 Suzanne Logcher 2012-05-18 20:53:46 UTC 
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 12 Jiri Olsa 2012-08-29 10:58:08 UTC 
fixed by BZ  846702 - perf tool 3.5 sync

*** This bug has been marked as a duplicate of bug 846702 ***