Description of problem: By default "perf" cannot get user-space stack traces unless the program is built with -fno-omit-frame-pointer -ggdb . On x86_64 Fedora omits frame pointers by default, so perf can only get the symbol under the current instruction pointer register, but not the stack. perf was patched in 3.6 to add support for "perf record -g dwarf", which uses libunwind to analyse the stack even if the dso was built with -fomit-frame-pointer. However, Fedora's kernels are not built with libunwind-devel installed, so perf doesn't use it and doesn't enable -g dwarf. The Fedora kernel builds perf with NO_LIBUNWIND=1 and doesn't build-depend on libunwind, so the Fedora perf for x86_64 is vastly less useful. Version-Release number of selected component (if applicable): perf-3.11.6-200.fc19.x86_64
Bug #863781 suggests that libunwind is being phased out from Fedora and replaced with the elfutils unwinder. So perhaps perf needs to be taught to use the elfutils unwinder instead, but until then it should be built with libunwind. See http://stackoverflow.com/questions/19719911/getting-user-space-stack-information-from-perf and its references.
https://lists.fedoraproject.org/pipermail/devel/2012-November/174579.html Libunwind isn't really supported in Fedora and wasn't even before elf-utils was ready. I'll look at revisiting the elf-utils support, but we aren't going to build against libunwind again.
Jiri, looking at upstream perf, it seems that still doesn't have support for using unwind-libdw instead of libunwind. Is that accurate? Can you update us on what is happening there, given that it was supposedly heading into 3.8?
I just checked elfutils and it seems the unwind support is still not in upstream.. Jan, please correct me if I'm wrong. As for perf, there's initial code working with some early version of the elfutils unwinder. It needs to be revisited and fixed to work with the latest interface. I plan to do that.
elfutils with unwinder should be released in two weeks, depending on the s390* and ppc* reviews upstream. x86* part should be checked in this week.
Thanks Jiri and Jan. I think what we'll do is use this bug to track the building of perf with elfutils unwind so we don't forget. I'm updating it accordingly.
Thank-you very much for taking a look at this. I can understand not wanting to add a new dependency on an obsolete library being phased out. For anybody who needs this functionality in the mean time you can rebuild the kernel with the instructions given in the above-linked Stack Overflow question, either to produce a new perf rpm for your current kernel, a whole new kernel, or just a stand-alone perf binary you can use. You need to rebuild if you're unable to get perf to collect user-space stack traces, only the entry-point from user-space into the kernel.
elfutils 0.158 has been released and is in fedora rawhide. Jiri posted a patch to use it from perf to lkml: http://thread.gmane.org/gmane.linux.kernel/1624492
I believe we can now build perf with libdw support with 3.15. I'm going to look at doing this tomorrow. I see that F20 has an eflutils 0.158 update. Is there any chance F19 is going to be updated to that as well?
Hm. Actually, if I understand the commits correctly it should have automatically been picked up in recent builds. However, on my machine it seems to fail the feature check? make -s -j8 -C tools/perf V=1 WERROR=0 NO_LIBUNWIND=1 HAVE_CPLUS_DEMANGLE=1 NO_GTK2=1 NO_LIBNUMA=1 NO_STRLCPY=1 NO_BIONIC=1 prefix=/usr DESTDIR=/home/jwboyer/rpmbuild/BUILDROOT/kernel-3.15.0-0.rc0.git7.1.fc21.x86_64 all BUILD: Doing 'make ^[[33m-j8^[[m' parallel build config/Makefile:288: No libdw DWARF unwind found, Please install elfutils-devel/libdw-dev >= 0.158 and/or set LIBDW_DIR config/Makefile:337: Disabling post unwind, no support found. Auto-detecting system features: ... dwarf: [ ^[[32mon^[[m ] ... glibc: [ ^[[32mon^[[m ] ... gtk2: [ ^[[32mon^[[m ] ... libaudit: [ ^[[32mon^[[m ] ... libbfd: [ ^[[32mon^[[m ] ... libelf: [ ^[[32mon^[[m ] ... libnuma: [ ^[[31mOFF^[[m ] ... libperl: [ ^[[32mon^[[m ] ... libpython: [ ^[[32mon^[[m ] ... libslang: [ ^[[32mon^[[m ] ... libunwind: [ ^[[32mon^[[m ] ... libdw-dwarf-unwind: [ ^[[31mOFF^[[m ] So we explicitly turned off libunwind with NO_LIBUNWIND=1, but it says it's on and I have libdw.so installed but it says it can't find it. I have both elfutils-devel and libunwind-devel installed: [jwboyer@vader linux]$ rpm -q libunwind-devel elfutils-devel libunwind-devel-1.1-3.fc20.x86_64 elfutils-devel-0.158-1.fc20.x86_64 [jwboyer@vader linux]$ It should have turned libunwind off without searching, but didn't and it should have found libdw, but didn't. If I build the libdw feature check code manually, it seems to find things just fine: [jwboyer@vader feature-checks]$ LDFLAGS=-ldw make test-libdw-dwarf-unwind.bin gcc -MD -o test-libdw-dwarf-unwind.bin test-libdw-dwarf-unwind.c -ldw [jwboyer@vader feature-checks]$ pwd /home/jwboyer/src/linux/tools/perf/config/feature-checks [jwboyer@vader feature-checks]$ Jiri, any idea what is going on here?
The latest build log in koji shows libunwind support being disabled, likely because libunwind-devel isn't installed. It still doesn't add the libdw-dwarf-unwind support though, even though eflutils-devel 0.158-2 is installed. http://kojipkgs.fedoraproject.org//packages/kernel/3.15.0/0.rc0.git5.1.fc21/data/logs/x86_64/
(In reply to Josh Boyer from comment #9) > I see that F20 has an eflutils 0.158 update. Is there any chance F19 is > going to be updated to that as well? I submitted an update: https://admin.fedoraproject.org/updates/elfutils-0.158-2.fc19
Thanks Mark. I think I also figured out why the autodetecting isn't working. I sent a patch to Jiri and the upstream maintainers. Once I hear back on whether it's correct or not I can add it and things should automagically work.
http://thread.gmane.org/gmane.linux.kernel/1677349 is the patch for posterity.
A similar fix from someone else was picked up. It's included in the 3.15.0-0.rc1.git3.1 build I just started. Should land in Rawhide tomorrow.
Or not. Mark, the build for perf with libdw fails on ARM with: libperf.a(unwind-libdw.o): In function `.LANCHOR0': unwind-libdw.c:(.rodata+0x1c): undefined reference to `libdw__arch_set_initial_registers' collect2: error: ld returned 1 exit status make[1]: *** [perf] Error 1 make[1]: *** Waiting for unfinished jobs.... perf did detect libdw-unwind being present, so it's at least available in some form in the buildroot: Auto-detecting system features: ... dwarf: [ [32mon[m ] ... glibc: [ [32mon[m ] ... gtk2: [ [31mOFF[m ] ... libaudit: [ [32mon[m ] ... libbfd: [ [32mon[m ] ... libelf: [ [32mon[m ] ... libnuma: [ [31mOFF[m ] ... libperl: [ [32mon[m ] ... libpython: [ [32mon[m ] ... libslang: [ [32mon[m ] ... libunwind: [ [31mOFF[m ] ... libdw-dwarf-unwind: [ [32mon[m ] ... DWARF post unwind library: libdw Any ideas?
Oh, nevermind Mark. That undefined function looks like something that perf itself is providing. Jiri, is libdw support only available on x86 for perf? If so, shouldn't the configuration system disable it on non-x86 architectures?
(In reply to Josh Boyer from comment #17) > Oh, nevermind Mark. That undefined function looks like something that perf > itself is providing. > > Jiri, is libdw support only available on x86 for perf? If so, shouldn't the > configuration system disable it on non-x86 architectures? I remember the change for arm is done, but not merged yet looks like bug in configuration system, you could use following workaround and disable it manualy: $ make NO_LIBDW_DWARF_UNWIND=1 I'll try to look on it soon jirka
Created attachment 887237 [details] perf tools: Disable libdw unwind on non x86 archs Could you check this one on arm? It should disable libdw unwind on all archs but x86. thanks
(In reply to Jiri Olsa from comment #19) > Created attachment 887237 [details] > perf tools: Disable libdw unwind on non x86 archs > > Could you check this one on arm? It should disable libdw unwind on > all archs but x86. > > thanks Sure. I'll give it a shot today.
(In reply to Josh Boyer from comment #20) > (In reply to Jiri Olsa from comment #19) > > Created attachment 887237 [details] > > perf tools: Disable libdw unwind on non x86 archs > > > > Could you check this one on arm? It should disable libdw unwind on > > all archs but x86. > > > > thanks > > Sure. I'll give it a shot today. Yep, that seems to have worked. Thanks Jiri. Should be included in today's build.
Built and in rawhide. Thanks everyone.