Description of problem: SystemTap cannot find any kernel tracepoints because the 32-bit ARM opcodes.h file is missing from the kernel-devel rpm. This is due to a recent change in sysreg.h which includes opcodes.h (see Additional info below). ~]# uname -r 4.4.0-0.rc4.git0.1.fc24.aarch64 ~]# stap --disable-cache -L 'kernel.trace("*")' | grep -o '".*"' | wc -l 0 If you manually copy the opcodes.h file from the prepped src rpm to /usr/src, then it works (be sure to clean the ~/.systemtap cache): ~]# cp /home/rpmbuild/rpmbuild/BUILD/kernel-4.3.fc23/linux-4.4.0-0.rc4.git0.1.fc23.aarch64/arch/arm/include/asm/opcodes.h \ /usr/src/kernels/4.4.0-0.rc4.git0.1.fc24.aarch64/arch/arm/include/asm/ ~]# stap --disable-cache -L 'kernel.trace("*")' | grep -o '".*"' | wc -l 439 Version-Release number of selected component (if applicable): kernel-4.4.0-0.rc4.git0.1.fc24.aarch64 How reproducible: always Steps to Reproduce: 1. rm -fr ~/.systemtap 2. stap --disable-cache -L 'kernel.trace("*")' | grep -o '".*"' | wc -l Actual results: 0 tracepoints found Expected results: 1+ tracepoints found Additional info: For reference, opcodes.h was added to sysreg.h as part of commit: 338d4f49d6f7 arm64: kernel: Add support for Privileged Access Never $ git describe --contains 338d4f49d6f7 v4.3-rc1~84^2~67 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=338d4f49d6f7 Because of this change, if the 32-bit opcodes.h is missing, stap with verbose output shows this error starting with sysreg.h: ~]# stap --disable-cache --vp 09 -l 'kernel.trace("*")' >/tmp/stap-vp09.out 2>&1 ~]# less /tmp/stap-vp09.out ... gcc -Wp,-MD,/tmp/stap5LkqWM/tracequery_kmod_1/.tracequery_kmod_1_10.o.d -nost dinc -isystem /usr/lib/gcc/aarch64-redhat-linux/5.1.1/include -I./arch/arm64/inc lude -Iarch/arm64/include/generated/uapi -Iarch/arm64/include/generated -Iinclu de -I./arch/arm64/include/uapi -Iarch/arm64/include/generated/uapi -I./include/u api -Iinclude/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -ml ittle-endian -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasi ng -fno-common -Werror-implicit-function-declaration -Wno-format-security -std=g nu89 -mgeneral-regs-only -fno-delete-null-pointer-checks -O2 --param=allow-store -data-races=0 -Wframe-larger-than=1024 -fno-stack-protector -Wno-unused-but-set- variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fvar-tracking-assi gnments -pg -fno-inline-functions-called-once -Wdeclaration-after-statement -Wno -pointer-sign -fno-strict-overflow -fconserve-stack -Werror=implicit-int -Werror =strict-prototypes -Werror=date-time -DCC_HAVE_ASM_GOTO -g -Wno-implicit-functio n-declaration -Werror -fno-ipa-icf -DMODULE -mcmodel=large -D"KBUILD_STR(s)=#s " -D"KBUILD_BASENAME=KBUILD_STR(tracequery_kmod_1_10)" -D"KBUILD_MODNAME=KBUILD _STR(tracequery_kmod_1_10)" -c -o /tmp/stap5LkqWM/tracequery_kmod_1/tracequery_k mod_1_10.o /tmp/stap5LkqWM/tracequery_kmod_1/tracequery_kmod_1_10.c In file included from ./arch/arm64/include/asm/sysreg.h:23:0, from ./arch/arm64/include/asm/cpufeature.h:13, from ./arch/arm64/include/asm/hw_breakpoint.h:20, from ./arch/arm64/include/asm/processor.h:33, from ./arch/arm64/include/asm/spinlock.h:21, from include/linux/spinlock.h:87, from include/linux/rcupdate.h:38, from include/linux/tracepoint.h:19, from /tmp/stap5LkqWM/tracequery_kmod_1/tracequery_kmod_1_3.c:2: ./arch/arm64/include/asm/opcodes.h:1:43: fatal error: ../../arm/include/asm/opcodes.h: No such file or directory In file included from ./arch/arm64/include/asm/sysreg.h:23:0, from ./arch/arm64/include/asm/cpufeature.h:13, from ./arch/arm64/include/asm/hw_breakpoint.h:20, from ./arch/arm64/include/asm/processor.h:33, from ./arch/arm64/include/asm/spinlock.h:21, from include/linux/spinlock.h:87, from include/linux/rcupdate.h:38, from include/linux/tracepoint.h:19, from /tmp/stap5LkqWM/tracequery_kmod_1/tracequery_kmod_1_2.c:2: ./arch/arm64/include/asm/opcodes.h:1:43: fatal error: ../../arm/include/asm/opcodes.h: No such file or directory compilation terminated. compilation terminated.
Created attachment 1106134 [details] proposed patch to include 32-bit arm opcodes.h 64-bit ARM opcodes.h references the 32-bit version: $ cat arch/arm64/include/asm/opcodes.h #include <../../arm/include/asm/opcodes.h> This proposed patch includes the 32-bit ARM opcodes.h in the kernel-devel rpm.
An alternative to the patch in comment 1 might be to include all of 32-bit arm/include/* in kernel-devel for aarch64. I don't think it will hurt anything other than increasing the size of kernel-devel by 1.4MB. This will help to avoid problems in the future if more aarch64 headers reference the 32-bit version. cp -a --parents arch/arm/include $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/
Committed your patch and pushed it out to all branches. The alternative of adding all of them is tempting, but I'd rather avoid that for now and go with only the known needed files. It'll mean we might hit this again, but I'd rather do that than get errant includes in a build just because they are there.
> 64-bit ARM opcodes.h references the 32-bit version: > $ cat arch/arm64/include/asm/opcodes.h > #include <../../arm/include/asm/opcodes.h> > > This proposed patch includes the 32-bit ARM opcodes.h in the kernel-devel > rpm. This file is already in the upstream kernel sources: arch/arm/include/asm/opcodes.h checking "git log arch/arm/include/asm/opcodes.h" on a upstream linus kernel checkout tells me it's been there since "Mon Dec 12 19:31:55 2011 +0100". So is the problem here that we're just not shipping it?
(In reply to Peter Robinson from comment #4) > > 64-bit ARM opcodes.h references the 32-bit version: > > $ cat arch/arm64/include/asm/opcodes.h > > #include <../../arm/include/asm/opcodes.h> > > > > This proposed patch includes the 32-bit ARM opcodes.h in the kernel-devel > > rpm. > > This file is already in the upstream kernel sources: > arch/arm/include/asm/opcodes.h > > checking "git log arch/arm/include/asm/opcodes.h" on a upstream linus kernel > checkout tells me it's been there since "Mon Dec 12 19:31:55 2011 +0100". So > is the problem here that we're just not shipping it? Yes. And Jeff's patch fixes it, which is what I committed today.