Bug 655231

Summary: kernel 2.6.32-84.el6 breaks systemtap
Product: Red Hat Enterprise Linux 6 Reporter: Aristeu Rozanski <arozansk>
Component: kernelAssignee: Aristeu Rozanski <arozansk>
Status: CLOSED ERRATA QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.1CC: arozansk, dhoward, fche, jburke, jistone, lzheng, pbenas
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-2.6.32-85.el6 Doc Type: Bug Fix
Doc Text:
A previously introduced patch that prevented kbuild to attempt to sign an out-of-the-tree module only fixed this issue for cases when a full kernel tree was used for compiling. Using the kernel-devel package for compilation remained broken. This update allows out-of-the-tree modules to compile using the kernel-devel package only.
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 12:44:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 658879    

Description Aristeu Rozanski 2010-11-19 22:30:48 UTC
Description of problem:
On kernel 2.6.32-84, a patch was introduced to prevent signing out-of-tree
modules when compiled using the whole tree:
    http://patchwork.usersys.redhat.com/patch/29221
This patch broke a systemtap command used on RHTS:
    /usr/bin/stap -L 'kernel.trace("*")'
running this command in verbose mode:
Pass 2: getting a tracequery for 1 headers:
  /lib/modules/2.6.32-84.el6test.x86_64/build/include/trace/syscall.h
Running make -C '/lib/modules/2.6.32-84.el6test.x86_64/build' M='/tmp/stapDuyPGf/tracequery_kmod_23' modules ARCH="x86_64" V=1
make: Entering directory `/usr/src/kernels/2.6.32-84.el6test.x86_64'
test -e include/linux/autoconf.h -a -e include/config/auto.conf || (            \
        echo;                                                           \
        echo "  ERROR: Kernel configuration is invalid.";               \
        echo "         include/linux/autoconf.h or include/config/auto.conf are missing.";      \
        echo "         Run 'make oldconfig && make prepare' on kernel src to fix it.";  \
        echo;                                                           \
        /bin/false)
mkdir -p /tmp/stapDuyPGf/tracequery_kmod_23/.tmp_versions ; rm -f /tmp/stapDuyPGf/tracequery_kmod_23/.tmp_versions/*
make -f scripts/Makefile.build obj=/tmp/stapDuyPGf/tracequery_kmod_23
  gcc -Wp,-MD,/tmp/stapDuyPGf/tracequery_kmod_23/.tracequery_kmod_23.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.4.4/include -Iinclude  -I/usr/src/kernels/2.6.32-8c
  set -e ; perl /usr/src/kernels/2.6.32-84.el6test.x86_64/scripts/recordmcount.pl "x86_64" "64" "objdump" "objcopy" "gcc" "ld" "nm" "" "" "1" "/tmp/stapDuyPGf/tracequery_kmod_23/;
(cat /dev/null;   echo kernel//tmp/stapDuyPGf/tracequery_kmod_23/tracequery_kmod_23.ko;) > /tmp/stapDuyPGf/tracequery_kmod_23/modules.order
make -f /usr/src/kernels/2.6.32-84.el6test.x86_64/scripts/Makefile.modpost
  scripts/mod/modpost -m -a -i /usr/src/kernels/2.6.32-84.el6test.x86_64/Module.symvers -I /tmp/stapDuyPGf/tracequery_kmod_23/Module.symvers  -o /tmp/stapDuyPGf/tracequery_kmod_2s
  gcc -Wp,-MD,/tmp/stapDuyPGf/tracequery_kmod_23/.tracequery_kmod_23.mod.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.4.4/include -Iinclude  -I/usr/src/kernels/2.6.c
  ld -r -m elf_x86_64 -T /usr/src/kernels/2.6.32-84.el6test.x86_64/scripts/module-common.lds --build-id -o /tmp/stapDuyPGf/tracequery_kmod_23/tracequery_kmod_23.ko.unsigned /tmp/ 
/bin/sh: -c: line 0: syntax error near unexpected token `;'
/bin/sh: -c: line 0: `set -e;  ; echo 'cmd_/tmp/stapDuyPGf/tracequery_kmod_23/tracequery_kmod_23.ko := ' > /tmp/stapDuyPGf/tracequery_kmod_23/.tracequery_kmod_23.ko.cmd'
make[1]: *** [/tmp/stapDuyPGf/tracequery_kmod_23/tracequery_kmod_23.ko] Error 1
make: *** [modules] Error 2
make: Leaving directory `/usr/src/kernels/2.6.32-84.el6test.x86_64'
Spawn waitpid result (0x85122): 2

for each module and ~/.systemtap/cache will have empty .ko files. Reverting
the patch in question fixes the problem.

Comment 2 Frank Ch. Eigler 2010-11-20 02:09:15 UTC
Reproduced with i686 .84.el6 also.

Comment 3 Frank Ch. Eigler 2010-11-20 02:34:30 UTC
Here is how systemtap is invoking a failing:

% cat Makefile 
EXTRA_CFLAGS := -g -Wno-implicit-function-declaration -Werror
obj-m := tracequery_kmod_8.o

% make -C /lib/modules/2.6.32-84.el6.i686/build M=`pwd` modules ARCH=i386 V=1

It seems like a pretty vanilla kernel-devel-only sort of build.
The diff between a successful run (-44.2 kernel) and an unsuccessful one (-84):

5,8c4,8
<   CC      /var/tmp/stapXGfy54/tracequery_kmod_8/tracequery_kmod_8.mod.o - due to target missing
<   LD [M]  /var/tmp/stapXGfy54/tracequery_kmod_8/tracequery_kmod_8.ko.unsigned - due to target missing
<   NO SIGN [M] /var/tmp/stapXGfy54/tracequery_kmod_8/tracequery_kmod_8.ko - due to target missing
< make: Leaving directory `/usr/src/kernels/2.6.32-44.2.el6.i686'
---
> /bin/sh: -c: line 0: syntax error near unexpected token `;'
> /bin/sh: -c: line 0: `set -e;  ; echo 'cmd_/var/tmp/stapXGfy54/tracequery_kmod_8/tracequery_kmod_8.ko := ' > /var/tmp/stapXGfy54/tracequery_kmod_8/.tracequery_kmod_8.ko.cmd'
> make[1]: *** [/var/tmp/stapXGfy54/tracequery_kmod_8/tracequery_kmod_8.ko] Error 1
> make: *** [modules] Error 2
> make: Leaving directory `/usr/src/kernels/2.6.32-84.el6.i686'

Comment 6 RHEL Program Management 2010-11-24 14:30:05 UTC
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

Comment 9 Aristeu Rozanski 2010-12-13 15:15:45 UTC
Patch(es) available on kernel-2.6.32-89.el6

Comment 14 Mike Gahagan 2011-02-14 21:36:08 UTC
Confirmed ""stap -L 'kernel.trace("*")' -vvv"" is working on 2.6.32-114.0.1.el6.x86_64. Also observed /kernel/tracepoints/operational has run and passed on multiple systems.

Comment 15 Martin Prpič 2011-05-18 14:38:56 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
A previously introduced patch that prevented kbuild to attempt to sign an out-of-the-tree module only fixed this issue for cases when a full kernel tree was used for compiling. Using the kernel-devel package for compilation remained broken. This update allows out-of-the-tree modules to compile using the kernel-devel package only.

Comment 16 errata-xmlrpc 2011-05-19 12:44:50 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html