Bug 820487

Created attachment 583452 [details]
ystemtap-make-installcheck-log.txt

Created attachment 583453 [details]
systemtap-log.tgz

Basesystem is just dependency order package with no content. Reassigning to systemtap (possibly for further reassignment, I haven't checked what's the culprit).

Created attachment 583510 [details]
Patch for bz6503 testcase.


------- Comment on attachment From srinivasa.tn.com 2012-05-10 11:19 EDT-------


Hi Frank,
   I have modified the bz6503.* to get the proper testing done.  I am attaching the patch for the same.

Regards,
Seenu.

Thanks for your patch for bz6503.exp.  While the patch gets the testcase working again, that test really needs some improvement in general.

The original test used jffs2/ext2 and your change makes it use fat/vfat.  However, if the system had a filesystem mounted using one of those filesystem types, this test will likely fail (since the module is already loaded) or perhaps unmount the filesystem.

Also, this test will fail if not run as root (since it runs modprobe/rmmod commands without checking for root access).

For the test, we should probably write a small kernel module that can be loaded/unloaded and probe that.  Some C code and test code could be borrowed from other tests, like systemtap.context/context.exp.

The problems with this test aren't ppc64 specific, it fails in the same way on all other platforms.  So, the bz6503.exp problems should be split off from this bug into its own bug.

Frank convinced me that writing a small kernel module was overkill for test
bz6503.esp.  I've updated the test in the following commit:

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=92da1faec06b276bec4a0c14515280bdc579b601>

This update adds the fat/vfat modules to the list of modules to test and fixes
the test execution for non-root users.

If the system being tested has a jffs2/ext2/fat/vfat filesystem mounted, the
test might fail, but the mounted filesystem will remain.

(In reply to comment #0)

... stuff deleted ...

>    I analysed the failures and found that some packages were missing on the
> system.  Missing packages are:
> 
> crash-6.0.5-1.fc17.ppc64
> systemtap-server-1.7-5.fc17.ppc64
> systemtap-initscript-1.7-5.fc17.ppc64
> systemtap-debuginfo-1.7-5.fc17.ppc64
> systemtap-grapher-1.7-5.fc17.ppc64
> glibc-debuginfo-2.15-32.fc17.ppc64
> glibc-debuginfo-common-2.15-32.fc17.ppc64
> avahi-0.6.30-7.fc17.ppc64
> avahi-debuginfo-0.6.30-7.fc17.ppc64

Let's walk through these.  A "regular" rpm can't require debuginfo rpms, so that takes care of 4 of those additional rpms (systemtap-debuginfo-1.7-5.fc17.ppc64, glibc-debuginfo-2.15-32.fc17.ppc64, glibc-debuginfo-common-2.15-32.fc17.ppc64, and avahi-debuginfo-0.6.30-7.fc17.ppc64).  The systemtap-grapher rpm has been deprecated upstream and isn't needed for testing.  The systemtap-initscript rpm isn't needed for testing.

I'm surprised that the systemtap-testsuite rpm doesn't require the systemtap-server rpm.  It should.

I see that testsuite/systemtap.base/crash.exp runs the 'crash' script from the crash rpm, so the systemtap-testsuite rpm should also require the crash rpm.

Do you remember what test started working once the avahi rpm was installed?  Since the systemtap-server rpm doesn't require that rpm, I'm surprised it was needed.

------- Comment From srinivasa.tn.com 2012-05-11 10:47 EDT-------
(In reply to comment #16)
> (In reply to comment #0)
\>
> Let's walk through these.  A "regular" rpm can't require debuginfo rpms, so
> that takes care of 4 of those additional rpms
> (systemtap-debuginfo-1.7-5.fc17.ppc64, glibc-debuginfo-2.15-32.fc17.ppc64,
> glibc-debuginfo-common-2.15-32.fc17.ppc64, and
> avahi-debuginfo-0.6.30-7.fc17.ppc64).

Did not get what you mean by that.  I did an

debuginfo-install glibc-2.15-32.fc17.ppc64
debuginfo-install systemtap-client-1.7-5.fc17.ppc64 systemtap-devel-1.7-5.fc17.ppc64

And the above commands intalled the systemtap-grapher and systemtap-initscript.

The systemtap-grapher rpm has been
> deprecated upstream and isn't needed for testing.  The systemtap-initscript
> rpm isn't needed for testing.
> I'm surprised that the systemtap-testsuite rpm doesn't require the
> systemtap-server rpm.  It should.

The systemtap-server rpm is there in the above list (systemtap-server-1.7-5.fc17.ppc64)

> I see that testsuite/systemtap.base/crash.exp runs the 'crash' script from
> the crash rpm, so the systemtap-testsuite rpm should also require the crash
> rpm.
>
> Do you remember what test started working once the avahi rpm was installed?
> Since the systemtap-server rpm doesn't require that rpm, I'm surprised it
> was needed.

After installing the avahi daemon, following testcases started working:

/usr/share/systemtap/testsuite/systemtap.server/client.exp

Also, we request you to look into the other failure cases.

(In reply to comment #8)
> ------- Comment From srinivasa.tn.com 2012-05-11 10:47 EDT-------
> (In reply to comment #16)
> > (In reply to comment #0)
> \>
> > Let's walk through these.  A "regular" rpm can't require debuginfo rpms, so
> > that takes care of 4 of those additional rpms
> > (systemtap-debuginfo-1.7-5.fc17.ppc64, glibc-debuginfo-2.15-32.fc17.ppc64,
> > glibc-debuginfo-common-2.15-32.fc17.ppc64, and
> > avahi-debuginfo-0.6.30-7.fc17.ppc64).
> 
> Did not get what you mean by that.  I did an
> 
> debuginfo-install glibc-2.15-32.fc17.ppc64
> debuginfo-install systemtap-client-1.7-5.fc17.ppc64
> systemtap-devel-1.7-5.fc17.ppc64
> 
> And the above commands intalled the systemtap-grapher and systemtap-initscript.

I'm sorry I confused you.  I meant that the systemtap-testsuite can't require those debuginfo rpms.  So, installing them will be a manual step.

> The systemtap-grapher rpm has been
> > deprecated upstream and isn't needed for testing.  The systemtap-initscript
> > rpm isn't needed for testing.
> > I'm surprised that the systemtap-testsuite rpm doesn't require the
> > systemtap-server rpm.  It should.
> 
> The systemtap-server rpm is there in the above list
> (systemtap-server-1.7-5.fc17.ppc64)

I've confused you again.  The systemtap-testsuite rpm should require the systemtap-server rpm so that the systemtap-server rpm gets automatically installed.
 
> > I see that testsuite/systemtap.base/crash.exp runs the 'crash' script from
> > the crash rpm, so the systemtap-testsuite rpm should also require the crash
> > rpm.
> >
> > Do you remember what test started working once the avahi rpm was installed?
> > Since the systemtap-server rpm doesn't require that rpm, I'm surprised it
> > was needed.
> 
> After installing the avahi daemon, following testcases started working:
> 
> /usr/share/systemtap/testsuite/systemtap.server/client.exp

I've learned that parts of the client.exp test case test automatic server detection which uses avahi.  The avahi rpm isn't required by the systemtap-server rpm, because clients can contact servers directly (without using avahi).


I've updated the systemtap-testsuite rpm's requires in commit d6d8634.

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=d6d8634e46c520ecdf5675f0590963497c206d4f>

> FAIL: systemtap.examples/general/badname build

Failed with errors like this:

OUT semantic error: not accessible at this address (0xc00000000024a364, dieoffset: 0x124ba95): identifier '$child' at badname.stp:16:7
        source:   if ($child->d_inode || $dir->i_flags & 16) next

It looks like systemtap is having trouble finding $child.  To debug
this one further, I'll need to see the output of the following
command:

# stap -L 'kernel.function("may_create@fs/namei.c")'

> FAIL: systemtap.examples/network/netdev build

Failed with the following errors;

OUT semantic error: not accessible at this address (0xc0000000006b75d0, dieoffset: 0x37f697e): identifier '$dev' at /usr/share/systemtap/tapset/networking.stp:159:29
        source: 	dev_name = get_netdev_name($dev)
                	                           ^
semantic error: not accessible at this address (0xc0000000006b75d0, dieoffset: 0x37f6975): identifier '$flags' at :160:10
        source: 	flags = $flags

This one was worked around with upstream commit 021bc9024, which isn't
in systemtap 1.7.

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=021bc902453a4b7375ca5b575c1bf1c980d43f81>

> FAIL: buildok/dentry-embedded.stp

semantic error: unable to find member 'mnt_parent' for struct vfsmount (alternatives: mnt_root mnt_sb mnt_flags): operator '->' at /usr/share/systemtap/tapset/dentry.stp:104:54
        source:                         if (@cast(vfsmnt, "vfsmount")->mnt_parent == vfsmnt)semantic error: unable to find member 'mnt_parent' for struct vfsmount (alternatives: mnt_root mnt_sb mnt_flags): operator '->' at /usr/share/systemtap/tapset/dentry.stp:104:54

This is bug PR13670 ("on 3.3 kernels, 'mnt_parent' has been moved from 'struct vfsmount'"):

<http://sourceware.org/bugzilla/show_bug.cgi?id=13670>

Fixed in commit 48ae989 (which isn't in systemtap 1.7).

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=48ae989e9a6dc69b3a275d8321ef09e9b7434eac>

> FAIL: buildok/ioscheduler-detailed.stp

semantic error: unable to find member 'elevator_name' for struct elevator_queue (alternatives: type elevator_data kobj sysfs_lock hash registered): operator '->' at /usr/share/systemtap/tapset/ioscheduler.stp:28:17
        source: 		semantic error: unable to find member 'elevator_name' for struct elevator_queue (alternatives: type elevator_data kobj sysfs_lock hash registered): operator '->' at /usr/share/systemtap/tapset/ioscheduler.stp:28:17
: $q->elevator->elevator_name)

This is bug PR13672 ("on 3.3 kernels, the ioscheduler.stp tapset can't
find elevator names"):
<http://sourceware.org/bugzilla/show_bug.cgi?id=13672>

Fixed in commit 37b2459 (which isn't in systemtap 1.7).

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=37b2459956942428917a434734d7affdc48219d9>

> FAIL: buildok/pretty.stp

WARNING: cannot find module /usr/bin/stap debuginfo: No DWARF information found
WARNING: cannot find module /usr/bin/stap debuginfo: No DWARF information found
semantic error: no match while resolving probe point process("stap").function("parse_cmdline")

This one should have worked if you had systemtap-debuginfo installed.
Are you sure the systemtap.log was from the 2nd run?

> FAIL: buildok/proc_mem-embedded.stp

/tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c: In function 'function__stp_valid_task':
/tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50: error: 'PF_STARTING' undeclared (first use in this function)
/tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50: note: each undeclared identifier is reported only once for each function it appears in
/tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c: In function 'function__stp_valid_task':

/tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50: error: 'PF_STARTING' undeclared (first use in this function)

/tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50: note: each undeclared identifier is reported only once for each function it appears in

The PF_STARTING flag was removed in kernel 3.3.  This was fixed in systemtap by
commit ca73974d:

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=ca73974d84f94065dbabce2a2044d17a83bade57>

> FAIL: semok/pretty.stp

Same problem as the buildok/pretty.stp failure above - no systemtap
debuginfo found.

> FAIL: semok/pretty2.stp

semantic error: no match while resolving probe point kernel.trace("*")

That would suggest that CONFIG_TRACEPOINTS isn't set in your kernel.
Try the following command:

# grep CONFIG_TRACEPOINTS /boot/config-`uname -r`

If CONFIG_TRACEPOINTS=y, try this:

# stap -l 'kernel.trace("*")'

If the lack of tracepoints is the problem, we could possible KFAIL
this pretty2.stp testcase in semok.exp.

> FAIL: semok/thirtynine.stp

semantic error: not accessible at this address (0xc0000000008005e0, dieoffset: 0x8f9d7e): identifier '$prev' at /usr/share/systemtap/testsuite/semok/thirtynine.stp:6:40
        source:   printf("switch from=0x%x to=0x%x\n", $prev, $next)
semantic error: not accessible at this address (0xc0000000008005e0, dieoffset: 0x8f9d7e): identifier '$prev' at /usr/share/systemtap/testsuite/semok/thirtynine.stp:6:40

This is a problem with inlined-function arguments (PR1155).  I'm
surprised we don't KFAIL this one already.

> FAIL: shared buffer guest
> FAIL: buffer sharing (0, 2)

WARNING: ".stp_print_flush_test1" [/tmp/stap7KDHzz/stap_cfaab46f985a0ad6e760ca58053dbf53_775.ko] undefined!
FAIL: shared buffer guest

This is a testcase problem.  The testcase wasn't expecting the leading
'.' on the 'stp_print_flush_test1' symbol (which only appears on ppc64).

I just fixed this by tweaking the testcase in commit 84cccb3.

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=84cccb3a506fd8cf761aa30ac281ca13675b2b28>

> FAIL: systemtap.stress/current.stp compilation
semantic error: no match while resolving probe point kernel.function("*@kernel/sched.c").call

This is a testcase problem, fixed in commit 199d126.

<http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;h=199d126dcb36439231c218f1bd3e60ce5a2ebcb2>

> FAIL: 64-bit signal nd_syscall
> FAIL: 32-bit signal nd_syscall

I'm unsure of why these are failing after looking at the log output.
I believe it has something to do with sigprocmask() and sigaction()
being unimplemented on ppc64 and the testcase not handling that output
correctly.  I'm surprised the "regular" signal syscall test passed.

> FAIL: 32-bit signalfd syscall

This is another odd one.  This may be an instance of PR11414
("nd_syscall.exp failures")/PR11424 ("dwarfless kprobe.* probes don't
validate at translate time").

<http://sourceware.org/bugzilla/show_bug.cgi?id=11414>
<http://sourceware.org/bugzilla/show_bug.cgi?id=11424>

------- Comment From srinivasa.tn.com 2012-05-13 11:07 EDT-------
(In reply to comment #18)
> (In reply to comment #8)
> > (In reply to comment #16)
> > > (In reply to comment #0)
> > \>
> > > Let's walk through these.  A "regular" rpm can't require debuginfo rpms, so
> > > that takes care of 4 of those additional rpms
> > > (systemtap-debuginfo-1.7-5.fc17.ppc64, glibc-debuginfo-2.15-32.fc17.ppc64,
> > > glibc-debuginfo-common-2.15-32.fc17.ppc64, and
> > > avahi-debuginfo-0.6.30-7.fc17.ppc64).
> > Did not get what you mean by that.  I did an
> > debuginfo-install glibc-2.15-32.fc17.ppc64
> > debuginfo-install systemtap-client-1.7-5.fc17.ppc64
> > systemtap-devel-1.7-5.fc17.ppc64
> > And the above commands intalled the systemtap-grapher and systemtap-initscript.

Then why there is a message like "Missing separate debuginfos, use: debuginfo-install glibc-2.15-32.fc17.ppc64" and "Missing separate debuginfos, use: debuginfo-install systemtap-client-1.7-5.fc17.ppc64 systemtap-devel-1.7-5.fc17.ppc64" in the attached systemtap.log?  (The log file in the attachment is when I do make installcheck before installing the additional/missing rpms).

> I'm sorry I confused you.  I meant that the systemtap-testsuite can't
> require those debuginfo rpms.  So, installing them will be a manual step.
> > The systemtap-grapher rpm has been
> > > deprecated upstream and isn't needed for testing.  The systemtap-initscript
> > > rpm isn't needed for testing.
> > > I'm surprised that the systemtap-testsuite rpm doesn't require the
> > > systemtap-server rpm.  It should.
> >
> > The systemtap-server rpm is there in the above list
> > (systemtap-server-1.7-5.fc17.ppc64)
> I've confused you again.  The systemtap-testsuite rpm should require the
> systemtap-server rpm so that the systemtap-server rpm gets automatically
> installed.
> > > I see that testsuite/systemtap.base/crash.exp runs the 'crash' script from
> > > the crash rpm, so the systemtap-testsuite rpm should also require the crash
> > > rpm.
> > >
> > > Do you remember what test started working once the avahi rpm was installed?
> > > Since the systemtap-server rpm doesn't require that rpm, I'm surprised it
> > > was needed.
> >
> > After installing the avahi daemon, following testcases started working:
> >
> > /usr/share/systemtap/testsuite/systemtap.server/client.exp
> I've learned that parts of the client.exp test case test automatic server
> detection which uses avahi.  The avahi rpm isn't required by the
> systemtap-server rpm, because clients can contact servers directly (without
> using avahi).

Without installing avahi rpm, we were getting "Failed to create Avahi client: Daemon not running" when the cmd "spawn stap --list-servers=online" is executed as part of /usr/share/systemtap/testsuite/systemtap.server/client.exp?

> I've updated the systemtap-testsuite rpm's requires in commit d6d8634.
>
> <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=d6d8634e46c520ecdf5675f0590963497c206d4f>

Thanks.

Regards,
Seenu.

------- Comment From srinivasa.tn.com 2012-05-14 13:24 EDT-------
(In reply to comment #19)
> > FAIL: systemtap.examples/general/badname build
>
> Failed with errors like this:
>
> OUT semantic error: not accessible at this address (0xc00000000024a364,
> dieoffset: 0x124ba95): identifier '$child' at badname.stp:16:7
> source:   if ($child->d_inode || $dir->i_flags & 16) next
>
> It looks like systemtap is having trouble finding $child.  To debug
> this one further, I'll need to see the output of the following
> command:
>
> # stap -L 'kernel.function("may_create@fs/namei.c")'
>

[root@miz04 ~]# stap -L 'kernel.function("may_create@fs/namei.c")'
kernel.function("may_create@fs/namei.c:1953")

> > FAIL: systemtap.examples/network/netdev build
>
> Failed with the following errors;
>
> OUT semantic error: not accessible at this address (0xc0000000006b75d0,
> dieoffset: 0x37f697e): identifier '$dev' at
> /usr/share/systemtap/tapset/networking.stp:159:29
> source: 	dev_name = get_netdev_name($dev)
> ^
> semantic error: not accessible at this address (0xc0000000006b75d0,
> dieoffset: 0x37f6975): identifier '$flags' at :160:10
> source: 	flags = $flags
>
> This one was worked around with upstream commit 021bc9024, which isn't
> in systemtap 1.7.
>
> <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=021bc902453a4b7375ca5b575c1bf1c980d43f81>
>
Applied the patch and it has solved the issue.

> > FAIL: buildok/dentry-embedded.stp
>
> semantic error: unable to find member 'mnt_parent' for struct vfsmount
> (alternatives: mnt_root mnt_sb mnt_flags): operator '->' at
> /usr/share/systemtap/tapset/dentry.stp:104:54
> source:                         if (@cast(vfsmnt, "vfsmount")->mnt_parent ==
> vfsmnt)semantic error: unable to find member 'mnt_parent' for struct
> vfsmount (alternatives: mnt_root mnt_sb mnt_flags): operator '->' at
> /usr/share/systemtap/tapset/dentry.stp:104:54
>
> This is bug PR13670 ("on 3.3 kernels, 'mnt_parent' has been moved from
> 'struct vfsmount'"):
>
> <http://sourceware.org/bugzilla/show_bug.cgi?id=13670>
>
> Fixed in commit 48ae989 (which isn't in systemtap 1.7).
>
> <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=48ae989e9a6dc69b3a275d8321ef09e9b7434eac>
>
Applied the patch but it is still a failure.  Check the attachment.

> > FAIL: buildok/ioscheduler-detailed.stp
>
> semantic error: unable to find member 'elevator_name' for struct
> elevator_queue (alternatives: type elevator_data kobj sysfs_lock hash
> registered): operator '->' at
> /usr/share/systemtap/tapset/ioscheduler.stp:28:17
> source: 		semantic error: unable to find member 'elevator_name' for struct
> elevator_queue (alternatives: type elevator_data kobj sysfs_lock hash
> registered): operator '->' at
> /usr/share/systemtap/tapset/ioscheduler.stp:28:17
> : $q->elevator->elevator_name)
>
> This is bug PR13672 ("on 3.3 kernels, the ioscheduler.stp tapset can't
> find elevator names"):
> <http://sourceware.org/bugzilla/show_bug.cgi?id=13672>
>
> Fixed in commit 37b2459 (which isn't in systemtap 1.7).
>
> <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=37b2459956942428917a434734d7affdc48219d9>
>

Applied the patch and it solves the issue.

> > FAIL: buildok/pretty.stp
>
> WARNING: cannot find module /usr/bin/stap debuginfo: No DWARF information
> found
> WARNING: cannot find module /usr/bin/stap debuginfo: No DWARF information
> found
> semantic error: no match while resolving probe point
> process("stap").function("parse_cmdline")
>
> This one should have worked if you had systemtap-debuginfo installed.
> Are you sure the systemtap.log was from the 2nd run?
>
After installing the debuginfo, we are getting a different error.  Check the attachment.

> > FAIL: buildok/proc_mem-embedded.stp
>
> /tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c: In
> function 'function__stp_valid_task':
> /tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50:
> error: 'PF_STARTING' undeclared (first use in this function)
> /tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50:
> note: each undeclared identifier is reported only once for each function it
> appears in
> /tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c: In
> function 'function__stp_valid_task':
>
> /tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50:
> error: 'PF_STARTING' undeclared (first use in this function)
>
> /tmp/stapUAQ1cG/stap_e270805edb0b0a29087ae0295ac1ca68_12499_src.c:1640:50:
> note: each undeclared identifier is reported only once for each function it
> appears in
>
> The PF_STARTING flag was removed in kernel 3.3.  This was fixed in systemtap
> by
> commit ca73974d:
>
> <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=ca73974d84f94065dbabce2a2044d17a83bade57>
>
Applied the patch and it is working fine.

> > FAIL: semok/pretty.stp
>
> Same problem as the buildok/pretty.stp failure above - no systemtap
> debuginfo found.
>
> > FAIL: semok/pretty2.stp
>
> semantic error: no match while resolving probe point kernel.trace("*")
>
> That would suggest that CONFIG_TRACEPOINTS isn't set in your kernel.
> Try the following command:
>
> # grep CONFIG_TRACEPOINTS /boot/config-`uname -r`
>
> If CONFIG_TRACEPOINTS=y, try this:
>
> # stap -l 'kernel.trace("*")'
>
> If the lack of tracepoints is the problem, we could possible KFAIL
> this pretty2.stp testcase in semok.exp.
>
> > FAIL: semok/thirtynine.stp
>
> semantic error: not accessible at this address (0xc0000000008005e0,
> dieoffset: 0x8f9d7e): identifier '$prev' at
> /usr/share/systemtap/testsuite/semok/thirtynine.stp:6:40
> source:   printf("switch from=0x%x to=0x%x\n", $prev, $next)
> semantic error: not accessible at this address (0xc0000000008005e0,
> dieoffset: 0x8f9d7e): identifier '$prev' at
> /usr/share/systemtap/testsuite/semok/thirtynine.stp:6:40
>
> This is a problem with inlined-function arguments (PR1155).  I'm
> surprised we don't KFAIL this one already.
>
I have done KFAIL for the above three errors.  Check the attached patch for modifications.

> > FAIL: shared buffer guest
> > FAIL: buffer sharing (0, 2)
>
> WARNING: ".stp_print_flush_test1"
> [/tmp/stap7KDHzz/stap_cfaab46f985a0ad6e760ca58053dbf53_775.ko] undefined!
>
> This is a testcase problem.  The testcase wasn't expecting the leading
> '.' on the 'stp_print_flush_test1' symbol (which only appears on ppc64).
>
> I just fixed this by tweaking the testcase in commit 84cccb3.
>
> <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=84cccb3a506fd8cf761aa30ac281ca13675b2b28>
>

Applied the patch and it has solved the problem.

> > FAIL: systemtap.stress/current.stp compilation
> semantic error: no match while resolving probe point
> kernel.function("*@kernel/sched.c").call
>
> This is a testcase problem, fixed in commit 199d126.
>
> <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> h=199d126dcb36439231c218f1bd3e60ce5a2ebcb2>
>
Applied the patch but has resulted in different error.  Check the attachment for log.

> > FAIL: 64-bit signal nd_syscall
> > FAIL: 32-bit signal nd_syscall
>
> I'm unsure of why these are failing after looking at the log output.
> I believe it has something to do with sigprocmask() and sigaction()
> being unimplemented on ppc64 and the testcase not handling that output
> correctly.  I'm surprised the "regular" signal syscall test passed.
>
> > FAIL: 32-bit signalfd syscall
>
> This is another odd one.  This may be an instance of PR11414
> ("nd_syscall.exp failures")/PR11424 ("dwarfless kprobe.* probes don't
> validate at translate time").
>
> <http://sourceware.org/bugzilla/show_bug.cgi?id=11414>
> <http://sourceware.org/bugzilla/show_bug.cgi?id=11424>

What is our further action for the above 3 errors?

There is one more error "crash - crash(8) data" which was reported earlier.  Can you have a look into it?

Regards,
Seenu.

Created attachment 584365 [details]
system.log after applying the commits mentioned by you.


------- Comment (attachment only) From srinivasa.tn.com 2012-05-14 13:27 EDT-------

Created attachment 584366 [details]
Patch for semok.exp in the directory /usr/share/systemtap/testsuite/systemtap.pass1-4


------- Comment (attachment only) From srinivasa.tn.com 2012-05-14 13:32 EDT-------

commit 2d69ca83228e56f192bbd5399d21b20a20498eec

Disables most of the client.exp tests when avahi-daemon is not available or not working.

(In reply to comment #11)
> ------- Comment From srinivasa.tn.com 2012-05-13 11:07 EDT-------
> (In reply to comment #18)
> > (In reply to comment #8)
> > > (In reply to comment #16)
> > > > (In reply to comment #0)
> > > \>
> > > > Let's walk through these.  A "regular" rpm can't require debuginfo rpms, so
> > > > that takes care of 4 of those additional rpms
> > > > (systemtap-debuginfo-1.7-5.fc17.ppc64, glibc-debuginfo-2.15-32.fc17.ppc64,
> > > > glibc-debuginfo-common-2.15-32.fc17.ppc64, and
> > > > avahi-debuginfo-0.6.30-7.fc17.ppc64).
> > > Did not get what you mean by that.  I did an
> > > debuginfo-install glibc-2.15-32.fc17.ppc64
> > > debuginfo-install systemtap-client-1.7-5.fc17.ppc64
> > > systemtap-devel-1.7-5.fc17.ppc64
> > > And the above commands intalled the systemtap-grapher and systemtap-initscript.
> 
> Then why there is a message like "Missing separate debuginfos, use:
> debuginfo-install glibc-2.15-32.fc17.ppc64" and "Missing separate debuginfos,
> use: debuginfo-install systemtap-client-1.7-5.fc17.ppc64
> systemtap-devel-1.7-5.fc17.ppc64" in the attached systemtap.log?  (The log file
> in the attachment is when I do make installcheck before installing the
> additional/missing rpms).

I see I still confused you (or I'm misreading your reply).

Yes, you are very correct, installing those 4 debuginfo rpms should help those tests pass.

However, the systemtap-testsuite rpm can't "Require:" those rpms (which would get them automatically installed), because normal rpms can't require debuginfo rpms.

So, installing those 4 debuginfo rpms will have to remain a manual step performed by the tester.

(In reply to comment #12)
> ------- Comment From srinivasa.tn.com 2012-05-14 13:24 EDT-------
> (In reply to comment #19)
> > > FAIL: systemtap.examples/general/badname build
> >
> > Failed with errors like this:
> >
> > OUT semantic error: not accessible at this address (0xc00000000024a364,
> > dieoffset: 0x124ba95): identifier '$child' at badname.stp:16:7
> > source:   if ($child->d_inode || $dir->i_flags & 16) next
> >
> > It looks like systemtap is having trouble finding $child.  To debug
> > this one further, I'll need to see the output of the following
> > command:
> >
> > # stap -L 'kernel.function("may_create@fs/namei.c")'
> >
> 
> [root@miz04 ~]# stap -L 'kernel.function("may_create@fs/namei.c")'
> kernel.function("may_create@fs/namei.c:1953")

I'd guess that may_create has been inlined and systemtap can't find its arguments.

There probably isn't much we can do here.  If you want to verify that this is the problem, work through the steps listed on this wiki page:

<http://sourceware.org/systemtap/wiki/TipContextVariables>


> > > FAIL: buildok/dentry-embedded.stp
> >
> > semantic error: unable to find member 'mnt_parent' for struct vfsmount
> > (alternatives: mnt_root mnt_sb mnt_flags): operator '->' at
> > /usr/share/systemtap/tapset/dentry.stp:104:54
> > source:                         if (@cast(vfsmnt, "vfsmount")->mnt_parent ==
> > vfsmnt)semantic error: unable to find member 'mnt_parent' for struct
> > vfsmount (alternatives: mnt_root mnt_sb mnt_flags): operator '->' at
> > /usr/share/systemtap/tapset/dentry.stp:104:54
> >
> > This is bug PR13670 ("on 3.3 kernels, 'mnt_parent' has been moved from
> > 'struct vfsmount'"):
> >
> > <http://sourceware.org/bugzilla/show_bug.cgi?id=13670>
> >
> > Fixed in commit 48ae989 (which isn't in systemtap 1.7).
> >
> > <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> > h=48ae989e9a6dc69b3a275d8321ef09e9b7434eac>
> >
> Applied the patch but it is still a failure.  Check the attachment.

Are you sure that patch applied successfully?  This failure isn't a ppc-specific failure and that patch fixed the problem on other platforms.

To make sure we're talking about the right patch, this one had a commit message of:

Fixed PR13670 by getting task_dentry_path() working on 3.3 kernels.

* tapset/dentry.stp (real_mount): New function.
  (task_dentry_path): Uses real_mount() to get the 'struct mount' pointer
  for a 'struct vfsmount' pointer (needed on 3.3 kernels).
* testsuite/buildok/dentry-embedded.stp: Added real_mount() test.


> > > FAIL: buildok/pretty.stp
> >
> > WARNING: cannot find module /usr/bin/stap debuginfo: No DWARF information
> > found
> > WARNING: cannot find module /usr/bin/stap debuginfo: No DWARF information
> > found
> > semantic error: no match while resolving probe point
> > process("stap").function("parse_cmdline")
> >
> > This one should have worked if you had systemtap-debuginfo installed.
> > Are you sure the systemtap.log was from the 2nd run?
> >
> After installing the debuginfo, we are getting a different error.  Check the
> attachment.

The new error is:

semantic error: no match while resolving probe point kernel.trace("sched_switch")

This is an odd error.  Here's the code:

===============
# pretty-printing in dwarf kernel context
probe kernel.function("schedule_tail")
    %( CONFIG_TRACEPOINTS == "y" %?
    # pretty-printing in tracepoints
    , kernel.trace("sched_switch")
    %)
{
    log($prev->fs$)
    log($prev->fs$$)
    log($prev->comm$)
    log($prev->comm$$)
    log($prev->comm[0]$)
    log($prev->comm[0]$$)
    log($prev->comm[i]$)
    log($prev->comm[i]$$)
}
===============

I thought from our earlier conversation that your kernel had CONFIG_TRACEPOINTS=n, so systemtap shouldn't have tried to look for 'kernel.trace("sched_switch")'.

Looking at your patch, CONFIG_TRACEPOINTS=y, but there are no tracepoints?  How does that happen?

> > > FAIL: semok/pretty.stp
> >
> > Same problem as the buildok/pretty.stp failure above - no systemtap
> > debuginfo found.
> >
> > > FAIL: semok/pretty2.stp
> >
> > semantic error: no match while resolving probe point kernel.trace("*")
> >
> > That would suggest that CONFIG_TRACEPOINTS isn't set in your kernel.
> > Try the following command:
> >
> > # grep CONFIG_TRACEPOINTS /boot/config-`uname -r`
> >
> > If CONFIG_TRACEPOINTS=y, try this:
> >
> > # stap -l 'kernel.trace("*")'
> >
> > If the lack of tracepoints is the problem, we could possible KFAIL
> > this pretty2.stp testcase in semok.exp.
> >
> > > FAIL: semok/thirtynine.stp
> >
> > semantic error: not accessible at this address (0xc0000000008005e0,
> > dieoffset: 0x8f9d7e): identifier '$prev' at
> > /usr/share/systemtap/testsuite/semok/thirtynine.stp:6:40
> > source:   printf("switch from=0x%x to=0x%x\n", $prev, $next)
> > semantic error: not accessible at this address (0xc0000000008005e0,
> > dieoffset: 0x8f9d7e): identifier '$prev' at
> > /usr/share/systemtap/testsuite/semok/thirtynine.stp:6:40
> >
> > This is a problem with inlined-function arguments (PR1155).  I'm
> > surprised we don't KFAIL this one already.
> >
> I have done KFAIL for the above three errors.  Check the attached patch for
> modifications.


Your patch is a bit too broad.  Once I get an understanding of how tracepoints work on your system I might be able to suggest a better patch.


> > > FAIL: systemtap.stress/current.stp compilation
> > semantic error: no match while resolving probe point
> > kernel.function("*@kernel/sched.c").call
> >
> > This is a testcase problem, fixed in commit 199d126.
> >
> > <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> > h=199d126dcb36439231c218f1bd3e60ce5a2ebcb2>
> >
> Applied the patch but has resulted in different error.  Check the attachment
> for log.

There are several warnings of the form:

WARNING: probe kernel.function("generic_calibrate_decr@arch/powerpc/kernel/time.c:710").call (address 0xc000000000b247f0) registration error (rc -22)

Error 22 is EINVAL.  The only thing those functions have in common that I can see is that they are all marked as '__init', but systemtap doesn't have problems probing '__init' functions on other platforms.

It is possible that these warnings are throwing off the testcase.  But, I don't see the expected output after the warnings.

It is also possible that we aren't waiting long enough on ppc for the output to appear. Could you modify current.exp and change the 'after 10000' line to 'after 20000' and rerun the test?

To run this individual test, do:

# make installcheck RUNTESTFLAGS=current.exp

> > > FAIL: 64-bit signal nd_syscall
> > > FAIL: 32-bit signal nd_syscall
> >
> > I'm unsure of why these are failing after looking at the log output.
> > I believe it has something to do with sigprocmask() and sigaction()
> > being unimplemented on ppc64 and the testcase not handling that output
> > correctly.  I'm surprised the "regular" signal syscall test passed.
> >
> > > FAIL: 32-bit signalfd syscall
> >
> > This is another odd one.  This may be an instance of PR11414
> > ("nd_syscall.exp failures")/PR11424 ("dwarfless kprobe.* probes don't
> > validate at translate time").
> >
> > <http://sourceware.org/bugzilla/show_bug.cgi?id=11414>
> > <http://sourceware.org/bugzilla/show_bug.cgi?id=11424>
> 
> What is our further action for the above 3 errors?

Until someone decides to work on those bugs above and finds the time, there isn't much we can do.  If you'd like to help here we'd be grateful.

> There is one more error "crash - crash(8) data" which was reported earlier. 
> Can you have a look into it?

I looked at it, but I don't really understand what is going on with it.  I may have time to look at it in further detail later in the week.

Note that in general these test results look quite good - perhaps even better than x86_64's results.

------- Comment From srinivasa.tn.com 2012-05-15 07:08 EDT-------
(In reply to comment #24)
> commit 2d69ca83228e56f192bbd5399d21b20a20498eec
> Disables most of the client.exp tests when avahi-daemon is not available or
> not working.

Thanks.

Regards,
Seenu.

------- Comment From srinivasa.tn.com 2012-05-15 07:09 EDT-------
(In reply to comment #25)
> (In reply to comment #11)
> > (In reply to comment #18)
> > > (In reply to comment #8)
> > > > (In reply to comment #16)
> > > > > (In reply to comment #0)

> I see I still confused you (or I'm misreading your reply).
> Yes, you are very correct, installing those 4 debuginfo rpms should help
> those tests pass.
> However, the systemtap-testsuite rpm can't "Require:" those rpms (which
> would get them automatically installed), because normal rpms can't require
> debuginfo rpms.
>
> So, installing those 4 debuginfo rpms will have to remain a manual step
> performed by the tester.

I completely agree with you that they have to be manually installed.

Regards,
Seenu.

------- Comment From srinivasa.tn.com 2012-05-15 07:39 EDT-------
(In reply to comment #26)
> > > > FAIL: buildok/dentry-embedded.stp
> > > <http://sourceware.org/bugzilla/show_bug.cgi?id=13670>
> > >
> > > Fixed in commit 48ae989 (which isn't in systemtap 1.7).
> > >
> > > <http://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commitdiff;
> > > h=48ae989e9a6dc69b3a275d8321ef09e9b7434eac>
> > >
> > Applied the patch but it is still a failure.  Check the attachment.
>
> Are you sure that patch applied successfully?  This failure isn't a

Sorry, there was a mistake from my side in applying the patch.  After applying the patch correctly, the test is passing.

> > > > FAIL: systemtap.stress/current.stp compilation

> > Applied the patch but has resulted in different error.  Check the attachment
> > for log.
>
> There are several warnings of the form:
>
> WARNING: probe
> kernel.function("generic_calibrate_decr@arch/powerpc/kernel/time.c:710").
> call (address 0xc000000000b247f0) registration error (rc -22)
>
> Error 22 is EINVAL.  The only thing those functions have in common that I
> can see is that they are all marked as '__init', but systemtap doesn't have
> problems probing '__init' functions on other platforms.
>
> It is possible that these warnings are throwing off the testcase.  But, I
> don't see the expected output after the warnings.
>
> It is also possible that we aren't waiting long enough on ppc for the output
> to appear. Could you modify current.exp and change the 'after 10000' line to
> 'after 20000' and rerun the test?

Even after changing the value from 10000 to 20000 did not solve this problem and we are getting the same error.

> Note that in general these test results look quite good - perhaps even
> better than x86_64's results.

I also feel the same.  When can we have the version of systemtap from RH (fedora) with all these commits?

Regards,
Seenu.

------- Comment From srinivasa.tn.com 2012-05-28 05:35 EDT-------
Hi RH,
Do we have any updates?

Regards,
Seenu.

(In reply to comment #20)
> ------- Comment From srinivasa.tn.com 2012-05-28 05:35 EDT-------
> Do we have any updates?

If you want to see a Systemtap 1.8 release sooner, please help coordinate on the upstream mailing list: http://sourceware.org/ml/systemtap/

systemtap-1.8-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/systemtap-1.8-1.fc17

Package systemtap-1.8-1.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemtap-1.8-1.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9620/systemtap-1.8-1.fc17
then log in and leave karma (feedback).

systemtap-1.8-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

------- Comment From onmahaja.com 2013-01-21 07:09 EDT-------
Red Hat,
This bug is confirmed to be resolved from our side. Marking as CLOSED