Bug 685064

Summary: Running systemtap in parallel can result in a collision
Product: Red Hat Enterprise Linux 6 Reporter: Petr Šplíchal <psplicha>
Component: systemtapAssignee: Frank Ch. Eigler <fche>
Status: CLOSED ERRATA QA Contact: qe-baseos-tools-bugs
Severity: low Docs Contact:
Priority: low    
Version: 6.1CC: mjw, ohudlick, pmuller, wcohen
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemtap-1.6-1.el6 Doc Type: Bug Fix
Doc Text:
Cause The uprobe.ko kernel module is needed when probing a user-space module. This uprobes.ko kernel module is built if none currently exists. This can lead to a race condition when multiple scripts probing user-space applications note a uprobes.ko kernel module needs to be built Consequence The multiple parallel builds of uprobes.ko can interfere with each other and cause the scripts to fail to build and run. Fix The stap command now caches the resulting uprobe kernel module like other systemtap instrumentation to avoid interferring with systemtap instrumentation instances. Result Multiple systemtap scripts probing userspace applications will not interfere with each other regardless whether a uprobe.ko kernel module exists.
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 15:17:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 683483    
Bug Blocks:    

Description Petr Šplíchal 2011-03-15 08:04:37 UTC
Description of problem:

If two systemtap scripts are run simultaneously on a fresh system
they can clash when trying to rebuild uprobes.ko.

Version-Release number of selected component (if applicable):
systemtap-1.2-9.el6.x86_64

Steps to Reproduce:
stap script1.stp & stap script2.stp
  
Actual results:
The failure can look like this:

> Pass 1: parsed user script and 69 library script(s) using 83956virt/20868res/2424shr kb, in 130usr/10sys/142real ms.
> Pass 1: parsed user script and 69 library script(s) using 83960virt/20880res/2424shr kb, in 140usr/20sys/157real ms.
> Pass 2: analyzed script: 2 probe(s), 13 function(s), 4 embed(s), 2 global(s) using 93988virt/25276res/4248shr kb, in 710usr/1090sys/5767real ms.
> Pass 2: analyzed script: 2 probe(s), 6 function(s), 3 embed(s), 1 global(s) using 94092virt/25260res/4248shr kb, in 680usr/1090sys/5694real ms.
> Pass 3: translated to C into "/tmp/stapGoohFi/stap_45c4d31c9a6fd424692598f20d205124_5605.c" using 93820virt/25404res/4352shr kb, in 40usr/170sys/235real ms.
> Pass 4, preamble: (re)building SystemTap's version of uprobes.
> Pass 3: translated to C into "/tmp/stapDuYGzm/stap_c2ecdfd215a17ee48c16da7c15076a09_3797.c" using 93804virt/25388res/4356shr kb, in 60usr/170sys/224real ms.
> Pass 4, preamble: (re)building SystemTap's version of uprobes.
> mv: cannot stat `/usr/share/systemtap/runtime/uprobes/.tmp_mx_uprobes.o': No such file or directory
> rm: cannot remove `/usr/share/systemtap/runtime/uprobes/.tmp_mc_uprobes.o': No such file or directory
> rm: cannot remove `/usr/share/systemtap/runtime/uprobes/.tmp_mc_uprobes.s': No such file or directory
> mv: cannot stat `/usr/share/systemtap/runtime/uprobes/.uprobes.o.tmp': No such file or directory
> make[3]: *** [/usr/share/systemtap/runtime/uprobes/uprobes.o] Error 1
> make[2]: *** [_module_/usr/share/systemtap/runtime/uprobes] Error 2
> make[1]: *** [default] Error 2
> Pass 4: compiled C into "stap_45c4d31c9a6fd424692598f20d205124_5605.ko" in 1400usr/510sys/2254real ms.
> Pass 4: compilation failed.  Try again with another '--vp 0001' option.
> make[2]: *** Deleting file `/tmp/stapDuYGzm/stapconf_665804123654012ea35c70fb0980ad69_518.h'
> make[2]: *** [/tmp/stapDuYGzm/stapconf_665804123654012ea35c70fb0980ad69_518.h] Terminated
> make[1]: *** [_module_/tmp/stapDuYGzm] Terminated
> Pass 4: compiled C into "stap_c2ecdfd215a17ee48c16da7c15076a09_3797.ko" in 2590usr/1140sys/4196real ms.
> Pass 4: compilation failed.  Try again with another '--vp 0001' option.

Expected results:
No collision.

Comment 1 Petr Muller 2011-03-15 08:57:35 UTC
If this is not a regression, I would suggest postponing to 6.2. This has a trivial workaround.

Comment 2 RHEL Program Management 2011-04-04 01:54:38 UTC
Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 3 Petr Muller 2011-04-04 12:24:50 UTC
Proposing to 6.2

Comment 4 Frank Ch. Eigler 2011-06-09 21:02:20 UTC
We don't have a patch for this yet.

Comment 5 Frank Ch. Eigler 2011-06-10 21:08:39 UTC
We're trying to address this and related problems under this upstream bug:
http://sourceware.org/bugzilla/show_bug.cgi?id=5163

Comment 6 Frank Ch. Eigler 2011-07-12 21:11:15 UTC
Backporting upstream commit #7d26ee02 may be possible to systemtap-1.4,
or preferably to a later rebase.

Comment 10 William Cohen 2011-11-09 16:52:16 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
    The uprobe.ko kernel module is needed when probing a user-space module.
    This uprobes.ko kernel module is built if none currently exists. This can
    lead to a race condition when multiple scripts probing user-space
    applications note a uprobes.ko kernel module needs to be built
Consequence
    The multiple parallel builds of uprobes.ko can interfere with each other
    and cause the scripts to fail to build and run.
Fix
    The stap command now caches the resulting uprobe kernel module
    like other systemtap instrumentation to avoid interferring with
    systemtap instrumentation instances.
Result
    Multiple systemtap scripts probing userspace applications will not
    interfere with each other regardless whether a uprobe.ko kernel
    module exists.

Comment 11 errata-xmlrpc 2011-12-06 15:17:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1517.html