Bug 1634684 - bcc usdt probes failing
Summary: bcc usdt probes failing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: bcc
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jiri Olsa
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-10-01 11:17 UTC by Marko Myllynen
Modified: 2018-11-10 03:29 UTC (History)
3 users (show)

Fixed In Version: bcc-0.7.0-4.fc29 bcc-0.7.0-4.fc28
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-10 03:18:32 UTC
Type: Bug


Attachments (Terms of Use)

Description Marko Myllynen 2018-10-01 11:17:45 UTC
Description of problem:
This simple test case works on Fedora 28 using bcc-0.6.1 from update and bcc-0.7.0 from updates-testing:

# cat test.py
#!/usr/bin/python3

import time

def foo():
  print("foo")

while True:
  foo()
  time.sleep(1)
# ./test.py > /dev/null 2>&1 &
# /usr/share/bcc/tools/trace -p $! 'u:/usr/lib64/libpython3.6m.so.1.0:function__entry'
PID     TID     COMM            FUNC             
6334    6334    test.py         function__entry  
6334    6334    test.py         function__entry  
6334    6334    test.py         function__entry  
...

On Fedora 29 with bcc-0.7.0-1.fc29.x86_64 the above doesn't produce any output.

Comment 1 Rafael Fonseca 2018-10-01 14:58:46 UTC
Is it exclusive to trace or does the same happen to other tools too?

Comment 2 Rafael Fonseca 2018-10-01 17:01:09 UTC
I wonder if python3.7 has anything with it. Apart from a small difference in kernel version, the only discrepancy I can see is python3.6 in f28 vs python3.7 in f29.

Comment 3 Marko Myllynen 2018-10-02 06:04:57 UTC
I tested a bit more, this is not limited to trace and I also put SELinux in Permissive just to make sure it is not causing this.

On Fedora 28 with pmcd running this will cause two lines to be printed by trace:

# /usr/share/bcc/tools/trace -p $(pidof pmcd) 'u:/usr/libexec/pcp/bin/pmcd:PMCD' &
# pminfo -f | grep -c value

On Fedora 29 nothing is printed although the pmcd binary contains the same probes:

# /usr/share/bcc/tools/tplist -l /usr/libexec/pcp/bin/pmcd
b'/usr/libexec/pcp/bin/pmcd' b'pcp_probe':b'PMCD'
b'/usr/libexec/pcp/bin/pmcd' b'pcp_probe':b'PMCD_PDU'

So reporting probe hits from non-Python programs (C in case of pmcd) fails also.

This is also causing PCP BCC plugin test cases testing USDT related functionality to fail (the modules being tested are implemented in Python but do not use trace):

# dnf install pcp-testsuite
# cd /var/lib/pcp/testsuite
# ./check 1150 1173

Thanks.

Comment 4 Rafael Fonseca 2018-10-02 08:17:02 UTC
I tried using systemtap[1] in an f29 box and I'm getting:

$ sudo stap show-call-hierarchy.stp -c "python3 test.py" -v
Pass 1: parsed user script and 475 library scripts using 77836virt/57472res/9304shr/48020data kb, in 90usr/30sys/137real ms.
Pass 2: analyzed script: 1 probe, 10 functions, 4 embeds, 2 globals using 249032virt/230248res/10740shr/219216data kb, in 720usr/350sys/1391real ms.
Pass 3: translated to C into "/tmp/stapLT7aot/stap_a89be0d493b99e63d473a48f8c5c5978_5208_src.c" using 249032virt/230448res/10940shr/219216data kb, in 20usr/90sys/99real ms.
Pass 4: compiled C into "stap_a89be0d493b99e63d473a48f8c5c5978_5208.ko" in 2030usr/310sys/2451real ms.
Pass 5: starting run.
ERROR: Build-id mismatch [man error::buildid]: "/usr/lib64/libpython3.7m.so.1.0" byte 0 (0x81 vs 0x00) address 0x7f267e0d3248 rc 0
WARNING: task_finder mmap inode-uprobes callback for task 3736 failed: 1
foo
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run completed in 10usr/20sys/306real ms.
Pass 5: run failed.  [man error::pass5]

The same command works in f28 with python3.6.


[1] https://docs.python.org/3/howto/instrumentation.html

Comment 5 Rafael Fonseca 2018-10-02 08:23:52 UTC
Nevermind. It works when I use -DSTP_NO_BUILDID_CHECK

Comment 6 Marko Myllynen 2018-10-09 15:10:50 UTC
I haven't checked but I suspect this might well be related:

https://github.com/iovisor/bcc/issues/1998

Comment 7 Marko Myllynen 2018-10-09 15:12:39 UTC
(In reply to Marko Myllynen from comment #6)
> I haven't checked but I suspect this might well be related:
> 
> https://github.com/iovisor/bcc/issues/1998

IOW, do we know whether the default Fedora build options for packages have changed in a way contributing to this issue?

Comment 8 Rafael Fonseca 2018-10-09 15:33:01 UTC
Good question. Yes, there was a change https://fedoraproject.org/wiki/Changes/HardeningFlags28

Comment 9 Marko Myllynen 2018-10-10 09:04:22 UTC
The patch proposal at https://github.com/iovisor/bcc/issues/1998#issuecomment-428416633 did not solve this issue.

Comment 10 Marko Myllynen 2018-10-31 15:40:16 UTC
There's now a patch in upstream that solves this, we will just need to backport the fix (or wait for the next release where it will be included):

https://github.com/iovisor/bcc/issues/1998#issuecomment-434692114

Comment 11 Rafael Fonseca 2018-10-31 16:19:12 UTC
I'll backport the patch as soon as a dev reviews and merges it.

Comment 12 Fedora Update System 2018-11-01 10:49:18 UTC
bcc-0.7.0-4.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-117a2e19c0

Comment 13 Fedora Update System 2018-11-01 11:23:30 UTC
bcc-0.7.0-4.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-967cbccab6

Comment 14 Fedora Update System 2018-11-01 15:41:09 UTC
bcc-0.7.0-4.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-117a2e19c0

Comment 15 Fedora Update System 2018-11-01 20:16:50 UTC
bcc-0.7.0-4.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-967cbccab6

Comment 16 Fedora Update System 2018-11-10 03:18:32 UTC
bcc-0.7.0-4.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2018-11-10 03:29:19 UTC
bcc-0.7.0-4.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.