Created attachment 1343237 [details]
Description of problem:
Running an example script netttop.stp from
/usr/share/systemtap/examples/network/nettop.stp does not collect any RECV_PK or
# stap /usr/share/systemtap/examples/network/nettop.stp
PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
2058 0 eth0 20026 0 1419 0 sshd
0 0 eth0 4068 0 263 0 swapper/0
2061 0 eth0 234 0 15 0 scp
Version-Release number of selected component (if applicable):
This is happening since: kernel-3.10.0-514.el7
Actually looking into code it seems to be this commit:
RH commit: 988b8276bc4cf6cd653a9633ecdc4514e2ab7f44
There is a change in the netif_receive_skb() which is now calling netif_receive_skb_internal()
The systemtap script uses probe 'netdev.receive' which is defined as:
probe netdev.receive =
I was running live crash on the latest kernel:
crash> sys | grep REL
crash> dis netif_receive_skb | head -5
0xffffffff81587290 <netif_receive_skb>: callq 0xffffffff816b6df0 <ftrace_regs_caller>
0xffffffff81587295 <netif_receive_skb+5>: push %rbp
0xffffffff81587296 <netif_receive_skb+6>: mov %rsp,%rbp
0xffffffff81587299 <netif_receive_skb+9>: push %r12
0xffffffff8158729b <netif_receive_skb+11>: mov %rdi,%r12
Despite of I can see the stap module is bind there, I am getting no output.
The stap script is really easy one and is attached as:
Using the slightly modified version of it, hooked to the netif_receive_skb_internal
works as charm. Stap attached as well as netif_receive_skb_internal.stp.
just running stap /usr/share/systemtap/tapset/linux/networking.stp on one of the
no data collected on RECV_PKT or RECV_KB
data collected on RECV_PKT or RECV_KB
Attaching patch which will actually fix the issue from systemtap point of view.
But I am more interested why the probe does not work as I can see it is hooked in the disassembly.
Created attachment 1343238 [details]
Created attachment 1343239 [details]
systemtap fix for the issue
(In reply to Lukas Herbolt from comment #0)
> Created attachment 1343237 [details]
> Description of problem:
> Running an example script netttop.stp from
> /usr/share/systemtap/examples/network/nettop.stp does not collect any
> RECV_PK or
> # stap /usr/share/systemtap/examples/network/nettop.stp
> PID UID DEV XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
> 2058 0 eth0 20026 0 1419 0 sshd
> 0 0 eth0 4068 0 263 0 swapper/0
> 2061 0 eth0 234 0 15 0 scp
> Version-Release number of selected component (if applicable):
> This is happening since: kernel-3.10.0-514.el7
> Actually looking into code it seems to be this commit:
> RH commit: 988b8276bc4cf6cd653a9633ecdc4514e2ab7f44
> Upstream: ae78dbfa40c629f79c72ab93525508ef49e798b6
> There is a change in the netif_receive_skb() which is now calling
Excellent debugging work here. Those commits do appear to be the problem.
... stuff deleted ...
> Additional info:
> Attaching patch which will actually fix the issue from systemtap point of
> But I am more interested why the probe does not work as I can see it is
> hooked in the disassembly.
The netdev.receive probe hooks netif_receive_skb(). Before those commits, all the traffic received by the kernel went through netif_receive_skb(). After those commits you referenced, that is no longer the case. So, even though that probe is placed at that function, the function is no longer called as much (probably only by drivers in a loadable module).
I checked in your patch upstream as commit 94b3978aa:
This patch will need to be backported.
*** Bug 1518462 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.