This looks like a casualty of kaslr being enabled by default.

If I boot with 'nokaslr' the backtraces are ok again:

[root@rhel7u5-node2 ~]# !186
stap -v --all-modules -e 'probe kernel.function("kmem_cache_alloc") { print_backtrace(); exit() }'
Pass 1: parsed user script and 472 library scripts using 240076virt/42000res/3456shr/38624data kb, in 750usr/70sys/911real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 281224virt/84040res/4296shr/79772data kb, in 890usr/540sys/2163real ms.
Pass 3: translated to C into "/tmp/stappqNV2S/stap_43e81a9b193918462a43c57525217ba7_3325_src.c" using 294372virt/91084res/4720shr/92920data kb, in 4520usr/490sys/5158real ms.
Pass 4: compiled C into "stap_43e81a9b193918462a43c57525217ba7_3325.ko" in 13940usr/1940sys/16000real ms.
Pass 5: starting run.
WARNING: DWARF expression stack underflow in CFI
 0xffffffff811f7240 : kmem_cache_alloc+0x0/0x1f0 [kernel]
 0xffffffff815da62a : __build_skb+0x2a/0x1b0 [kernel]
 0xffffffff815da919 : __netdev_alloc_skb+0xc9/0x110 [kernel]
 0xffffffffc00e624e : page_to_skb+0x4e/0x1f0 [virtio_net]
 0xffffffffc00e8529 : virtnet_poll+0x2c9/0x750 [virtio_net]
 0xffffffff815ec3af : net_rx_action+0x26f/0x390 [kernel]
 0xffffffff8109a945 : __do_softirq+0xf5/0x280 [kernel]
 0xffffffff81722d2c : call_softirq+0x1c/0x30 [kernel]
 0xffffffff8102d625 : do_softirq+0x65/0xa0 [kernel]
 0xffffffff8109acc5 : irq_exit+0x105/0x110 [kernel]
 0xffffffff81723fc6 : __irqentry_text_start+0x56/0xf0 [kernel]
 0xffffffff81716362 : ret_from_intr+0x0/0x15 [kernel]
 0x0 (inexact)
Pass 5: run completed in 20usr/140sys/766real ms.
[root@rhel7u5-node2 ~]# stap -v --all-modules -e 'probe kernel.function("kmem_cache_alloc") { print_backtrace(); exit() }'
Pass 1: parsed user script and 472 library scripts using 240072virt/42000res/3456shr/38620data kb, in 730usr/50sys/781real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 281220virt/84036res/4296shr/79768data kb, in 1220usr/340sys/1562real ms.
Pass 3: using cached /root/.systemtap/cache/43/stap_43e81a9b193918462a43c57525217ba7_3325.c
Pass 4: using cached /root/.systemtap/cache/43/stap_43e81a9b193918462a43c57525217ba7_3325.ko
Pass 5: starting run.
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry with -DDEBUG_UNWIND)'
 0xffffffff811f7240 : kmem_cache_alloc+0x0/0x1f0 [kernel]
 0xffffffff810c19a6 : prepare_creds+0x26/0x1c0 [kernel]
 0xffffffff81218f9c : SyS_faccessat+0x4c/0x230 [kernel]
 0xffffffff81219198 : SyS_access+0x18/0x20 [kernel]
 0xffffffff8171f7d5 : system_call_fastpath+0x1c/0x21 [kernel]
 0x7fd357bd6cb7
 0x0 (inexact)
Pass 5: run completed in 20usr/100sys/398real ms.
[root@rhel7u5-node2 ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-862.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS0,115200 nokaslr

(In reply to Dave Wysochanski from comment #2)
> This looks like a casualty of kaslr being enabled by default.

Thanks for that quick diagnosis.

For reference this refers to "KASLR fully supported and enabled by default":
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/new_features_kernel#BZ1491226

There are two bugs here. First when using the stap DWARF unwinder looking up the kernel address in the unwind table fails. The problem seems to be that the unwind table itself is constructed "offline" from the addresses on disk, but then at runtime there is no correction for the kernel load address (this might have to be adjusted in a couple more places). Secondly when falling back on the builtin kernel unwinder stap fails because it is looking for dump_trace, but that has been replaced by the new "unwind" one in the new kernel.

As a start of a workaround/fix I added a new kernel fallback unwinder:
https://sourceware.org/git/?p=systemtap.git;a=commit;h=553b6df07c9b7ab30ed468a6a4374cbdf73d1c0d

This patch has been added to the fedora systemtap build for f28:
http://koji.fedoraproject.org/koji/buildinfo?buildID=1072496

There is also a copr build for testing on f27, centos/rhel7 x86_64:
https://copr.fedorainfracloud.org/coprs/mjw/systemtap-fallback-unwind/

With EPEL yum-plugin-copr it can be enabled and installed with:
$ yum install yum-plugin-copr
$ yum copr enable mjw/systemtap-fallback-unwind
$ yum upgrade systemtap

The result is of course totally unsupported, untested and might totally break your system. But feedback on how well it works would be appreciated.

The above doesn't fix the stap DWARF unwinder, but the fallback looks pretty good.

I have another fix, this time for the kernel DWARF unwinder:
https://sourceware.org/git/?p=systemtap.git;a=commit;h=17ee540dd61113fe4f557f191db3480db875cca1

Both fixes together should resolve this issue.

I integrated it into the fedora 28 package and the copr package
(including the bug fix for Bug 1566422) for testing
(see comment #5 for install instructions)

Thanks for the quick fix!  Just testing updated systemtap looks good:

Original:
[root@rhel7u5-node1 ~]# rpm -qa | grep systemtap
systemtap-devel-3.2-4.el7.x86_64
systemtap-runtime-3.2-4.el7.x86_64
systemtap-3.2-4.el7.x86_64
systemtap-client-3.2-4.el7.x86_64
[root@rhel7u5-node1 ~]# stap -v -e 'probe kernel.function("kmem_cache_alloc") { print_backtrace(); exit() }' 
Pass 1: parsed user script and 472 library scripts using 240076virt/41992res/3452shr/38624data kb, in 710usr/50sys/763real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 281216virt/84036res/4296shr/79764data kb, in 1240usr/270sys/1512real ms.
Pass 3: using cached /root/.systemtap/cache/b9/stap_b95ff1718ccfdfde276cd3c92c4bb869_1292.c
Pass 4: using cached /root/.systemtap/cache/b9/stap_b95ff1718ccfdfde276cd3c92c4bb869_1292.ko
Pass 5: starting run.
 0xffffffffa4df7240 : kmem_cache_alloc+0x0/0x1f0 [kernel]
 0x0 (inexact)
Pass 5: run completed in 10usr/70sys/365real ms.


Fixed:
[root@rhel7u5-node1 ~]# stap -v -e 'probe kernel.function("kmem_cache_alloc") { print_backtrace(); exit() }' 
Pass 1: parsed user script and 472 library scripts using 240072virt/41992res/3452shr/38620data kb, in 720usr/60sys/775real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 281212virt/84040res/4300shr/79760data kb, in 1280usr/250sys/1529real ms.
Pass 3: translated to C into "/tmp/stapV4eb1t/stap_0eafdab18e2cf4df683a97ec5f0e4f60_1292_src.c" using 281212virt/84368res/4628shr/79760data kb, in 750usr/170sys/921real ms.
Pass 4: compiled C into "stap_0eafdab18e2cf4df683a97ec5f0e4f60_1292.ko" in 26700usr/4000sys/30861real ms.
Pass 5: starting run.
WARNING: Missing unwind data for a module, rerun with 'stap -d (unknown; retry with -DDEBUG_UNWIND)'
 0xffffffffa4df7240 : kmem_cache_alloc+0x0/0x1f0 [kernel]
 0xffffffffa4cc19a6 : prepare_creds+0x26/0x1c0 [kernel]
 0xffffffffa4e18f9c : SyS_faccessat+0x4c/0x230 [kernel]
 0xffffffffa4e19198 : SyS_access+0x18/0x20 [kernel]
 0xffffffffa531f7d5 : system_call_fastpath+0x1c/0x21 [kernel]
 0x7f96efdf6cb7
Pass 5: run completed in 30usr/70sys/363real ms.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3168