Bug 2029758

There looks to be two issues involved with this:
- The liveness analysis code isn't proper determining it can't read the compress binary
- When the vmlinux is installed and the binary is read the liveness analysis ends up taking quite a bit of memory to run and runs out of memory

There is an upstream dyninst question discussing how to resolve these issues:
https://github.com/dyninst/dyninst/issues/1168

There appears to be 5 sub-issues:
-Correctly punt analysis if can't find/read binary
-When debuginfod running get elf_path to binary as
 /boot/vmlinuz-4.18.0-353.el8.x86_64,
 but using locally installed kernel-debuginfo get
 /usr/lib/debug/lib/modules/4.18.0-353.el8.x86_64/vmlinux
-Handle compressed kernels
-Handle compressed modules
-Memory efficiency
  In this particular bug the dyninst parser is trying to parse the entire vmlinux
  rather than just on demand.  There has been some discussion in the upstream dyninst issue
  about this, https://github.com/dyninst/dyninst/issues/1168.  However, the dyninst
  on-demand parsing requires a recompile of dyninst with another option set to
  replace the existing parsing and these has not been exercised in a while.  Tried some
  experiments on a qemu VM and reproducer was able to successfully complete with
  8GB of memory (using about 70% of available memory).

> -When debuginfod running get elf_path to binary as
>  /boot/vmlinuz-4.18.0-353.el8.x86_64,
>  but using locally installed kernel-debuginfo get
>  /usr/lib/debug/lib/modules/4.18.0-353.el8.x86_64/vmlinux

Why is this a problem?

> -Memory efficiency
>  [...]

Yes, it'd be nice to get dyninst more efficient here.

Let's add this simple item to the todo list:

- a bad_alloc() exception should be caught by systemtap around the dyninst call(s),
  so we have a chance at releasing dyninst data, printing a diagnostic, and
  continuing work

Have a proposed patch on https://sourceware.org/git/?p=systemtap.git;a=shortlog;h=refs/heads/wcohen/liveness_mem_use to address this issue.

With the code from the branch compiled rather than getting the exception get the following with the reproducer:

$ ../install/bin/stap -p2 -v -g fork.stp 
Pass 1: parsed user script and 497 library scripts using 351948virt/152412res/17948shr/136876data kb, in 290usr/40sys/391real ms.
WARNING: unable to allocate memory for liveness of /usr/lib/debug/lib/modules/4.18.0-359.el8.x86_64/vmlinux: identifier '$return' at fork.stp:26:13
 source:             $return = -1
                     ^
# globals
sshd_fork_count:long
fork_to_fail:long = 1
# functions
execname:string ()
{ %{ /* pure */ /* unprivileged */ /* stable */
	strlcpy (STAP_RETVALUE, current->comm, MAXSTRINGLEN);
%} }
pid:long ()
{ return %{ /* pure */ /* unprivileged */ /* stable */ current->tgid %} }
ppid:long ()
{ %{ /* pure */ /* unprivileged */ /* stable */
#if defined(STAPCONF_REAL_PARENT)
	STAP_RETVALUE = current->real_parent->tgid;
#else
	STAP_RETVALUE = current->parent->tgid;
#endif
%} }
__private_fork_stp__dwarf_tvar_get_child_tidptr_4:long ()
{ return @kregister(8) }
__private_fork_stp__dwarf_tvar_get_clone_flags_0:long ()
{ return @kregister(5) }
__private_fork_stp__dwarf_tvar_get_parent_tidptr_3:long ()
{ return @kregister(2) }
__private_fork_stp__dwarf_tvar_get_return_6:long ()
{ return @kregister(0) }
__private_fork_stp__dwarf_tvar_get_stack_size_2:long ()
{ return @kregister(1) }
__private_fork_stp__dwarf_tvar_get_stack_start_1:long ()
{ return @kregister(4) }
__private_fork_stp__dwarf_tvar_get_tls_5:long ()
{ return @kregister(9) }
__private_fork_stp__dwarf_tvar_set_return_8:long (rvalue:long)
{ return (@kregister(0)) = (rvalue) }
# probes
begin /* <- begin */
{ printf("Started\\n\\n") }
kernel.function("_do_fork@kernel/fork.c:2340").call /* pc=_stext+0xeb5c0 */ /* <- kernel.function("_do_fork@kernel/fork.c:2340").call */
{ if ((execname()) == ("sshd")) {
(__global_sshd_fork_count)++;
printf(">>>%s[%5d] %s\\n", execname(), pid(), sprintf("clone_flags=%#x stack_start=%#x stack_size=%#x parent_tidptr=%#x child_tidptr=%#x tls=%#x", __private_fork_stp__dwarf_tvar_get_clone_flags_0(), __private_fork_stp__dwarf_tvar_get_stack_start_1(), __private_fork_stp__dwarf_tvar_get_stack_size_2(), __private_fork_stp__dwarf_tvar_get_parent_tidptr_3(), __private_fork_stp__dwarf_tvar_get_child_tidptr_4(), __private_fork_stp__dwarf_tvar_get_tls_5()));
printf("  PPID=%d\\n", ppid());
printf("  sshd_fork_count=%d\\n", __global_sshd_fork_count);
}
 }
kernel.function("_do_fork@kernel/fork.c:2340").return /* pc=_stext+0xeb5c0 */ /* <- kernel.function("_do_fork@kernel/fork.c:2340").return */
{ if ((execname()) == ("sshd")) {
printf("<<<%s[%5d] %s\\n", execname(), pid(), sprintf("return=%#x", __private_fork_stp__dwarf_tvar_get_return_6()));
printf("  PPID=%d\\n", ppid());
printf("  RETURN:%d\\n\\n", __private_fork_stp__dwarf_tvar_get_return_6());
if ((__global_sshd_fork_count) == (__global_fork_to_fail)) {
printf("!!! fork() will return -1\\n");
__private_fork_stp__dwarf_tvar_set_return_8(-1);
}
;
}
 }
Pass 2: analyzed script: 3 probes, 11 functions, 0 embeds, 2 globals using 3597308virt/250144res/4888shr/3382236data kb, in 3550usr/4850sys/33988real ms.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (systemtap bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7538