Bug 2215412 - Performance issue with systemd-coredump and container process linking 2000 shared libraries.
Summary: Performance issue with systemd-coredump and container process linking 2000 sh...
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: systemd
Version: 9.2
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: David Tardon
QA Contact: Frantisek Sumsal
URL:
Whiteboard:
Depends On:
Blocks: 2222259
TreeView+ depends on / blocked
 
Reported: 2023-06-15 21:01 UTC by Paulo Andrade
Modified: 2023-08-04 11:05 UTC (History)
6 users (show)

Fixed In Version: systemd-252-16.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2222259 (view as bug list)
Environment:
Last Closed:
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github redhat-plumbers systemd-rhel9 pull 177 0 None open Discard PT_LOAD segment early based on the start address 2023-06-26 13:37:24 UTC
Github systemd systemd pull 28093 0 None Merged elf-util: discard PT_LOAD segment early based on the start address. 2023-06-26 07:55:31 UTC
Github systemd systemd pull 28128 0 None Merged elf-util: check for overflow when computing end of core's PT_LOAD segments 2023-06-26 07:55:44 UTC
Red Hat Issue Tracker RHELPLAN-160017 0 None None None 2023-06-15 21:02:43 UTC

Internal Links: 2218039

Description Paulo Andrade 2023-06-15 21:01:23 UTC
Sample reproducer:

"""
cat > lib.c << EOF
int TOTO = 42;
EOF

cat > toto.c << EOF
#include <stdio.h>
int main() {
        printf("started\n");
        int* i = 0;
        *i = 42;
}
EOF


LIBS=""
for i in `seq 2000`; do
        [[ -f liba$i.so ]] || gcc -shared -fPIC -o liba$i.so lib.c -D TOTO=var$i &
        LIBS="${LIBS} -la$i"
done
wait

echo libs built
gcc -o toto toto.c -L. -Wl,-rpath,$(pwd) ${LIBS}
echo running binary
ulimit -c unlimited
time ./toto
"""

  This will trigger systemd-coredump (forking and calling itself sd-parse-elf)
calling elf_gfetdata_rawchunk from elfutils, that does a linear search
(could be better). Then, for every chunk parsed earliersd-parse-elf calls again
elf_getdata_rawchunk, making it of O(N^2) complexity. That is already bad.

  See https://sourceware.org/pipermail/elfutils-devel/2023q2/006225.html for
a sample/proposed patch for systemd-coredump, that would turn the quadratic
performance issue into a linear one. The above link also shows that while
elf_gfetdata_rawchunk could be made better, systemd-coredump is not using it
correctly.

  The possible performance issue in elfutils is https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=8db849976f07046d27b4217e9ebd08d5623acc4f

Comment 2 Mark Wielaard 2023-06-22 11:54:26 UTC
Note that the systemd upstream patch seems to fix the performance issue by reducing the calls to elfutils libelf elf_getdata_rawchunk.
But there was indeed also a performance issue in elf_getdata_rawchunk. Proposed patch for that is:
https://inbox.sourceware.org/elfutils-devel/3465f95ee22b1ac433f1268f113e3813430be70a.camel@klomp.org/
We can also try to backport that one. The systemd fix seems easier though, and has a bigger performance impact.

Comment 3 Romain Geissler 2023-06-22 23:39:07 UTC
Please note that the systemd patch has a follow up fix here: https://github.com/systemd/systemd/pull/28128

Comment 4 Scott Dodson 2023-06-26 13:59:43 UTC
Requesting 9.2.z


Note You need to log in before you can comment on or make changes to this bug.