Bug 1935084 - [abrt] systemd: greedy_realloc(): systemd-resolved killed by SIGABRT
Summary: [abrt] systemd: greedy_realloc(): systemd-resolved killed by SIGABRT
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 33
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:1a045c90f9752cf34a934316127...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-04 11:03 UTC by Colin.Simpson
Modified: 2021-04-01 15:20 UTC (History)
11 users (show)

Fixed In Version: systemd-248~rc3-1.fc35 systemd-248~rc4-3.fc34 systemd-246.13-1.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-25 00:18:47 UTC
Type: ---


Attachments (Terms of Use)
File: backtrace (27.46 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: core_backtrace (6.83 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: cpuinfo (2.52 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: dso_list (4.67 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: environ (359 bytes, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: limits (1.29 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: maps (30.33 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: mountinfo (7.74 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: open_fds (1.74 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
File: proc_pid_status (1.33 KB, text/plain)
2021-03-04 11:03 UTC, Colin.Simpson
no flags Details
Valgrind output (7.62 KB, text/plain)
2021-03-08 10:46 UTC, Colin.Simpson
no flags Details
Valgrind with debuginfo (8.50 KB, text/plain)
2021-03-08 12:10 UTC, Colin.Simpson
no flags Details

Description Colin.Simpson 2021-03-04 11:03:22 UTC
Description of problem:
Using systemd-resolved in the mode where /etc/resolv.conf is a symlink to stub-resolv.conf.

The only thing maybe slightly unusual about this is this is a corporate environment where there are DNS resolves to 
find AD DC's etc so more (and slightly unusual) TXT records than a home user say.

Version-Release number of selected component:
systemd-246.10-1.fc33

Additional info:
reporter:       libreport-2.14.0
backtrace_rating: 4
cgroup:         0::/system.slice/systemd-resolved.service
cmdline:        /usr/lib/systemd/systemd-resolved
crash_function: greedy_realloc
executable:     /usr/lib/systemd/systemd-resolved
journald_cursor: s=fdd891952425460bbca7beed6c664b5f;i=1cec55;b=3378b7098c7842098a06f8019f356a34;m=13d56f1;t=5bc3ab2ec26f0;x=887c91819e0b76e6
kernel:         5.10.17-200.fc33.x86_64
rootdir:        /
runlevel:       unknown
type:           CCpp
uid:            193

Truncated backtrace:
Thread no. 1 (10 frames)
 #6 greedy_realloc at ../src/basic/alloc-util.c:63
 #7 bus_match_parse at ../src/libsystemd/sd-bus/bus-match.c:791
 #8 bus_add_match_full at ../src/libsystemd/sd-bus/sd-bus.c:3319
 #9 sd_bus_add_match_async at ../src/libsystemd/sd-bus/sd-bus.c:3405
 #10 sd_bus_track_add_name at ../src/libsystemd/sd-bus/bus-track.c:223
 #11 dns_query_bus_track at ../src/resolve/resolved-dns-query.c:1019
 #12 bus_method_resolve_address at ../src/resolve/resolved-bus.c:499
 #13 method_callbacks_run at ../src/libsystemd/sd-bus/bus-objects.c:415
 #14 object_find_and_run at ../src/libsystemd/sd-bus/bus-objects.c:1325
 #15 bus_process_object at ../src/libsystemd/sd-bus/bus-objects.c:1445

Comment 1 Colin.Simpson 2021-03-04 11:03:28 UTC
Created attachment 1760648 [details]
File: backtrace

Comment 2 Colin.Simpson 2021-03-04 11:03:29 UTC
Created attachment 1760649 [details]
File: core_backtrace

Comment 3 Colin.Simpson 2021-03-04 11:03:31 UTC
Created attachment 1760650 [details]
File: cpuinfo

Comment 4 Colin.Simpson 2021-03-04 11:03:32 UTC
Created attachment 1760651 [details]
File: dso_list

Comment 5 Colin.Simpson 2021-03-04 11:03:34 UTC
Created attachment 1760652 [details]
File: environ

Comment 6 Colin.Simpson 2021-03-04 11:03:35 UTC
Created attachment 1760653 [details]
File: limits

Comment 7 Colin.Simpson 2021-03-04 11:03:37 UTC
Created attachment 1760654 [details]
File: maps

Comment 8 Colin.Simpson 2021-03-04 11:03:39 UTC
Created attachment 1760655 [details]
File: mountinfo

Comment 9 Colin.Simpson 2021-03-04 11:03:40 UTC
Created attachment 1760656 [details]
File: open_fds

Comment 10 Colin.Simpson 2021-03-04 11:03:42 UTC
Created attachment 1760657 [details]
File: proc_pid_status

Comment 11 Colin.Simpson 2021-03-04 11:07:06 UTC
This is happening maybe 100 times a day!

Comment 12 Zbigniew Jędrzejewski-Szmek 2021-03-05 16:24:45 UTC
The crash is in greedy_realloc(), but it is the first allocation of that pointer, not a reallocation.
It seems we are calling malloc() and glibc detects a previous corruption. Unfortunately this means
that the backtrace is not terribly useful: it tells us where the error was detected, but not where
it happened.

I don't see anything wrong in the code. I also wrote a fuzzer for that part of the code, but so
far it hasn't found anything useful (https://github.com/systemd/systemd/pull/18890).

Since this is repeatable for you, maybe you could run the code under valgrind? That'd be very
helpful. In a terminal, do:

  sudo systemctl stop systemd-resolved && sudo valgrind /usr/lib/systemd/systemd-resolved

If it print outs any errors, please attach them here. You can kill it with ^C at any time.
Do 'sudo systemctl start systemd-resolved' to restart the service afterwards.

Comment 13 Colin.Simpson 2021-03-08 10:45:51 UTC
Thanks for following up.

This bombed pretty quick for me:

[sae]csimpson: systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved 
Positive Trust Anchors:
. IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Using system hostname 'sae'.
Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting.
Aborted


I'll attach the Valgrind output.

Comment 14 Colin.Simpson 2021-03-08 10:46:51 UTC
Created attachment 1761551 [details]
Valgrind output

Comment 15 Zbigniew Jędrzejewski-Szmek 2021-03-08 11:17:23 UTC
Thanks for the report. Unfortunately it is not useful without debuginfo. Please add them:

  sudo dnf debuginfo-install systemd

(The version of systemd.rpm and systemd-debuginfo.rpm should match.)

Comment 16 Colin.Simpson 2021-03-08 12:09:56 UTC

systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved 
Positive Trust Anchors:
. IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Using system hostname 'sae'.
Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting.
Aborted

Comment 17 Colin.Simpson 2021-03-08 12:10:58 UTC
Created attachment 1761589 [details]
Valgrind with debuginfo

Comment 18 Zbigniew Jędrzejewski-Szmek 2021-03-08 13:46:59 UTC
This should hopefully be fixed by https://github.com/systemd/systemd/pull/18832. I'll need to backport that
patch to systemd-246 though, so it'll be a few days. I'll ask you test it then.

Comment 19 Colin.Simpson 2021-03-08 14:54:27 UTC
Great, I look forward to testing.

Comment 20 Zbigniew Jędrzejewski-Szmek 2021-03-09 13:39:49 UTC
*** Bug 1936559 has been marked as a duplicate of this bug. ***

Comment 21 Fedora Update System 2021-03-23 11:35:31 UTC
FEDORA-2021-1c1a870ceb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb

Comment 22 Fedora Update System 2021-03-23 14:07:01 UTC
FEDORA-2021-ea92e5703f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f

Comment 23 Fedora Update System 2021-03-24 02:44:16 UTC
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ea92e5703f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 24 Fedora Update System 2021-03-24 11:57:49 UTC
FEDORA-2021-1c1a870ceb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb

Comment 25 Fedora Update System 2021-03-25 00:18:47 UTC
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 26 Fedora Update System 2021-03-25 01:31:43 UTC
FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-1c1a870ceb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 27 Fedora Update System 2021-03-27 01:11:04 UTC
FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 28 Colin.Simpson 2021-04-01 15:20:15 UTC
With 
systemd-246.13-1.fc33.x86_64

I'm still seeing this crashing
traps: systemd-resolve[212290] general protection fault ip:7f4fd55982d7 sp:7ffc171684b0 error:0 in libsystemd-shared-246.so[7f4fd54c4000+1a8000]

Do you need me to re-run valgrind against this?


Note You need to log in before you can comment on or make changes to this bug.