Description of problem: Using systemd-resolved in the mode where /etc/resolv.conf is a symlink to stub-resolv.conf. The only thing maybe slightly unusual about this is this is a corporate environment where there are DNS resolves to find AD DC's etc so more (and slightly unusual) TXT records than a home user say. Version-Release number of selected component: systemd-246.10-1.fc33 Additional info: reporter: libreport-2.14.0 backtrace_rating: 4 cgroup: 0::/system.slice/systemd-resolved.service cmdline: /usr/lib/systemd/systemd-resolved crash_function: greedy_realloc executable: /usr/lib/systemd/systemd-resolved journald_cursor: s=fdd891952425460bbca7beed6c664b5f;i=1cec55;b=3378b7098c7842098a06f8019f356a34;m=13d56f1;t=5bc3ab2ec26f0;x=887c91819e0b76e6 kernel: 5.10.17-200.fc33.x86_64 rootdir: / runlevel: unknown type: CCpp uid: 193 Truncated backtrace: Thread no. 1 (10 frames) #6 greedy_realloc at ../src/basic/alloc-util.c:63 #7 bus_match_parse at ../src/libsystemd/sd-bus/bus-match.c:791 #8 bus_add_match_full at ../src/libsystemd/sd-bus/sd-bus.c:3319 #9 sd_bus_add_match_async at ../src/libsystemd/sd-bus/sd-bus.c:3405 #10 sd_bus_track_add_name at ../src/libsystemd/sd-bus/bus-track.c:223 #11 dns_query_bus_track at ../src/resolve/resolved-dns-query.c:1019 #12 bus_method_resolve_address at ../src/resolve/resolved-bus.c:499 #13 method_callbacks_run at ../src/libsystemd/sd-bus/bus-objects.c:415 #14 object_find_and_run at ../src/libsystemd/sd-bus/bus-objects.c:1325 #15 bus_process_object at ../src/libsystemd/sd-bus/bus-objects.c:1445
Created attachment 1760648 [details] File: backtrace
Created attachment 1760649 [details] File: core_backtrace
Created attachment 1760650 [details] File: cpuinfo
Created attachment 1760651 [details] File: dso_list
Created attachment 1760652 [details] File: environ
Created attachment 1760653 [details] File: limits
Created attachment 1760654 [details] File: maps
Created attachment 1760655 [details] File: mountinfo
Created attachment 1760656 [details] File: open_fds
Created attachment 1760657 [details] File: proc_pid_status
This is happening maybe 100 times a day!
The crash is in greedy_realloc(), but it is the first allocation of that pointer, not a reallocation. It seems we are calling malloc() and glibc detects a previous corruption. Unfortunately this means that the backtrace is not terribly useful: it tells us where the error was detected, but not where it happened. I don't see anything wrong in the code. I also wrote a fuzzer for that part of the code, but so far it hasn't found anything useful (https://github.com/systemd/systemd/pull/18890). Since this is repeatable for you, maybe you could run the code under valgrind? That'd be very helpful. In a terminal, do: sudo systemctl stop systemd-resolved && sudo valgrind /usr/lib/systemd/systemd-resolved If it print outs any errors, please attach them here. You can kill it with ^C at any time. Do 'sudo systemctl start systemd-resolved' to restart the service afterwards.
Thanks for following up. This bombed pretty quick for me: [sae]csimpson: systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved Positive Trust Anchors: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test Using system hostname 'sae'. Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting. Aborted I'll attach the Valgrind output.
Created attachment 1761551 [details] Valgrind output
Thanks for the report. Unfortunately it is not useful without debuginfo. Please add them: sudo dnf debuginfo-install systemd (The version of systemd.rpm and systemd-debuginfo.rpm should match.)
systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved Positive Trust Anchors: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test Using system hostname 'sae'. Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting. Aborted
Created attachment 1761589 [details] Valgrind with debuginfo
This should hopefully be fixed by https://github.com/systemd/systemd/pull/18832. I'll need to backport that patch to systemd-246 though, so it'll be a few days. I'll ask you test it then.
Great, I look forward to testing.
*** Bug 1936559 has been marked as a duplicate of this bug. ***
FEDORA-2021-1c1a870ceb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb
FEDORA-2021-ea92e5703f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ea92e5703f` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-1c1a870ceb` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
With systemd-246.13-1.fc33.x86_64 I'm still seeing this crashing traps: systemd-resolve[212290] general protection fault ip:7f4fd55982d7 sp:7ffc171684b0 error:0 in libsystemd-shared-246.so[7f4fd54c4000+1a8000] Do you need me to re-run valgrind against this?