Bug 1935084

Created attachment 1760648 [details]
File: backtrace

Created attachment 1760649 [details]
File: core_backtrace

Created attachment 1760650 [details]
File: cpuinfo

Created attachment 1760651 [details]
File: dso_list

Created attachment 1760652 [details]
File: environ

Created attachment 1760653 [details]
File: limits

Created attachment 1760654 [details]
File: maps

Created attachment 1760655 [details]
File: mountinfo

Created attachment 1760656 [details]
File: open_fds

Created attachment 1760657 [details]
File: proc_pid_status

This is happening maybe 100 times a day!

The crash is in greedy_realloc(), but it is the first allocation of that pointer, not a reallocation.
It seems we are calling malloc() and glibc detects a previous corruption. Unfortunately this means
that the backtrace is not terribly useful: it tells us where the error was detected, but not where
it happened.

I don't see anything wrong in the code. I also wrote a fuzzer for that part of the code, but so
far it hasn't found anything useful (https://github.com/systemd/systemd/pull/18890).

Since this is repeatable for you, maybe you could run the code under valgrind? That'd be very
helpful. In a terminal, do:

  sudo systemctl stop systemd-resolved && sudo valgrind /usr/lib/systemd/systemd-resolved

If it print outs any errors, please attach them here. You can kill it with ^C at any time.
Do 'sudo systemctl start systemd-resolved' to restart the service afterwards.

Thanks for following up.

This bombed pretty quick for me:

[sae]csimpson: systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved 
Positive Trust Anchors:
. IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Using system hostname 'sae'.
Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting.
Aborted


I'll attach the Valgrind output.

Created attachment 1761551 [details]
Valgrind output

Thanks for the report. Unfortunately it is not useful without debuginfo. Please add them:

  sudo dnf debuginfo-install systemd

(The version of systemd.rpm and systemd-debuginfo.rpm should match.)

systemctl stop systemd-resolved && sudo valgrind --log-file=/root/resolved-valgrind.txt /usr/lib/systemd/systemd-resolved 
Positive Trust Anchors:
. IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Using system hostname 'sae'.
Assertion 'p->n_ref > 0' failed at src/resolve/resolved-dns-query.c:69, function dns_query_candidate_unref(). Aborting.
Aborted

Created attachment 1761589 [details]
Valgrind with debuginfo

This should hopefully be fixed by https://github.com/systemd/systemd/pull/18832. I'll need to backport that
patch to systemd-246 though, so it'll be a few days. I'll ask you test it then.

Great, I look forward to testing.

*** Bug 1936559 has been marked as a duplicate of this bug. ***

FEDORA-2021-1c1a870ceb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb

FEDORA-2021-ea92e5703f has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f

FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ea92e5703f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea92e5703f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-1c1a870ceb has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb

FEDORA-2021-ea92e5703f has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-1c1a870ceb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-1c1a870ceb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-1c1a870ceb has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

With 
systemd-246.13-1.fc33.x86_64

I'm still seeing this crashing
traps: systemd-resolve[212290] general protection fault ip:7f4fd55982d7 sp:7ffc171684b0 error:0 in libsystemd-shared-246.so[7f4fd54c4000+1a8000]

Do you need me to re-run valgrind against this?