Bug 2070153

Summary: [abrt] Use-after-free under dnf_context_invalidate_full()
Product: [Fedora] Fedora Reporter: Thomas Citharel <bugzilla-redhat>
Component: libdnfAssignee: amatej
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 37CC: ajtbecool, amatej, daniel.mach, gnome-sig, jmracek, jonathan, jrohel, kxra, mblaha, mcrha, pkratoch, rdieter, rhughes, rpm-software-management, smparrish
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
URL: https://retrace.fedoraproject.org/faf/reports/bthash/9ba287614aa71099928ceedfc0f565aa70b4746d
Whiteboard: abrt_hash:03ed8790114c8a564aabc5db1dbaebad9f2227f0;VARIANT_ID=silverblue;
Fixed In Version: libdnf-0.70.2-1.fc38 libdnf-0.70.2-1.fc37 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-30 01:39:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: core_backtrace
none
File: cpuinfo
none
File: dso_list
none
File: environ
none
File: exploitable
none
File: limits
none
File: maps
none
File: mountinfo
none
File: open_fds
none
File: proc_pid_status
none
File: var_log_messages
none
File: backtrace
none
reproducer none

Description Thomas Citharel 2022-03-30 13:51:20 UTC
Version-Release number of selected component:
gnome-software-42.0-1.fc36

Additional info:
reporter:       libreport-2.17.1
backtrace_rating: 3
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/app-gnome-org.gnome.Software-6438.scope
cmdline:        /usr/bin/gnome-software --gapplication-service
crash_function: dnf_context_invalidate_full
executable:     /usr/bin/gnome-software
journald_cursor: s=b91dbf804a664a4ba9764e42ddeeefe9;i=147c0;b=91ec8075647348f283520c40867825cf;m=32d66f89a;t=5db6f7e6b4cd6;x=d148deb755509f65
kernel:         5.17.0-300.fc36.x86_64
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000

Truncated backtrace:
Thread no. 1 (9 frames)
 #0 dnf_context_invalidate_full at /usr/src/debug/libdnf-0.66.0-1.fc36.x86_64/libdnf/dnf-context.cpp:2746
 #2 signal_emit_unlocked_R.isra.0 at ../gobject/gsignal.c:3743
 #5 g_cclosure_marshal_VOID__OBJECTv at ../gobject/gmarshal.c:1910
 #6 _g_closure_invoke_va at ../gobject/gclosure.c:893
 #9 ??
 #10 ??
 #14 g_main_context_iterate.constprop.0 at ../glib/gmain.c:4211
 #15 g_main_context_iteration at ../glib/gmain.c:4276
 #16 g_application_run at ../gio/gapplication.c:2569

Comment 1 Thomas Citharel 2022-03-30 13:51:23 UTC
Created attachment 1869436 [details]
File: backtrace

Comment 2 Thomas Citharel 2022-03-30 13:51:25 UTC
Created attachment 1869437 [details]
File: core_backtrace

Comment 3 Thomas Citharel 2022-03-30 13:51:26 UTC
Created attachment 1869438 [details]
File: cpuinfo

Comment 4 Thomas Citharel 2022-03-30 13:51:28 UTC
Created attachment 1869439 [details]
File: dso_list

Comment 5 Thomas Citharel 2022-03-30 13:51:29 UTC
Created attachment 1869440 [details]
File: environ

Comment 6 Thomas Citharel 2022-03-30 13:51:30 UTC
Created attachment 1869441 [details]
File: exploitable

Comment 7 Thomas Citharel 2022-03-30 13:51:31 UTC
Created attachment 1869442 [details]
File: limits

Comment 8 Thomas Citharel 2022-03-30 13:51:32 UTC
Created attachment 1869443 [details]
File: maps

Comment 9 Thomas Citharel 2022-03-30 13:51:34 UTC
Created attachment 1869444 [details]
File: mountinfo

Comment 10 Thomas Citharel 2022-03-30 13:51:35 UTC
Created attachment 1869445 [details]
File: open_fds

Comment 11 Thomas Citharel 2022-03-30 13:51:36 UTC
Created attachment 1869446 [details]
File: proc_pid_status

Comment 12 Thomas Citharel 2022-03-30 13:51:38 UTC
Created attachment 1869447 [details]
File: var_log_messages

Comment 13 Milan Crha 2022-03-31 07:29:19 UTC
Thanks for a bug report. If I read the backtrace properly, then the dnf_context_invalidate_full() is called by dnf_repo_loader_invalidate(), which is called from a signal callback dnf_repo_loader_mount_changed_cb(), which is connected inside dnf_repo_loader_init(), but it's not disconnected from it in the dnf_repo_loader_finalize(). Either the dnf_repo_loader_init() should disconnect in the finalize(), or it can use g_signal_connect_object() instead. I do not know libdnf, this is only a rough guess.

Comment 14 Milan Crha 2022-04-04 06:45:55 UTC
*** Bug 2071465 has been marked as a duplicate of this bug. ***

Comment 15 amatej 2022-04-05 06:07:35 UTC
Thank you Milan for the analysis, disconnecting the signal in dnf_repo_loader_finalize seems like the right approach however I have not been able to reproduce the crash.

I can see I can trigger the signal by mounting but it never crashed for me. Any idea how to ensure the context is NULL so that we can reproduce it?

Comment 16 Milan Crha 2022-04-05 06:18:24 UTC
I did not try to reproduce it myself. My idea of a reproducer would be to trigger the callback after the context is freed. It won't set the context to NULL (it cannot), but it can trigger a use-after-free, which can be caught by tools like valgrind or libasan, thus being written in a valgring log when you run your 'reproducer' under it:

   $ G_SLICE=always-malloc valgrind --track-origins=yes --aspace-minaddr=0x100000000 ./reproducer

A very naive debug patch would add debug prints to the context constructor and destructor (printing the context address), with a similar print in the callback to verify the callback was called after the object was freed.

Comment 17 Mateus Rodrigues Costa 2022-05-16 20:57:48 UTC
Similar problem has been detected:

Laptop was suspended at the time.

reporter:       libreport-2.17.1
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/dbus-:1.2-org.gnome.Software
cmdline:        /usr/bin/gnome-software --gapplication-service
crash_function: dnf_context_invalidate_full
executable:     /usr/bin/gnome-software
journald_cursor: s=4d9209d19c614f01ad854f5039827b78;i=2b668;b=65e28c8a8b774e5185a74a4fe0ef9108;m=28f22a34c;t=5df2711603525;x=dc1b2b2159be29c5
kernel:         5.17.6-300.fc36.x86_64
package:        gnome-software-42.1-1.fc36
reason:         gnome-software killed by SIGSEGV
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000

Comment 18 Mateus Rodrigues Costa 2022-05-16 20:57:50 UTC
Created attachment 1880289 [details]
File: backtrace

Comment 19 Milan Crha 2022-05-25 06:04:19 UTC
*** Bug 2089723 has been marked as a duplicate of this bug. ***

Comment 20 Joe Mou 2022-06-10 14:07:30 UTC
Similar problem has been detected:

Nothing; crashes when starting Gnome

reporter:       libreport-2.17.1
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/app-gnome-org.gnome.Software-1850.scope
cmdline:        /usr/bin/gnome-software --gapplication-service
crash_function: dnf_context_invalidate_full
executable:     /usr/bin/gnome-software
journald_cursor: s=1794480aea06419fb643eea69e6ed8ae;i=3872;b=dea4a017d28b474eb2ff022687d35f62;m=6ceeb16;t=5e1172d47ed58;x=5f88130de6d84e19
kernel:         5.17.12-300.fc36.x86_64
package:        gnome-software-42.2-1.fc36
reason:         gnome-software killed by SIGSEGV
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000

Comment 21 Giovanni Campagna 2022-07-22 21:16:19 UTC
Similar problem has been detected:

Happens just by itself, in the background. Seems to correlate with unmounting external disks, but hard to tell.

reporter:       libreport-2.17.1
backtrace_rating: 4
cgroup:         0::/user.slice/user-1000.slice/user/app.slice/app-gnome-org.gnome.Software-2974.scope
cmdline:        /usr/bin/gnome-software --gapplication-service
crash_function: dnf_context_invalidate_full
executable:     /usr/bin/gnome-software
journald_cursor: s=f85d087d84bf49beaa6f50595ce40b07;i=132600;b=19fecc490cf844c584dc2de551dc5b1f;m=37155c1;t=5e46a68152b3a;x=912e68afefdf7bf7
kernel:         5.18.11-200.fc36.x86_64
package:        gnome-software-42.3-1.fc36
reason:         gnome-software killed by SIGSEGV
rootdir:        /
runlevel:       N 5
type:           CCpp
uid:            1000

Comment 22 Milan Crha 2022-10-03 06:18:57 UTC
*** Bug 2131281 has been marked as a duplicate of this bug. ***

Comment 23 Milan Crha 2022-10-03 06:25:14 UTC
@amatej There are people hitting this, not talking that use-after-free is a bad thing. There are usually filled CVE-s for such things, because their consequences can be disastrous (like due to it modifying unpredictable part of the code).

I can understand the need for a reproducer, to be able to confirm that the change really works, but I have nothing than valgrind or a custom build with debug prints confirming it works (for such debug prints, I'd add a print at the place where the signal is added, where would be two handlers added, one for a debug print that the signal handler had been called and with what contenxt and the original one; and then a print at the context free function. The sequence of the prints will show when was done what.)

Comment 24 Jaroslav Mracek 2022-10-06 07:54:10 UTC
Thank you very much for highlighting the issue. The issue is in the part that originate in LIBHIF - a library of PackageKit. I know that the issue is in the LIBDNF library, but I guess that only experts on PackageKit and workflows there could resolve the issue, because there is no reproducer, and no other hint from valgrin. Please take it as a request for a help - I am changing the component to PackageKit.

Comment 25 Milan Crha 2022-10-06 12:32:23 UTC
Created attachment 1916468 [details]
reproducer

The first line contains a comment how to compile & run it. Current output:

> Before first pass
> After first pass
>
> (process:67130): GLib-GObject-WARNING **: 14:29:43.721: instance with invalid (NULL) class pointer
> 
> (process:67130): GLib-GObject-CRITICAL **: 14:29:43.721: g_signal_emit_valist: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
> Segmentation fault (core dumped)

while it should end with "After second pass", no runtime warnings and no crash.

Comment 26 amatej 2022-10-12 06:43:57 UTC
Thank you again for helping @mcrha!

I made a PR for libdnf: https://github.com/rpm-software-management/libdnf/pull/1582

Comment 27 Fedora Update System 2023-07-28 13:35:07 UTC
FEDORA-2023-d3d8f7571a has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-d3d8f7571a

Comment 28 Fedora Update System 2023-07-28 13:35:11 UTC
FEDORA-2023-05391614c9 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-05391614c9

Comment 29 Fedora Update System 2023-07-29 02:17:56 UTC
FEDORA-2023-d3d8f7571a has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-d3d8f7571a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-d3d8f7571a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 30 Fedora Update System 2023-07-29 02:19:31 UTC
FEDORA-2023-05391614c9 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-05391614c9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-05391614c9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 31 Fedora Update System 2023-07-30 01:39:06 UTC
FEDORA-2023-d3d8f7571a has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 32 Fedora Update System 2023-08-08 02:58:20 UTC
FEDORA-2023-05391614c9 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.