Created attachment 1485270 [details] backtrace of the crash packagekitd seems to crash when it tries to do something while another process is locking the libdnf swdb. I saw this twice in a VM while I was trying to debug *another* crash with abrt - presumably abrt was locking the swdb while installing debugging symbols. Here's journal messages from around the time of the crash: Sep 20 12:03:07 localhost-live.happyassassin.net gnome-software[9072]: Only 3 apps for recent list, hiding Sep 20 12:03:07 localhost-live.happyassassin.net packagekitd[10758]: terminate called after throwing an instance of 'SQLite3::LibException' Sep 20 12:03:07 localhost-live.happyassassin.net packagekitd[10758]: what(): Exec failed: database is locked Sep 20 12:03:07 localhost-live.happyassassin.net audit[10758]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:rpm_t:s0 pid=10758 comm="packagekitd" exe="/usr/libexec/packagekitd" sig=6 res=1 Sep 20 12:03:07 localhost-live.happyassassin.net systemd[1]: Started Process Core Dump (PID 12002/UID 0). Sep 20 12:03:07 localhost-live.happyassassin.net audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@2-12002-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 20 12:03:07 localhost-live.happyassassin.net gnome-software[9072]: couldn't parse execption 'org.freedesktop.DBus.Error.NoReply', please report Sep 20 12:03:07 localhost-live.happyassassin.net gnome-software[9072]: not handling error failed for action search-provides: Message recipient disconnected from message bus without replying Sep 20 12:03:07 localhost-live.happyassassin.net audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=packagekit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Sep 20 12:03:07 localhost-live.happyassassin.net systemd[1]: packagekit.service: Main process exited, code=dumped, status=6/ABRT Sep 20 12:03:07 localhost-live.happyassassin.net systemd[1]: packagekit.service: Failed with result 'core-dump'. I will attach the backtrace of the crash.
Looks like it crashed here: https://github.com/hughsie/PackageKit/blob/master/backends/dnf/pk-backend-dnf.c#L471
https://retrace.fedoraproject.org/faf/reports/2315896/ is the FAF for this. I couldn't get abrt to report it properly, had to generate the backtrace via coredumpctl.
I can reproduce broadly the same crash on demand by running 'dnf -y install (something)' in one tab, and 'pkcon -y install (something else)' in another tab. So long as the timing is right, packagekitd crashes.
Reassigning to libdnf. Looks like it's crashing somewhere in swdb code.
PackageKit seems to do some locking via libdnf in pk_backend_repo_set_data_thread , but we don't go through that in this backtrace, AFAICS. We go through pk_transaction_run -> pk_backend_get_update_detail -> pk_backend_job_set_context , where the crash happens. Nothing in that chain does any locking AFAICS.
note the PK code here is like two years old, so the thing that changed here seems to be the introduction of swdb. If libdnf wants consumers to do some locking around calls that now use swdb, that should've been communicated, I guess.
To be specific, the libdnf-side bits here all basically date to e2f2862bedd0c278ed9781624ddc6861686a4760 , "[swdb]: C++ implementation with SWIG bindings."
Aha, I was able to produce a very similar crash in dnf on another try, using a combination of commands suggested by George Goffe in https://bugzilla.redhat.com/show_bug.cgi?id=1596827#c32 . I ran `dnf -y reinstall libreoffice*` in one tab, and `dnf upgrade` in another tab. The `dnf upgrade` crashed: [root@localhost-live tmp]# dnf upgrade Last metadata expiration check: 1:59:58 ago on Thu 20 Sep 2018 11:32:14 AM PDT. Traceback (most recent call last): File "/usr/bin/dnf", line 58, in <module> main.user_main(sys.argv[1:], exit_code=True) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 179, in user_main errcode = main(args) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main return _main(base, args, cli_class, option_parser_class) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 99, in _main return cli_run(cli, base) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 123, in cli_run ret = resolving(cli, base) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 146, in resolving base.resolve(cli.demands.allow_erasing) File "/usr/lib/python3.7/site-packages/dnf/base.py", line 736, in resolve self._transaction = self._goal2transaction(goal) File "/usr/lib/python3.7/site-packages/dnf/base.py", line 633, in _goal2transaction ts.add_install(pkg, obs, reason) File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 256, in add_install ti_new = self.new(new, libdnf.transaction.TransactionItemAction_INSTALL, reason) File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 219, in new rpm_item = self._pkg_to_swdb_rpm_item(pkg) File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 210, in _pkg_to_swdb_rpm_item rpm_item = self.history.swdb.createRPMItem() File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 291, in swdb self._swdb = libdnf.transaction.Swdb(self.dbpath) File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 729, in __init__ this = _transaction.new_Swdb(*args) RuntimeError: C++ std::exception: Exec failed: database is locked it definitely seems like libdnf is missing some locking around swdb use, here.
Proposing this as a Final blocker as a violation of "The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated." I'm gonna argue this is basic enough to constitute a violation of "appropriately", there: multiple libdnf-using-things should not be able to crash each other over a database lock like this.
Discussed during the 2018-09-24 blocker review meeting: [1] The decision to classify this bug as a "RejectedBlocker" was made as this doesn't seem to actually cause any permanent problems or impair the system's ability to install software, it's just a transient ugly error. If we find it can cause permanent issues we'll revisit the bug and go from there. [1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-09-24/f29-blocker-review.2018-09-24-16.05.txt
Note, this seems to be happening in real life quite a lot - msuchy just posted this link to devel@: https://retrace.fedoraproject.org/faf/reports/?opsysreleases=20&opsysreleases=127&component_names=dnf%2Cdnf-plugins-core&associate=__None&first_occurrence_daterange=2018-09-29%3A2018-10-12&last_occurrence_daterange=&order_by=last_occurrence all the ones there with `__init__` listed as the crash function seem to be the same as this (the `__init__` in question, in each case, is libdnf/transaction.py __init__, they're all crashing at this same place).
*** Bug 1636886 has been marked as a duplicate of this bug. ***
*** Bug 1608686 has been marked as a duplicate of this bug. ***
I just got this at the very end of an upgrade after downloading, installing, cleaning up, and verifying. Here's the output: The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Traceback (most recent call last): File "/usr/bin/dnf", line 58, in <module> main.user_main(sys.argv[1:], exit_code=True) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 179, in user_main errcode = main(args) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main return _main(base, args, cli_class, option_parser_class) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 99, in _main return cli_run(cli, base) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 123, in cli_run ret = resolving(cli, base) File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 154, in resolving base.do_transaction(display=displays) File "/usr/lib/python3.7/site-packages/dnf/cli/cli.py", line 235, in do_transaction tid = super(BaseCli, self).do_transaction(display) File "/usr/lib/python3.7/site-packages/dnf/base.py", line 847, in do_transaction # put back our depcheck callback File "/usr/lib/python3.7/site-packages/dnf/base.py", line 997, in _run_transaction try: File "/usr/lib/python3.7/site-packages/dnf/base.py", line 1035, in _verify_transaction File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 508, in end self.swdb.initTransaction() File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 291, in swdb self._swdb = libdnf.transaction.Swdb(self.dbpath) File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 729, in __init__ this = _transaction.new_Swdb(*args) RuntimeError: Exec failed: database is locked
Which dnf, libdnf and PackageKit versions did you use? Have you restarted PackageKit after libdnf upgrade to libdnf-0.22.0-1.fc29 ? Is it reproducible with following builds? libdnf-0.22.0-1.fc29.x86_64 dnf-4.0.4-1.fc29.noarch
Similar problem has been detected: happened during an while running 'dnf update' and there was a kernel update in that transaction reporter: libreport-2.9.6 cmdline: /usr/bin/python3 /bin/dnf -y install --disablerepo=* /tmp/akmods.DZ8oVlsM/results/kmod-xtables-addons-4.18.14-300.fc29.x86_64-3.1-1.fc29.x86_64.rpm crash_function: __init__ exception_type: RuntimeError executable: /bin/dnf interpreter: python3-3.7.0-9.fc29.x86_64 kernel: 4.18.10-300.fc29.x86_64 package: dnf-3.6.1-2.fc29 reason: transaction.py:729:__init__:RuntimeError: Exec failed: database is locked runlevel: N 5 type: Python3 uid: 0
Similar problem has been detected: I typing into tilix terminal: sudo dnf upgrade This problem is coming at home too... but just once. After installing new updates i not meet with this dnf bug. I think work well. The true problem is gnome-software who's dont like to show me the new updates. I like to read package changes, and gnome-software show this informations and urls... but i dont like to update from gnome-sofware because - not found updates, - want to reboot my system - show tothing if i staring upgrade I would like to see what doing my applicaions without starting from a private terminal... I love Fedora <3 reporter: libreport-2.9.6 cmdline: /usr/bin/python3 /usr/bin/dnf upgrade crash_function: __init__ exception_type: RuntimeError executable: /usr/bin/dnf interpreter: python3-3.7.1-2.fc29.x86_64 kernel: 4.18.17-300.fc29.x86_64 package: dnf-4.0.4-2.fc29 reason: transaction.py:729:__init__:RuntimeError: Exec failed: database is locked runlevel: N 5 type: Python3 uid: 0
Similar problem has been detected: trying to update by running sudo dnf upgrade --refresh reporter: libreport-2.10.0 cmdline: /usr/bin/python3 /usr/bin/dnf upgrade --refresh crash_function: __init__ exception_type: RuntimeError executable: /usr/bin/dnf interpreter: python3-3.7.2-5.fc29.x86_64 kernel: 5.0.4-200.fc29.x86_64 package: dnf-4.2.2-1.fc29 reason: transaction.py:729:__init__:RuntimeError: Exec failed: database is locked runlevel: N 5 type: Python3 uid: 0
*** Bug 1630488 has been marked as a duplicate of this bug. ***
*** Bug 1634674 has been marked as a duplicate of this bug. ***
*** Bug 1639965 has been marked as a duplicate of this bug. ***
*** Bug 1676440 has been marked as a duplicate of this bug. ***
*** Bug 1708594 has been marked as a duplicate of this bug. ***
*** Bug 1709198 has been marked as a duplicate of this bug. ***
*** Bug 1710086 has been marked as a duplicate of this bug. ***
*** Bug 1641261 has been marked as a duplicate of this bug. ***
The minimal reproducer is: #!/usr/bin/python3 from libdnf.transaction import * from multiprocessing import Process def f(): while 1: db = Swdb("history.sqlite") db.closeDatabase() if __name__ == '__main__': for num in range(10): Process(target=f).start() The root cause is setting busy timeout after executing pragmas. This causes pragmas fail on locked database, because they are regular statements and it doesn't matter they don't manipulate with tables. PR: https://github.com/rpm-software-management/libdnf/pull/741 Needs to be backported to f29, f30 and rawhide
*** Bug 1716315 has been marked as a duplicate of this bug. ***
*** Bug 1702423 has been marked as a duplicate of this bug. ***
*** Bug 1632822 has been marked as a duplicate of this bug. ***
*** Bug 1730193 has been marked as a duplicate of this bug. ***
*** Bug 1730416 has been marked as a duplicate of this bug. ***
FEDORA-2019-d4b6ede072 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-d4b6ede072
dnf-4.2.5-4.fc29, dnf-plugins-extras-4.0.4-2.fc29, libdnf-0.31.0-6.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d4b6ede072
*** Bug 1741097 has been marked as a duplicate of this bug. ***
dnf-4.2.5-4.fc29, dnf-plugins-extras-4.0.4-2.fc29, libdnf-0.31.0-6.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.