Hide Forgot
Description of problem: running `rpm-ostree upgrade` always results in «error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`» or «error: Could not activate remote peer.» Version-Release number of selected component: rpm-ostree-2021.1-2.fc33 Additional info: reporter: libreport-2.14.0 backtrace_rating: 4 cmdline: /usr/bin/rpm-ostree start-daemon crash_function: _ostree_loose_path executable: /usr/bin/rpm-ostree journald_cursor: s=34b72559ba2f43ba96d6004be842ad67;i=bbdb4;b=feba4eebbf4a4d09935bf4dce5883c43;m=3060e983e2;t=5ba971d585a2c;x=3cf0e83a11df26f5 kernel: 5.10.12-200.fc33.x86_64 rootdir: / runlevel: N 5 type: CCpp uid: 0 Truncated backtrace: Thread no. 1 (10 frames) #0 _ostree_loose_path at src/libostree/ostree-core.c:1647 #1 load_metadata_internal at src/libostree/ostree-repo.c:3734 #2 ostree_repo_load_commit at src/libostree/ostree-repo.c:4540 #3 rpmostree_pkgcache_find_pkg_header(OstreeRepo*, char const*, char const*, GVariant**, GCancellable*, GError**) at src/libpriv/rpmostree-core.cxx:1053 #4 finalize_overlays at src/daemon/rpmostree-sysroot-upgrader.cxx:854 #5 rpmostree_sysroot_upgrader_prep_layering at src/daemon/rpmostree-sysroot-upgrader.cxx:1249 #6 rpmostree_sysroot_upgrader_prep_layering(RpmOstreeSysrootUpgrader*, RpmOstreeSysrootUpgraderLayeringType*, gboolean*, GCancellable*, GError**) at src/daemon/rpmostree-sysroot-upgrader.cxx:1225 #7 deploy_transaction_execute(RpmostreedTransaction*, GCancellable*, GError**) at src/daemon/rpmostreed-transaction-types.cxx:1330 #8 transaction_execute_thread(GTask*, gpointer, gpointer, GCancellable*) at src/daemon/rpmostreed-transaction.cxx:339 #9 g_task_thread_pool_thread at ../gio/gtask.c:1417
Created attachment 1755242 [details] File: backtrace
Created attachment 1755243 [details] File: cgroup
Created attachment 1755244 [details] File: core_backtrace
Created attachment 1755245 [details] File: cpuinfo
Created attachment 1755246 [details] File: dso_list
Created attachment 1755247 [details] File: environ
Created attachment 1755248 [details] File: exploitable
Created attachment 1755249 [details] File: limits
Similar problem has been detected: running `rpm-ostree upgrade` always results in «error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`» or «error: Could not activate remote peer.» reporter: libreport-2.14.0 backtrace_rating: 4 cmdline: /usr/bin/rpm-ostree start-daemon crash_function: _ostree_loose_path executable: /usr/bin/rpm-ostree journald_cursor: s=34b72559ba2f43ba96d6004be842ad67;i=bbdb4;b=feba4eebbf4a4d09935bf4dce5883c43;m=3060e983e2;t=5ba971d585a2c;x=3cf0e83a11df26f5 kernel: 5.10.12-200.fc33.x86_64 package: rpm-ostree-2021.1-2.fc33 reason: rpm-ostree killed by SIGSEGV rootdir: / runlevel: N 5 type: CCpp uid: 0
I think with https://github.com/coreos/rpm-ostree/pull/2545, it'll make it error out instead of crash. Though I'm still not sure why it would error. Can you show all the packages you have layered? E.g. `rpm-ostree status -vb`, then `rpm-ostree db diff $base_commit $commit`. Also the output of `ostree refs`.
This is on one machine experiencing this Attempt one: ``` $ rpm-ostree status -vb error: Could not activate remote peer. ``` Attempt two: ``` $ rpm-ostree status -vb State: idle AutomaticUpdates: disabled BootedDeployment: ● ostree://fedora:fedora/33/x86_64/testing/silverblue Version: 33.20210202.0 (2021-02-02T01:34:58Z) BaseCommit: 7231de59746b9df7f14e892bd95b8e41ac231379d5ee6ce8a12a4e3e51011f53 ├─ repo-0 (2020-10-19T23:27:19Z) ├─ repo-1 (2021-02-02T01:03:29Z) ├─ repo-2 (2021-02-02T01:06:12Z) └─ repo-3 (2021-02-01T01:40:54Z) Commit: de1a81e88de39ad87ce58740a7fcb766133d4fef80db5f782c0d127aac2e4bc2 ├─ fedora-cisco-openh264 (2020-08-25T19:10:34Z) ├─ jkonecny-kakoune (2021-02-02T00:14:08Z) ├─ fedora-modular (2020-10-19T23:04:43Z) ├─ updates-modular (2021-01-29T02:39:23Z) ├─ copr:copr.fedorainfracloud.org:fuhrmann:ctop (2020-10-25T22:01:17Z) ├─ nodesource (2021-01-04T18:56:08Z) ├─ rpmfusion-free-updates (2021-02-01T13:31:03Z) ├─ updates-testing-modular (2021-01-26T02:18:15Z) ├─ updates-testing (2021-02-02T01:50:03Z) ├─ updates (2021-02-02T03:01:09Z) ├─ code (2021-02-02T03:05:14Z) ├─ google-chrome-beta (2021-01-28T19:29:44Z) ├─ rpmfusion-free (2020-10-16T17:39:10Z) ├─ rpmfusion-nonfree (2020-10-16T18:04:38Z) ├─ rpmfusion-nonfree-updates (2021-02-01T13:50:36Z) ├─ rpmfusion-free-updates-testing (2021-02-01T13:32:46Z) ├─ fedora (2020-10-19T23:27:19Z) ├─ rpmfusion-nonfree-updates-testing (2021-02-01T13:50:42Z) └─ updates-archive (2021-02-02T04:44:18Z) Staged: no StateRoot: fedora GPGSignature: 1 signature Signature made Mon 01 Feb 2021 07:35:07 PM CST using RSA key ID 49FD77499570FF31 Good signature from "Fedora <fedora-33-primary@fedoraproject.org>" RemovedBasePackages: pulseaudio-module-bluetooth 14.0-2.fc33 InactiveRequests: google-chrome-beta LayeredPackages: NetworkManager-l2tp-gnome NetworkManager-strongswan OpenIPMI OpenIPMI-tools PackageKit-gstreamer-plugin ShellCheck abrt-desktop alsa-firmware alsa-tools apr-devel asciidoc autojump autossh avahi-compat-libdns_sd-devel axel baobab bind-dnssec-utils blueman cabal-install cargo ccache celluloid certbot cmake code crontabs docker-compose epiphany et f2fs-tools fakeroot fedora-workstation-repositories ffmpeg firewall-config fuse-devel git-cola gitg gitk gnome-tweak-tool gnutls-devel gstreamer-ffmpeg gstreamer-plugin-crystalhd gstreamer-plugins-bad gstreamer-plugins-bad-nonfree gstreamer-plugins-ugly gstreamer1-libav gstreamer1-plugin-openh264 gstreamer1-plugins-bad-free-extras gstreamer1-plugins-bad-freeworld gstreamer1-plugins-fc gstreamer1-plugins-good-extras gstreamer1-plugins-ugly gstreamer1-plugins-ugly-free gstreamer1-vaapi help2man htop hw-probe iotop ipmitool java-1.8.0-openjdk java-latest-openjdk kakoune krb5-devel krb5-server libapreq2-devel libattr-devel libicu-devel libuuid-devel libvirt libvirt-devel make mariadb-connector-c mariadb-connector-c-devel meld moby-engine moreutils mosh mozilla-openh264 ncurses-devel net-tools nodejs nss-tools openh264 pam-devel parallel pavucontrol perltidy pitivi pulseaudio-module-bluetooth-freeworld pv pygtk2 pygtk2-devel python-devel python-wxpython4 python2-setuptools python3-yaml python3.6 python3.7 python3.8 python35 reptyr rubber ruby-devel rust samba screen scribus setroubleshoot snapd socat strongswan sushi texlive-beamer texlive-collection-fontsextra texlive-collection-fontsrecommended texlive-collection-fontutils texlive-collection-latexextra texlive-gmutils texlive-latex-fonts texlive-logical-markup-utils texlive-ttfutils tidy tig tilix tmate tmux totem traceroute udftools vagrant vagrant-libvirt vagrant-lxc valgrind vim virt-manager vlc xorriso yasm LocalPackages: rpmfusion-free-release-33-0.2.noarch rpmfusion-nonfree-release-33-0.3.noarch nodesource-release-fc33-1.noarch zoom-5.5.6955.0131-1.x86_64 google-chrome-beta-88.0.4324.50-1.x86_64 ```
Oops, posted before pasting the other two: ``` $ rpm-ostree db diff $base_commit $commit error: No pending or rollback deployment to diff against ``` ``` $ ostree refs rpmostree/base/0 fedora:fedora/33/x86_64/testing/silverblue fedora:fedora/31/x86_64/silverblue ostree/1/1/0 ```
On machine #2 experincing this issue: ``` ~> rpm-ostree status -vb State: idle AutomaticUpdates: disabled BootedDeployment: ● ostree://fedora:fedora/33/x86_64/silverblue Version: 33.20210204.0 (2021-02-04T01:41:19Z) BaseCommit: 39ac11c939acaa8cc7bb53635bd946b7c76ac74a805272ed1720b615b75dfcb4 ├─ repo-0 (2020-10-19T23:27:19Z) ├─ repo-1 (2021-02-04T01:12:54Z) └─ repo-2 (2021-02-04T01:17:13Z) Commit: 848ad80eea2ed5390a777f632e30dd33bb066a075060917bd5921c2e005be238 ├─ nodesource (2021-01-04T18:55:59Z) ├─ updates (2021-02-03T00:58:58Z) ├─ fedora-cisco-openh264 (2020-08-25T19:10:34Z) ├─ google-chrome-beta (2021-02-03T18:43:05Z) ├─ updates-testing (2021-02-03T02:14:55Z) ├─ copr:copr.fedorainfracloud.org:atim:ytop (2020-10-25T04:40:55Z) ├─ rpmfusion-free (2020-10-16T17:39:10Z) ├─ rpmfusion-nonfree-updates-testing (2021-02-01T13:50:42Z) ├─ fedora (2020-10-19T23:27:19Z) ├─ copr:copr.fedorainfracloud.org:fuhrmann:ctop (2020-10-25T22:01:17Z) ├─ rpmfusion-free-updates-testing (2021-02-01T13:32:46Z) ├─ rpmfusion-nonfree (2020-10-16T18:04:38Z) └─ updates-archive (2021-02-03T03:24:30Z) Staged: no StateRoot: fedora GPGSignature: 1 signature Signature made Wed 03 Feb 2021 07:41:27 PM CST using RSA key ID 49FD77499570FF31 Good signature from "Fedora <fedora-33-primary@fedoraproject.org>" LayeredPackages: ShellCheck abrt-desktop adb alsa-firmware android-tools autojump autossh axel baobab cabal-install cargo celluloid certbot ctop emacs et exa f2fs-tools fd-find fedora-workstation-repositories ffmpeg firewall-config fish ghc git-annex git-cola gitg gitk gnome-tweak-tool google-chrome-beta gstreamer-ffmpeg gstreamer-plugins-bad gstreamer-plugins-bad-nonfree gstreamer-plugins-espeak gstreamer1-libav gstreamer1-plugin-openh264 gstreamer1-plugins-bad-freeworld gstreamer1-plugins-ugly gstreamer1-plugins-ugly-free gstreamer1-vaapi haskell-platform htop idris iotop kakoune langpacks-en magic-wormhole mosh nodejs pavucontrol reptyr ripgrep rust screen scribus setroubleshoot snapd sushi ternimal tig tilix tmux tokei totem traceroute udftools vim virt-manager vlc ytop LocalPackages: nodesource-release-fc32-1.noarch code-insiders-1.48.0-1597141686.el7.x86_64 rpmfusion-free-release-33-0.2.noarch rpmfusion-nonfree-release-33-0.3.noarch Publii-0.36.1-1.x86_64 zoom-5.5.6955.0131-1.x86_64 ``` ``` ~> rpm-ostree db diff $base_commit $commit error: No pending or rollback deployment to diff against ``` ``` ~> ostree refs fedora:fedora/33/x86_64/silverblue rpmostree/base/0 ostree/0/1/0 fedora:fedora/32/x86_64/testing/silverblue ```
Someone reported this on Reddit as well. Any ideas of a workaround to get unstuck and able to upgrade again? Before these errors started, I saw upon attempting to upgrade a warning about inactive base removals for `pulseaudio-module-bluetooth`.
We've reverted libsolv for now while we investigate this: https://pagure.io/workstation-ostree-config/pull-request/190. Until this is fixed, you will have to drop your overrides using `rpm-ostree reset -o` until you upgrade to the next update with the libsolv revert.
FWIW I don't have any overrides and I've rolled back to Silverblue 33.20210202.0 (2021-02-02T02:52:11Z) with libsolv-0.7.15-1.fc33 and rpm-ostree-2021.1-2.fc33 then did usroverlay + rpm-ostreed restart with rpm-ostree-2021.1-3, rpm-ostree upgrade still ends with stack trace starting in _ostree_loose_path Also same SEGV with /etc/rpm/macros.rpm-ostree as suggested in https://github.com/coreos/rpm-ostree/pull/2545#issuecomment-775388706
> We've reverted libsolv for now while we investigate this: https://pagure.io/workstation-ostree-config/pull-request/190 When is the updated image expected to hit fedora:fedora/33/x86_64/silverblue ? As of today Feb 10th rpm-ostree upgrade --preview shows: ... AvailableUpdate: Version: 33.20210208.0 (2021-02-08T00:58:46Z) Commit: 655b20930843360a5945a2711de72d9085b94a05a627d44dd2eb9b273f09a58a ... libsolv 0.7.15-1.fc33 -> 0.7.17-1.fc33
The nuclear fix for this is to: 1. wait until https://bodhi.fedoraproject.org/updates/FEDORA-2021-9091468793 is in stable and in a compose 2. drop all layerings (`rpm-ostree reset`) 3. upgrade && reboot; this should be a pure OSTree operation, so no RPM should be involved 4. re-add any wanted layerings/overrides
FEDORA-2021-9091468793 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-9091468793
FEDORA-2021-9091468793 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
I think this bug has still not been fixed. I upgraded with `rpm-ostree rebase d9ce789c1b1932c58508e17b5bd18721540b57dca09097b9bf26b9014b702c92` hence my system is at: $ rpm-ostree status -v State: idle AutomaticUpdates: disabled Deployments: ● ostree://fedora:d9ce789c1b1932c58508e17b5bd18721540b57dca09097b9bf26b9014b702c92 Version: 33.20210212.0 (2021-02-12T00:49:25Z) BaseCommit: d9ce789c1b1932c58508e17b5bd18721540b57dca09097b9bf26b9014b702c92 ├─ repo-0 (2020-10-19T23:27:19Z) ├─ repo-1 (2021-02-12T00:16:46Z) └─ repo-2 (2021-02-12T00:23:48Z) Commit: 3c5751f12ca30edc20284b984a8d63ca84097f854f9382b54b74d3448a53173d Staged: no StateRoot: fedora GPGSignature: 1 signature Signature made Fri 12 Feb 2021 02:49:31 AM EET using RSA key ID 49FD77499570FF31 Good signature from "Fedora <fedora-33-primary@fedoraproject.org>" RemovedBasePackages: firefox 85.0.1-1.fc33 LayeredPackages: akmod-nvidia arm-image-installer fcust fedora-workstation-repositories gnome-shell-extension-appindicator gnome-shell-extension-dash-to-dock nextcloud-client pam-u2f pamu2fcfg simple-scan xorg-x11-drv-nvidia xorg-x11-drv-nvidia-cuda However I will get the following error: $ rpm-ostree upgrade --preview error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.
I agree it does not appear fixed for me either. $ rpm-ostree cleanup -bpm Deployments unchanged. $ rpm-ostree upgrade ⠂ Receiving metadata objects: 0/(estimating) -/s 0 bytes... Receiving metadata objects: 0/(estimating) -/s 0 bytes... done Checking out tree d9ce789... done error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.
Please follow the instructions in https://bugzilla.redhat.com/show_bug.cgi?id=1925584#c18. (And actually, after step three but before step four, you'll want to do `rpm-ostree cleanup -r` to flush out all cached packages as well).