Bug 1975660

Summary: [Fedora 33]rpm-ostree upgrade failed with error: Bus owner changed, aborting
Product: [Fedora] Fedora Reporter: Yuxin Sun <yuxisun>
Component: rpm-ostreeAssignee: Colin Walters <walters>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 33CC: atodorov, development.group, dustymabe, jlebon, jonathan, lucab, miabbott, obudai, robertthomasfairley, tgunders, travier, walters, xiaofwan, yih
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rpm-ostree-2021.6-2.fc33 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1959907 Environment:
Last Closed: 2021-07-04 01:08:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1959907    
Bug Blocks:    

Description Yuxin Sun 2021-06-24 07:18:22 UTC 
Also hit this issue in Fedora 33. The fix version is 2021.5. We may need a rebase in Fedora 33.

+++ This bug was initially created as a clone of Bug #1959907 +++

Description of problem:
When run rpm-ostree upgrade on RHEL 8.5 ostree system, rpm-ostreed crashed with core dump.
Do not have this issue on RHEL 8.5 ostree system.

[admin@vm-bios ~]$ journalctl -u rpm-ostreed
-- Logs begin at Wed 2021-05-12 14:38:25 UTC, end at Wed 2021-05-12 15:08:45 UTC. --
May 12 14:54:26 vm-bios systemd[1]: Starting rpm-ostree System Management Daemon...
May 12 14:54:26 vm-bios rpm-ostree[1074]: Reading config file '/etc/rpm-ostreed.conf'
May 12 14:54:26 vm-bios systemd[1]: Started rpm-ostree System Management Daemon.
May 12 14:54:26 vm-bios rpm-ostree[1074]: In idle state; will auto-exit in 60 seconds
May 12 14:54:26 vm-bios rpm-ostree[1074]: client(id:cli dbus:1.23 unit:session-3.scope uid:0) added; new total=1
May 12 14:54:26 vm-bios rpm-ostree[1074]: Locked sysroot
May 12 14:54:26 vm-bios rpm-ostree[1074]: Initiated txn Upgrade for client(id:cli dbus:1.23 unit:session-3.scope uid:0): /org/projectatomic/rpmostree1/edge_commit
May 12 14:54:30 vm-bios rpm-ostree[1074]: libostree pull from 'edge-commit' for rhel/8/x86_64/edge complete
                                          security: GPG: disabled
                                          security: SIGN: disabled http: TLS
                                          non-delta: meta: 98 content: 66
                                          transfer: secs: 4 size: 13.7 MB
May 12 14:54:32 vm-bios rpm-ostree[1074]: Created new deployment /ostree/deploy/edge-commit/deploy/4107a6f83a76d40fb045cfcfbcfae6c4128840cd4d7c5018f8643ced399f9c88.0
May 12 14:54:32 vm-bios rpm-ostree[1074]: sanitycheck(/usr/bin/true) successful
May 12 14:54:33 vm-bios rpm-ostree[1074]: **
May 12 14:54:33 vm-bios rpm-ostree[1074]: ERROR:src/lib/rpmostree-package.c:295:_rpm_ostree_package_list_for_commit: assertion failed (pkglist->len > 0): (0 > 0)
May 12 14:54:33 vm-bios systemd[1]: rpm-ostreed.service: Main process exited, code=killed, status=6/ABRT
May 12 14:54:33 vm-bios systemd[1]: rpm-ostreed.service: Failed with result 'signal'.

 [admin@vm-bios ~]$ coredumpctl info 1074
                   PID: 1074 (rpm-ostree)
                   UID: 0 (root)
                   GID: 0 (root)
                Signal: 6 (ABRT)
             Timestamp: Wed 2021-05-12 14:54:33 UTC (3min 3s ago)
          Command Line: /usr/bin/rpm-ostree start-daemon
            Executable: /usr/bin/rpm-ostree
         Control Group: /
                 Slice: -.slice
               Boot ID: bf95678384534218b03d95e06417d478
            Machine ID: 4e906e2d7c8b4cc78590f0196ce7fd85
              Hostname: vm-bios
               Storage: /var/lib/systemd/coredump/core.rpm-ostree.0.bf95678384534218b03d95e06417d478.1074.1620831273000000.lz4 (inaccessible)
               Message: Process 1074 (rpm-ostree) of user 0 dumped core.
         
                        Stack trace of thread 1077:
                        #0  0x00007f27ce52337f raise (libc.so.6)
                        #1  0x00007f27ce50ddb5 abort (libc.so.6)
                        #2  0x00007f27cf9130c3 g_assertion_message.cold.15 (libglib-2.0.so.0)
                        #3  0x00007f27cf96c4ba g_assertion_message_cmpnum (libglib-2.0.so.0)
                        #4  0x00007f27d1ca56cd _rpm_ostree_package_list_for_commit (librpmostree-1.so.1)
                        #5  0x00007f27d1ca4de2 rpm_ostree_db_diff_ext (librpmostree-1.so.1)
                        #6  0x00005561c976b4a0 rpm_diff_add_db_diff.constprop.20 (rpm-ostree)
                        #7  0x00005561c976d0a4 rpmostreed_update_generate_variant (rpm-ostree)
                        #8  0x00005561c9778c8a generate_update_variant (rpm-ostree)
                        #9  0x00005561c977d1e3 deploy_transaction_execute (rpm-ostree)
                        #10 0x00005561c976eacd transaction_execute_thread (rpm-ostree)
                        #11 0x00007f27cfef7367 g_task_thread_pool_thread (libgio-2.0.so.0)
                        #12 0x00007f27cf96d943 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                        #13 0x00007f27cf96cf3a g_thread_proxy (libglib-2.0.so.0)
                        #14 0x00007f27ce8b914a start_thread (libpthread.so.0)
                        #15 0x00007f27ce5e8dc3 __clone (libc.so.6)
         
                        Stack trace of thread 1074:
                        #0  0x00007f27ce5dda41 __poll (libc.so.6)
                        #1  0x00007f27cf944bf6 g_main_context_iterate.isra.21 (libglib-2.0.so.0)
                        #2  0x00007f27cf944d20 g_main_context_iteration (libglib-2.0.so.0)
                        #3  0x00005561c976745c rpmostreed_daemon_run_until_idle_exit (rpm-ostree)
                        #4  0x00005561c972d8d3 rpmostree_builtin_start_daemon (rpm-ostree)
                        #5  0x00005561c9716e0a main (rpm-ostree)
                        #6  0x00007f27ce50f493 __libc_start_main (libc.so.6)
                        #7  0x00005561c971717e _start (rpm-ostree)
         
                        Stack trace of thread 1075:
                        #0  0x00007f27ce5dda41 __poll (libc.so.6)
                        #1  0x00007f27cf944bf6 g_main_context_iterate.isra.21 (libglib-2.0.so.0)
                        #2  0x00007f27cf944d20 g_main_context_iteration (libglib-2.0.so.0)
                        #3  0x00007f27cf944d71 glib_worker_main (libglib-2.0.so.0)
                        #4  0x00007f27cf96cf3a g_thread_proxy (libglib-2.0.so.0)
                        #5  0x00007f27ce8b914a start_thread (libpthread.so.0)
                        #6  0x00007f27ce5e8dc3 __clone (libc.so.6)
         
                        Stack trace of thread 1076:
                        #0  0x00007f27ce5dda41 __poll (libc.so.6)
                        #1  0x00007f27cf944bf6 g_main_context_iterate.isra.21 (libglib-2.0.so.0)
                        #2  0x00007f27cf944fb2 g_main_loop_run (libglib-2.0.so.0)
                        #3  0x00007f27cff3a5aa gdbus_shared_thread_func (libgio-2.0.so.0)
                        #4  0x00007f27cf96cf3a g_thread_proxy (libglib-2.0.so.0)
                        #5  0x00007f27ce8b914a start_thread (libpthread.so.0)
                        #6  0x00007f27ce5e8dc3 __clone (libc.so.6)
         
                        Stack trace of thread 1088:
                        #0  0x00007f27ce5e352d syscall (libc.so.6)
                        #1  0x00007f27cf98bdae g_cond_wait_until (libglib-2.0.so.0)
                        #2  0x00007f27cf917131 g_async_queue_pop_intern_unlocked (libglib-2.0.so.0)
                        #3  0x00007f27cf917720 g_async_queue_timeout_pop (libglib-2.0.so.0)
                        #4  0x00007f27cf96d9e1 g_thread_pool_thread_proxy (libglib-2.0.so.0)
                        #5  0x00007f27cf96cf3a g_thread_proxy (libglib-2.0.so.0)
                        #6  0x00007f27ce8b914a start_thread (libpthread.so.0)
                        #7  0x00007f27ce5e8dc3 __clone (libc.so.6)


Version-Release number of selected component (if applicable):
rpm-ostree-libs-2020.7-4.el8.x86_64
ostree-2020.7-4.el8.x86_64
ostree-libs-2020.7-4.el8.x86_64
rpm-ostree-2020.7-4.el8.x86_64

libcurl-7.61.1-20.el8.x86_64
curl-7.61.1-20.el8.x86_64

How reproducible:

Steps to Reproduce:
1. Build an RHEL for Edge image (edge-commit) with image builder on RHEL 8.5
2. Setup a web servera as ostree repo
3. Extract edge-commit(tar) into /var/www/html/
4. Install ostree commit with kickstart on VM
5. Build an RHEL for Edge upgrade commit
6. Extract upgrade commit, pull and summary ostree repo
7. Run rpm-ostree upgrade on VM

Actual results:
⠠ Scanning metadata: 2310
Scanning metadata: 2310... done
Staging deployment... done
error: Bus owner changed, aborting. This likely means the daemon crashed; check logs with `journalctl -xe`.

Expected results:
upgrade successful

Additional info:

--- Additional comment from Jonathan Lebon on 2021-05-12 15:50:02 UTC ---

Since it's the same root issue, let's dedupe these two.
Comment 1 Jonathan Lebon 2021-06-24 15:06:35 UTC 
Given https://github.com/osbuild/osbuild-composer/pull/1498, it sounds like we can close this, right?
Comment 2 Colin Walters 2021-06-24 16:46:23 UTC 
I think we are good on this now yes.
Comment 3 Tom Gundersen 2021-06-24 16:56:21 UTC 
Note that the bug is still there, so building Fedora IoT images with Image Builder is still broken.

We had to disable our test for it as fixing it is out of our control and our development was therefore blocked.

Would love to see it fixed so we can reenable the tests, and most importantly restore the functionality.
Comment 4 Colin Walters 2021-06-25 11:57:27 UTC 
Hm, OK.  Any particular reason to still care about Fedora 33 instead of 34?

Anyways I'm building the new rpm-ostree for 33 too.
Comment 5 Tom Gundersen 2021-06-25 12:06:58 UTC 
Thanks Colin!

We try to make sure Image Builder works in Fedora until it is EOL, which is why we still test Fedora 33 in our CI (and release updates into, which is debatable, but makes our lives simpler). Ideally these tests would be part of Fedora gating, so we would not only catch regressions in other projects in our upstream CI, but we are not quite there yet.
Comment 6 Fedora Update System 2021-06-25 15:22:52 UTC 
FEDORA-2021-35a30ae214 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-35a30ae214
Comment 7 Fedora Update System 2021-06-26 01:51:24 UTC 
FEDORA-2021-35a30ae214 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-35a30ae214`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-35a30ae214

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 8 Xiaofeng Wang 2021-06-28 05:40:38 UTC 
Verifed on rpm-ostree-2021.6-2.fc33.x86_64 and rpm-ostree-libs-2021.6-2.fc33.x86_64.
Comment 9 Fedora Update System 2021-07-04 01:08:55 UTC 
FEDORA-2021-35a30ae214 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.