Recently, in openQA Rawhide testing, the test where we build a netinst then install from it has started to fail very frequently. The failure occurs when the install is complete and we hit the "Reboot System" button. The system never reboots; we just get stuck at a blank grey screen. Looking in the journal, I see messages indicating anaconda exit didn't work, and maybe that's why we're stuck: Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: DEBUG:anaconda.modules.boss.module_manager.module_manager:Stop modules. Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Modules.Services[2229]: DEBUG:dasbus.connection:Disconnecting from the bus. Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: DEBUG:anaconda.modules.boss.module_manager.module_manager:org.fedoraproject.Anaconda.Modules.Services has quit. Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Modules.Security[2235]: DEBUG:dasbus.connection:Disconnecting from the bus. Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: WARNING:dasbus.server.handler:The call org.fedoraproject.Anaconda.Boss.Quit has failed with an exception: Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: Traceback (most recent call last): Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib/python3.13/site-packages/dasbus/server/handler.py", line 455, in _method_callback Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: result = self._handle_call( Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: interface_name, Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ...<2 lines>... Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: **additional_args Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ) Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib/python3.13/site-packages/dasbus/server/handler.py", line 265, in _handle_call Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: return handler(*parameters, **additional_args) Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib64/python3.13/site-packages/pyanaconda/modules/boss/boss_interface.py", line 155, in Quit Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: self.implementation.stop() Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ~~~~~~~~~~~~~~~~~~~~~~~~^^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib64/python3.13/site-packages/pyanaconda/modules/boss/boss.py", line 76, in stop Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: self._module_manager.stop_modules() Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib64/python3.13/site-packages/pyanaconda/modules/boss/module_manager/module_manager.py", line 94, in stop_modules Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: observer.proxy.Quit() Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ~~~~~~~~~~~~~~~~~~~^^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib/python3.13/site-packages/dasbus/client/handler.py", line 450, in _call_method Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: return self._get_method_reply( Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ~~~~~~~~~~~~~~~~~~~~~~^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: self._client.sync_call, Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ^^^^^^^^^^^^^^^^^^^^^^^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: *args, Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ^^^^^^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: **kwargs, Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ^^^^^^^^^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ) Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib/python3.13/site-packages/dasbus/client/handler.py", line 483, in _get_method_reply Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: return self._handle_method_error(error) Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: File "/usr/lib/python3.13/site-packages/dasbus/client/handler.py", line 509, in _handle_method_error Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: raise exception from None Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com org.fedoraproject.Anaconda.Boss[2214]: dasbus.error.DBusError: Message recipient disconnected from message bus without replying Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com systemd[2354]: Received SIGHUP. Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com systemd[2354]: Reloading... Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com NetworkManager[2029]: <debug> [1732932902.5410] agent-manager: agent[69ab6569a9ac7558,:1.32/anaconda/0]: agent unregistered or disappeared Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com dbus-daemon[2211]: Reloaded configuration Nov 30 02:15:02 ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com systemd-logind[2022]: Session 1 logged out. Waiting for processes to exit. I'll attach the full journal, too. This seems to have started happening yesterday. The earliest occurrences are three on https://bodhi.fedoraproject.org/updates/FEDORA-2024-8947c15cd1 , pygobject3-3.50.0-1.fc42 - two on lab, one on prod. So that update *could* be the source of the problem. There's been a lot of weirdness in Rawhide lately, though, so I just restarted the tests without realizing this was a new problem, so it's now happening to all updates :/
Created attachment 2060434 [details] full journal of an affected install
Pinging to pygobject3 as that does seem like a likely suspect. anaconda-core does depend on python3-gobject-base which comes from that src.rpm.
Looks like this also caused today's Rawhide to fail: https://koji.fedoraproject.org/koji/taskinfo?taskID=126389014 . Look at the screenshot - https://kojipkgs.fedoraproject.org//work/tasks/9014/126389014/screenshot.png . It's also stuck on "Message recipient disconnected from message bus without replying". These installs are kickstart driven. When the kickstart finishes we run through the same exit handler as we do when you click Reboot after an interactive install, so this is the same thing.
OK, I think I bisected this to https://gitlab.gnome.org/GNOME/pygobject/-/commit/bf80d0f34af5062fd1201d71fbcd3a34bf1d1a80 . Just confirming that now.
For the record, this is currently worked around by reverting bf80d0f34af5062fd1201d71fbcd3a34bf1d1a80 , but we need a longer-term fix.
This is also reproducible by clicking 'Quit' button, while the installation is running. One does not need to wait for the installation to finish to click 'Reboot'.
Upstream fix available: https://github.com/rhinstaller/anaconda/pull/6059
Update: we still have the revert in pygobject3 at this point. I don't think anyone has fixed this 'properly' yet.
The anaconda patch did not fix the bug, so unmarking as POST.
I'm gonna unpropose this as a blocker, though, because it's not an active problem ATM as we have the pygobject3 revert.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 42 development cycle. Changing version to 42.
So, the reversion in pygobject3 package was removed in 3.52.3, some time ago: https://src.fedoraproject.org/rpms/pygobject3/c/d8b412c7e51acbae3644a0c785c3c7021795b159?branch=rawhide We have a workaround for this in openQA test code, but it should record a soft failure, and at a quick look, I'm not seeing any of those soft failures in recent F43 tests. So I *think* this somehow got resolved. I'm going to close it as WORKSFORME and drop the workaround from the openQA test code; if we start seeing failures again, we can re-open...
Ah, it looks like the change was reverted upstream, and has not since been restored: https://gitlab.gnome.org/GNOME/pygobject/-/commit/c958ef51598578b63c6055841ba51f5da786a8c1