Bug 1900878 - The systemd upgrading from 246.6-3.fc34 to 247~rc2-1.fc34 version cause lost gnome and ssh sessions.
Summary: The systemd upgrading from 246.6-3.fc34 to 247~rc2-1.fc34 version cause lost ...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: gnome-shell
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Florian Müllner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1900735 (view as bug list)
Depends On:
Blocks: F34BetaBlocker F34BetaFreezeException
TreeView+ depends on / blocked
 
Reported: 2020-11-23 21:53 UTC by Mikhail
Modified: 2020-12-18 16:57 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-10 17:25:09 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
dnf-terminal-output (4.18 KB, text/plain)
2020-11-23 21:54 UTC, Mikhail
no flags Details
dnf.log (802.16 KB, text/plain)
2020-11-23 21:54 UTC, Mikhail
no flags Details
dnf.rpm.log (740.45 KB, text/plain)
2020-11-23 21:55 UTC, Mikhail
no flags Details
dnf.librepo.log (176.87 KB, text/plain)
2020-11-23 21:55 UTC, Mikhail
no flags Details
hawkey.log (12.04 KB, text/plain)
2020-11-23 21:56 UTC, Mikhail
no flags Details
system-log (11.99 KB, text/plain)
2020-11-23 21:57 UTC, Mikhail
no flags Details
photo of shut down process (1.91 MB, image/jpeg)
2020-12-13 22:00 UTC, Mikhail
no flags Details
system log (680.90 KB, text/plain)
2020-12-14 16:52 UTC, Mikhail
no flags Details
system log (523.93 KB, text/plain)
2020-12-18 09:59 UTC, Mikhail
no flags Details

Description Mikhail 2020-11-23 21:53:31 UTC
Description of problem:
The systemd upgrading from 246.6-3.fc34 to 247~rc2-1.fc34 version cause lost gnome and ssh sessions.
This is a very dangerous problem because it turns the system into an inconsistent state.
And also can potentially lead the system to an inoperative state.


How reproducible:


Steps to Reproduce:
1. upgrade systemd from 246.6-3.fc34 to 247~rc2-1.fc34 version

Demonstration: https://youtu.be/rILx1K5j4U4

Comment 1 Mikhail 2020-11-23 21:54:05 UTC
Created attachment 1732703 [details]
dnf-terminal-output

Comment 2 Mikhail 2020-11-23 21:54:25 UTC
Created attachment 1732704 [details]
dnf.log

Comment 3 Mikhail 2020-11-23 21:55:02 UTC
Created attachment 1732705 [details]
dnf.rpm.log

Comment 4 Mikhail 2020-11-23 21:55:27 UTC
Created attachment 1732706 [details]
dnf.librepo.log

Comment 5 Mikhail 2020-11-23 21:56:05 UTC
Created attachment 1732719 [details]
hawkey.log

Comment 6 Mikhail 2020-11-23 21:57:02 UTC
Created attachment 1732720 [details]
system-log

Comment 7 Fedora Blocker Bugs Application 2020-11-23 22:01:43 UTC
Proposed as a Blocker and Freeze Exception for 34-beta by Fedora user mikhail using the blocker tracking app because:

 This is a very dangerous problem because it turns the system into an inconsistent state.
And also can potentially lead the system to an inoperative state.
Blocker bug criteria: The system should be upgraded without any issues.

Comment 8 Zbigniew Jędrzejewski-Szmek 2020-11-23 22:14:52 UTC
Maybe https://github.com/systemd/systemd/issues/17605 ?

Comment 9 Zbigniew Jędrzejewski-Szmek 2020-11-27 13:39:20 UTC
This should be fixed with systemd-247-1. Please check.

Comment 10 Adam Williamson 2020-12-02 17:40:04 UTC
*** Bug 1900735 has been marked as a duplicate of this bug. ***

Comment 11 Zbigniew Jędrzejewski-Szmek 2020-12-10 17:25:09 UTC
There have been no further reports, and I expect that systemd-247.1 fixes the issue.
Please reopen if it still occurs.

Comment 12 Mikhail 2020-12-13 22:00:21 UTC
Created attachment 1738815 [details]
photo of shut down process

> There have been no further reports, and I expect that systemd-247.1 fixes the issue.
Yes the issue was gone but with new systemd the system began to shut down for a long time.

Comment 13 Zbigniew Jędrzejewski-Szmek 2020-12-14 08:17:07 UTC
(In reply to Mikhail from comment #12)
> Yes the issue was gone but with new systemd the system began to shut down
> for a long time.

It seems that the systemd user instance is not shutting down properly. The reason should
be visible in the logs.

Comment 14 Mikhail 2020-12-14 16:52:16 UTC
Created attachment 1739006 [details]
system log

> It seems that the systemd user instance is not shutting down properly. The reason should be visible in the logs.


Dec 14 21:28:33 localhost.localdomain systemd[2105]: xdg-permission-store.service: Main process exited, code=exited, status=1/FAILURE
Dec 14 21:28:33 localhost.localdomain systemd[2105]: xdg-permission-store.service: Failed with result 'exit-code'.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: dbus-:1.2-org.gnome.Shell.CalendarServer: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: gvfs-udisks2-volume-monitor.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: dbus-:1.2-org.gnome.Identity: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: dbus-:1.2-ca.desrt.dconf: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: gvfs-afc-volume-monitor.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[1]: run-user-1000-gvfs.mount: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: run-user-1000-gvfs.mount: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[1]: Unmounted /run/user/1000/gvfs.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: evolution-source-registry.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: gvfs-goa-volume-monitor.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: gvfs-gphoto2-volume-monitor.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: evolution-calendar-factory.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: evolution-addressbook-factory.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: dbus-:1.2-org.freedesktop.problems.applet: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: gnome-session-restart-dbus.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: gvfs-daemon.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: app-gnome-org.gnome.SettingsDaemon.DiskUtilityNotify-2649.scope: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Starting D-Bus User Message Bus...
Dec 14 21:28:33 localhost.localdomain systemd[2105]: dbus-:1.2-org.gnome.OnlineAccounts: Succeeded.
Dec 14 21:28:33 localhost.localdomain dbus-broker-launch[3355]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +31: Eavesdropping is deprecated and ignored
Dec 14 21:28:33 localhost.localdomain dbus-broker-launch[3355]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +33: Eavesdropping is deprecated and ignored
Dec 14 21:28:33 localhost.localdomain dbus-broker-lau[3355]: Ready
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Started D-Bus User Message Bus.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Started dbus-:1.1-ca.desrt.dconf.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: gvfs-metadata.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: org.gnome.Shell-disable-extensions.service: Succeeded.
Dec 14 21:30:09 localhost.localdomain systemd[2105]: Starting Mark boot as successful...
Dec 14 21:30:09 localhost.localdomain systemd[2105]: grub-boot-success.service: Succeeded.
Dec 14 21:30:09 localhost.localdomain systemd[2105]: Finished Mark boot as successful.
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: State 'stop-sigterm' timed out. Killing.
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: Killing process 2105 (systemd) with signal SIGKILL.
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: Killing process 3355 (dbus-broker-lau) with signal SIGKILL.
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: Killing process 3356 (dbus-broker) with signal SIGKILL.
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: Killing process 3358 (dconf-service) with signal SIGKILL.
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: Main process exited, code=killed, status=9/KILL
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: Failed with result 'timeout'.
Dec 14 21:30:33 localhost.localdomain systemd[1]: Stopped User Manager for UID 1000.
Dec 14 21:30:33 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user@1000 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Dec 14 21:30:33 localhost.localdomain systemd[1]: user: Consumed 14.485s CPU time.

Comment 15 Mikhail 2020-12-18 09:59:40 UTC
Created attachment 1740195 [details]
system log

Did you see why user: State 'stop-sigterm' is timed out?
I am look at the log and did not seen any suspisious.
Or needed logs with another verbose level?
I just want report that the problem still actual even with yesterday systemd 247.2-1

Comment 16 Zbigniew Jędrzejewski-Szmek 2020-12-18 14:04:31 UTC
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Stopping GNOME Shell on Wayland...
Dec 14 21:28:33 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=livesys comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 14 21:28:33 localhost.localdomain systemd[1]: fwupd.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[1]: Stopped Firmware update daemon.
Dec 14 21:28:33 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fwupd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 14 21:28:33 localhost.localdomain uresourced[1442]: Setting resources on user-1000.slice (MemoryMin: 0, MemoryLow: 0, CPUWeight: 100, IOWeight: 100)
Dec 14 21:28:33 localhost.localdomain uresourced[1442]: Setting resources on user (MemoryMin: 0, MemoryLow: 0, CPUWeight: 100, IOWeight: 100)
Dec 14 21:28:33 localhost.localdomain uresourced[1442]: Setting resources on user.slice (MemoryMin: 0, MemoryLow: 0, CPUWeight: -, IOWeight: -)
Dec 14 21:28:33 localhost.localdomain systemd[1]: Started Show Plymouth Reboot Screen.
Dec 14 21:28:33 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plymouth-reboot comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 14 21:28:33 localhost.localdomain systemd[1]: systemd-logind.service: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[1]: Stopped User Login Management.
Dec 14 21:28:33 localhost.localdomain polkitd[1149]: Unregistered Authentication Agent for unix-session:2 (system bus name :1.81, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Dec 14 21:28:33 localhost.localdomain systemd[2105]: org.gnome.Shell: Main process exited, code=killed, status=11/SEGV
Dec 14 21:28:33 localhost.localdomain systemd[2105]: dbus-:1.2-org.gnome.Shell.Notifications: Succeeded.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: org.gnome.Shell: Failed with result 'signal'.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Stopped GNOME Shell on Wayland.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: org.gnome.Shell: Triggering OnFailure= dependencies.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: org.gnome.Shell: Consumed 6.412s CPU time.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Stopped target GNOME Session Manager is ready.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Stopped target Tasks to be run before GNOME Session starts.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Stopped target Session services which should run early before the graphical session is brought up.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Reached target Shutdown running GNOME Session.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Starting Restart DBus after GNOME Session shutdown...
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Started Disable GNOME Shell extensions after failure.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Stopped target Shutdown running GNOME Session.
Dec 14 21:28:33 localhost.localdomain systemd[2105]: Started Restart DBus after GNOME Session shutdown.

It seems to be a problem with gnome-shell: it crashes and has:
OnFailure=org.gnome.Shell-disable-extensions.service gnome-session-shutdown.target
OnFailureJobMode=replace-irreversibly

This means that the shutdown transaction that systemd was executing gets replaced by the transaction
to start gnome-session-shutdown.target, which does not cause the user@ instance to exit and thus
leads to the observed timeout.

Let's reassign this to gnome-shell for comments.

Comment 17 Jonas Ådahl 2020-12-18 14:33:41 UTC
Can you attach the backtrace? I suspect it's a crash that should already have been fixed upstream.

Comment 18 Mikhail 2020-12-18 16:55:32 UTC
> It seems to be a problem with gnome-shell: it crashes and has:

How to find the backtrace of this crash?

# coredumpctl 
TIME                            PID   UID   GID SIG COREFILE  EXE
Wed 2020-12-09 11:57:31 +05    2216  1000  1000  11 missing   /usr/bin/su
Wed 2020-12-09 11:57:34 +05    2257  1000  1000  11 missing   /usr/bin/su
Wed 2020-12-09 13:14:08 +05    8554  1000  1000  11 missing   /usr/bin/su
Wed 2020-12-09 13:14:11 +05    8572  1000  1000  11 missing   /usr/bin/su
Wed 2020-12-09 13:51:30 +05   12066  1000  1000  11 missing   /usr/bin/su
Wed 2020-12-09 13:57:37 +05    2223  1000  1000  11 missing   /usr/bin/su
Wed 2020-12-09 15:21:52 +05    6745  1000  1000  11 missing   /usr/bin/su
Wed 2020-12-09 15:44:39 +05    8390  1000  1000  11 missing   /usr/bin/su
Thu 2020-12-10 14:04:23 +05   56556  1000  1000  11 missing   /usr/lib64/firefox/plugin-container

Comment 19 Jonas Ådahl 2020-12-18 16:57:16 UTC
If anywhere it should have been visible via coredumpctl, so I don't know where it went.


Note You need to log in before you can comment on or make changes to this bug.