Bug 1154416 - systemd: Forcibly powering off as result of failure.
Summary: systemd: Forcibly powering off as result of failure.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 21
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-19 14:34 UTC by Andre Robatino
Modified: 2015-01-16 06:19 UTC (History)
13 users (show)

Fixed In Version: systemd-216-7.fc21
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-01-16 06:19:27 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Andre Robatino 2014-10-19 14:34:59 UTC
Description of problem:
Recently, in both my F21 and Rawhide VirtualBox guests, I've had shutdowns, usually just after starting a yum distro-sync and just before the transaction starts, but once in the middle of the transaction. /var/log/messages shows "systemd: Forcibly powering off as result of failure." when this happens. See my thread https://lists.fedoraproject.org/pipermail/test/2014-October/123402.html . Presumably this is a timing issue caused by my hardware being slow, but it's not *that* slow (a 3 GHz E8400 Intel Core Duo, and each guest is using both cores).

Version-Release number of selected component (if applicable):
systemd-216-4.fc21.x86_64

How reproducible:
sometimes

Comment 1 Jan Synacek 2014-10-20 11:11:47 UTC
What exactly do you run before the poweroff happens? Do you have a custom service for that?

Comment 2 Zbigniew Jędrzejewski-Szmek 2014-10-20 12:59:52 UTC
It's probably the functionality to poweroff if default.target is not reached fast enough (10 minutes?). On the TODO list is changing that to only limit the time until basic.target.

Comment 3 Andre Robatino 2014-10-20 13:55:11 UTC
I have no custom service. Most days I just boot into GNOME, immediately do a yum distro-sync, then power down. The forced shutdown usually happens after I start the distro-sync but just before the transaction starts, after most or all of the drpms have finished building (although once it happened during the transaction, in F21).

Comment 4 Andre Robatino 2014-10-20 15:58:49 UTC
I think it just happened again (in F21), but this time, it was after a completed transaction, and there was no error message in /var/log/messages, so I can't be sure it was a forced shutdown. However, I was in graphical mode, and there were no orphan inodes cleaned up, so it probably was. /var/log/wtmp wasn't cleaned up either (both output lines in "last -2 reboot" say "still running", all output lines in "last reboot" except for the first two show a shutdown, so the earlier forced shutdowns did update wtmp).

Comment 5 Andre Robatino 2014-10-20 17:18:30 UTC
This bug may be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1152220 ?

Comment 6 Adam Williamson 2014-10-20 18:55:57 UTC
well, is initial-setup.service installed / enabled / running on your system?

Comment 7 Andre Robatino 2014-10-20 19:04:44 UTC
[root@localhost ~]# systemctl status initial-setup-
initial-setup-graphical.service  initial-setup-text.service
[root@localhost ~]# systemctl status initial-setup-graphical.service 
● initial-setup-graphical.service - Initial Setup configuration program
   Loaded: loaded (/usr/lib/systemd/system/initial-setup-graphical.service; disabled)
   Active: inactive (dead)
[root@localhost ~]# systemctl status initial-setup-text.service 
● initial-setup-text.service - Initial Setup configuration program (text mode)
   Loaded: loaded (/usr/lib/systemd/system/initial-setup-text.service; disabled)
   Active: inactive (dead)
[root@localhost ~]#

I only appear to have these two, not initial-setup.service.

Comment 8 Adam Williamson 2014-10-21 00:02:14 UTC
those are the relevant services indeed, and they're both disabled, so this doesn't appear to be a dupe of the i-s case at least.

Can you poke around with systemctl before the system powers down and see if you can spot what services are still attempting to start up?

Comment 9 Andre Robatino 2014-10-21 12:26:38 UTC
I think I can pretty reliably reproduce this by just booting in graphical mode and leaving it alone for around 10 minutes. The problem is that when it shuts down that way, the system logs will be truncated - I don't even get the "Forcibly powering off" message in /var/log/messages. I wouldn't be surprised if a lot more people are experiencing this and just attribute it to gremlins because there's nothing in the logs.

To debug this, I can try repeatedly running something like "systemctl --all status > foo.txt". Is there a better way - either getting systemd to say which failed service is making it shut down, or at least getting it to trigger a specified command prior to shutdown (such as the systemctl command) so I can get the output at the exact correct time?

Comment 10 Zbigniew Jędrzejewski-Szmek 2014-10-21 13:23:04 UTC
I'll disable the shutdown functionality on next update. When it's all ready and ironed out upstream than we can enable it in F22.

Comment 11 Andre Robatino 2014-10-21 13:31:06 UTC
I take back my claim that I could reliably reproduce it by just leaving the system alone for 10 minutes - that doesn't always happen. However, at least twice now I've seen it shut down while truncating the system logs so it would be almost impossible to track down.

Comment 12 Kamil Páral 2014-11-03 09:29:58 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #10)
> I'll disable the shutdown functionality on next update. When it's all ready
> and ironed out upstream than we can enable it in F22.

Just a note, please also consider GNOME's offline updates functionality. It would be very unfortunate if it got affected by this new feature as well. Rebooting during system update has a good chance of leaving the system totally broken. Thanks.

Comment 13 Zbigniew Jędrzejewski-Szmek 2014-11-03 12:36:01 UTC
(In reply to Kamil Páral from comment #12)
> Just a note, please also consider GNOME's offline updates functionality. It
> would be very unfortunate if it got affected by this new feature as well.
> Rebooting during system update has a good chance of leaving the system
> totally broken. Thanks.
systemd-216-7.fc21 update has the new upstream mechanism (JobTimeoutAction and related settings), but no timeouts are configured. So unless someone explicitly configures a timeout, they should be safe.

Comment 14 Rik van Riel 2014-11-03 18:14:24 UTC
Zbigniew, how will this feature interact with a system that does a periodic fsck on bootup, and has a very large / slow filesystem?

That automatic fsck could well take an hour on some systems, and having the system automatically power off during the fsck could turn an annoyingly long reboot into an essentially unrecoverable system.

Comment 15 Zbigniew Jędrzejewski-Szmek 2014-11-03 18:18:17 UTC
(In reply to Rik van Riel from comment #14)
> Zbigniew, how will this feature interact with a system that does a periodic
> fsck on bootup, and has a very large / slow filesystem?
Yes, that's another case. The timeout now is turned off (or will be when systemd is updated). I didn't want to patch out the functionality, to minimize the difference wrt. upstram, just all timeout are left unconfigured.

Comment 16 Zbigniew Jędrzejewski-Szmek 2015-01-16 06:19:27 UTC
This was fixed, but not properly closed.


Note You need to log in before you can comment on or make changes to this bug.