Bug 816842 - reboot hangs with systemd-44-7.fc17
reboot hangs with systemd-44-7.fc17
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
AcceptedBlocker
:
: 816714 817182 817475 817857 (view as bug list)
Depends On:
Blocks: F17Blocker/F17FinalBlocker 817857 818381
  Show dependency treegraph
 
Reported: 2012-04-27 03:25 EDT by Joachim Backes
Modified: 2012-05-06 11:27 EDT (History)
27 users (show)

See Also:
Fixed In Version: systemd-44-8.fc17
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 818381 (view as bug list)
Environment:
Last Closed: 2012-05-04 19:04:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
systemctl dump output (378.88 KB, text/plain)
2012-04-27 19:37 EDT, Ed Greshko
no flags Details
Output of "systemctl dump" (461.64 KB, application/octet-stream)
2012-04-28 01:47 EDT, Joachim Backes
no flags Details

  None (edit)
Description Joachim Backes 2012-04-27 03:25:56 EDT
Description of problem:

after updating to systemd-44-7.fc17 (and systemd-sysv-44-7.fc17), 
rebooting hangs up. Pushing the reset button is needed :-(

I had to downgrade to systemd-44-6.fc17 to get rid from this bug.
Version-Release number of selected component (if applicable):
systemd-44-7.fc17

How reproducible:
always

Steps to Reproduce:
1./usr/bin/reboot
2.
3.
  
Actual results:
Reboot hangs up, no shutdown done

Expected results:
reboot proceeds

Additional info:
Comment 1 Ed Greshko 2012-04-27 03:42:24 EDT
I am basically seeing the same thing.  In my case the system appears hung for about 2 minutes but then will shutdown or restart.  During the "hang" the screen is blank but I do see some disk activity.
Comment 2 Pedro Francisco 2012-04-27 03:45:39 EDT
Using the Magic Keys + R,E,I,S,U,B does not force the reboot.
Comment 3 Pedro Francisco 2012-04-27 04:21:28 EDT
Sorry, my mistake. Here is what I meant: on random previous hangups previously Magic Keys + R,E,I would force the processes to terminate and then the reboot would happen without needing the S,U,B part.

Now, Magic Keys R,E,I do nothing -- they only close the processes, but the system does not reboot after (at least not without 2 mins going by).

The reboot does happen with Magic Keys R,E,I,S,U,B.
Comment 4 Joachim Backes 2012-04-27 11:56:58 EDT
I found out that the problem is dependent from the current runlevel: If I boot into runlevel 3, then a subsequent reboot is performed flawlessly. The reboot/shutdown hangs up if performed in runlevel 5.
Comment 5 Michal Schmidt 2012-04-27 19:00:28 EDT
Joachim,

have you tried waiting 2 minutes to confirm Ed's observation from comment #1?

Is it perfectly repeatable? I.e. does 44-6 always work and does 44-7 always hang?

Could you attach the output of "systemctl dump" saved before a failed reboot attempt?
Comment 6 Ed Greshko 2012-04-27 19:37:39 EDT
Created attachment 580891 [details]
systemctl dump output

Here is the output from systemctl dump just prior to a "shutdown" which took 2 minutes to take effect.
Comment 7 Joachim Backes 2012-04-28 01:45:49 EDT
(In reply to comment #5)
> Joachim,
> 
> have you tried waiting 2 minutes to confirm Ed's observation from comment #1?

Yes, I can confirm this.

> 
> Is it perfectly repeatable? I.e. does 44-6 always work and does 44-7 always
> hang?

In the meantime, it happens with 44-6 too (after performing "yum update" yesterday evening; this update included:

Apr 27 21:36:05 Updated: poppler-0.18.4-2.fc17.x86_64
Apr 27 21:36:06 Updated: poppler-glib-0.18.4-2.fc17.x86_64
Apr 27 21:36:06 Updated: libwpd-0.9.4-3.fc17.x86_64
Apr 27 21:36:07 Updated: qjson-0.7.1-6.fc17.x86_64
Apr 27 21:36:08 Updated: libmygpo-qt-1.0.5-3.fc17.x86_64
Apr 27 21:36:08 Updated: OpenEXR-libs-1.7.0-4.fc17.x86_64
Apr 27 21:36:12 Updated: qtscriptbindings-0.1.0-18.fc17.x86_64
Apr 27 21:36:13 Updated: exiv2-libs-0.22-5.fc17.x86_64
Apr 27 21:36:15 Updated: transmission-common-2.50-2.fc17.x86_64
Apr 27 21:36:16 Updated: wxBase-2.8.12-4.fc17.x86_64
Apr 27 21:36:16 Updated: 1:gmp-5.0.2-6.fc17.x86_64
Apr 27 21:36:17 Updated: ppl-pwl-0.11.2-8.fc17.x86_64
Apr 27 21:36:18 Updated: libopenraw-0.0.8-7.fc17.x86_64
Apr 27 21:36:19 Updated: amarok-utils-2.5.0-7.fc17.x86_64
Apr 27 21:36:20 Updated: amarok-libs-2.5.0-7.fc17.x86_64
Apr 27 21:36:25 Updated: amarok-2.5.0-7.fc17.x86_64
Apr 27 21:36:26 Updated: 1:aiksaurus-1.2.1-25.fc17.x86_64
Apr 27 21:36:26 Updated: 1:aiksaurus-gtk-1.2.1-25.fc17.x86_64
Apr 27 21:36:27 Updated: libopenraw-gnome-0.0.8-7.fc17.x86_64
Apr 27 21:36:28 Updated: ppl-0.11.2-8.fc17.x86_64
Apr 27 21:36:29 Updated: wxGTK-2.8.12-4.fc17.x86_64
Apr 27 21:36:32 Updated: transmission-gtk-2.50-2.fc17.x86_64
Apr 27 21:36:33 Updated: strigi-libs-0.7.7-3.fc17.x86_64
Apr 27 21:36:35 Updated: opencv-2.3.1-8.fc17.x86_64
Apr 27 21:36:36 Updated: libwps-0.2.4-3.fc17.x86_64
Apr 27 21:36:36 Updated: libwpg-0.2.0-4.fc17.x86_64
Apr 27 21:36:38 Updated: tracker-0.14.0-2.fc17.x86_64
Apr 27 21:36:39 Updated: poppler-utils-0.18.4-2.fc17.x86_64
Apr 27 21:36:40 Updated: cryptopp-5.6.1-7.fc17.x86_64
Apr 27 21:36:40 Updated: xorg-x11-drv-vesa-2.3.1-1.fc17.x86_64
Apr 27 21:36:41 Updated: xorg-x11-drv-cirrus-1.4.0-1.fc17.x86_64
Apr 27 21:36:41 Updated: libmodman-2.0.1-4.fc17.x86_64
Apr 27 21:36:42 Updated: libssh2-1.4.1-2.fc17.x86_64
Apr 27 21:36:44 Updated: source-highlight-3.1.4-10.fc17.x86_64
Apr 27 21:36:46 Updated: iptstate-2.2.2-7.fc17.x86_64
Apr 27 21:36:47 Updated: libsidplay-1.36.60-5.fc17.x86_64
Apr 27 21:36:47 Updated: cairomm-1.10.0-4.fc17.x86_64
Apr 27 21:36:48 Updated: libxml++-2.34.2-3.fc17.x86_64
Apr 27 21:36:51 Updated: libprojectM-2.0.1-14.fc17.x86_64
Apr 27 21:36:53 Updated: 4:htdig-3.2.0-0.13.b6.fc17.x86_64
Apr 27 21:36:54 Updated: vamp-plugin-sdk-2.2.1-3.fc17.x86_64
Apr 27 21:36:55 Updated: xorg-x11-drv-synaptics-1.5.99.904-1.fc17.x86_64
Apr 27 21:36:55 Updated: xorg-x11-xinit-1.3.2-5.fc17.x86_64
Apr 27 21:36:57 Updated: libfreebob-1.0.11-11.fc17.x86_64
Apr 27 21:36:57 Updated: cdrdao-1.2.3-13.fc17.x86_64
Apr 27 21:36:58 Updated: kasumi-2.5-7.fc17.x86_64
Apr 27 21:36:58 Updated: gupnp-0.18.2-2.fc17.x86_64
Apr 27 21:36:59 Updated: dirac-libs-1.0.2-8.fc17.x86_64
Apr 27 21:37:01 Updated: setroubleshoot-plugins-3.0.38-1.fc17.noarch
Apr 27 21:37:01 Updated: LibRaw-0.14.3-4.fc17.x86_64
Apr 27 21:37:03 Updated: ttmkfdir-3.0.9-35.fc17.x86_64
Apr 27 21:37:04 Updated: libcmis-0.1.0-2.fc17.x86_64
Apr 27 21:37:05 Updated: compat-libstdc++-33-3.2.3-68.3.i686
)
> 
> Could you attach the output of "systemctl dump" saved before a failed reboot
> attempt?

See the following attchment!
Comment 8 Joachim Backes 2012-04-28 01:47:39 EDT
Created attachment 580932 [details]
Output of "systemctl dump"
Comment 9 Joachim Backes 2012-04-28 13:30:59 EDT
I (In reply to comment #3)
> Sorry, my mistake. Here is what I meant: on random previous hangups previously
> Magic Keys + R,E,I would force the processes to terminate and then the reboot
> would happen without needing the S,U,B part.
> 
> Now, Magic Keys R,E,I do nothing -- they only close the processes, but the
> system does not reboot after (at least not without 2 mins going by).
> 
> The reboot does happen with Magic Keys R,E,I,S,U,B.

By pressing Ctrl+Alt+Del inside a gnome session and then pressing the "Restart" pushbutton (and not /usr/bin/reboot inside a graphical terminal window (gnome-terminal for example)), my box boots immediately :-)
Comment 10 Josh Boyer 2012-04-30 11:04:10 EDT
*** Bug 817475 has been marked as a duplicate of this bug. ***
Comment 11 Jacek Pliszka 2012-05-01 03:35:00 EDT
In my case Joachim's way hangs as well. (Ctrl-Alt-Del  an then poweroff).

However the following way shuts down fast:

telinit 3
poweroff

while poweroff directly from runlevel 5 hangs.

Is there an easy way to compare shut down sequence in both cases ?
Comment 12 Michal Schmidt 2012-05-01 12:49:18 EDT
I have an explanation for the long shutdown. It is a regression caused by commit ecedd90 "service: place control command in subcgroup control/" that I backported to systemd-44-6.fc17 as a fix for blocker bug 805942.

To reproduce it, in the system there must be a service with these properties:
 - It has at least one ExecStartPre command defined (to cause the "control/"
   subgroup to be created).
 - Its main process spawns at least one child process.

When stopping such a service, the following can happen:
1. systemd sends SIGTERM to all the processes of the service.
2. The main process exits first (it is a race).
3. systemd checks the status of the cgroup. It still sees some live processes.
4. The remaining processes exit.
5. At this point systemd expects to receive a notification from
   systemd-cgroups-agent. The notification never arrives though, because the
   cgroup is not really empty - the existing "control/" subdirectory (with no
   tasks in it) is enough to make it non-empty.

dbus.service is often the actual unit that triggers it. For testing a simpler unit can be used - shutdownproblem.service:


[Unit]
Description=shutdown problem

[Service]
ExecStartPre=/bin/true
ExecStart=/bin/sh -c 'a(){ trap "sleep 3; exit 0" TERM; sleep 3600; }; a & sleep 3600'
Comment 13 Michal Schmidt 2012-05-01 12:53:54 EDT
*** Bug 816714 has been marked as a duplicate of this bug. ***
Comment 14 Tom Horsley 2012-05-01 19:11:14 EDT
I'm seeing this as well in my f17 partition. In my case if I first stop the
"network" service (I'm not using NetworkManager), I can shut down as fast
as I'd expect to (about 7 seconds for me). If I leave the network service
running and try to reboot, there is about an extra 90 second delay:

http://lists.fedoraproject.org/pipermail/test/2012-May/107663.html
Comment 15 Rex Dieter 2012-05-02 08:06:31 EDT
nominating as blocker per,
* All release-blocking desktops' offered mechanisms (if any) for shutting down, logging out and rebooting must work
Comment 16 Rex Dieter 2012-05-02 08:11:27 EDT
*** Bug 817857 has been marked as a duplicate of this bug. ***
Comment 17 Michal Schmidt 2012-05-02 09:04:49 EDT
I agree with this being a blocker.

I'll let Lennart come up with a proper fix.

In the meantime (for F17 GA) I'll revert commit ecedd90 "service: place control command in subcgroup control/" from F17.
To avoid bug 805942 I will also apply a revert of 8f53a7b "service: brutally slaughter processes that are running in the cgroup when we enter START_PRE and START".
Comment 18 Fedora Update System 2012-05-02 18:08:19 EDT
systemd-44-8.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/systemd-44-8.fc17
Comment 19 Joachim Backes 2012-05-02 23:58:01 EDT
(In reply to comment #18)
> systemd-44-8.fc17 has been submitted as an update for Fedora 17.
> https://admin.fedoraproject.org/updates/systemd-44-8.fc17

After having installed systemd-44-8.fc17, /usr/bin/reboot reboots immediately :-(
Comment 20 Joachim Backes 2012-05-03 01:34:44 EDT
(In reply to comment #19)
> (In reply to comment #18)
> > systemd-44-8.fc17 has been submitted as an update for Fedora 17.
> > https://admin.fedoraproject.org/updates/systemd-44-8.fc17
> 
> After having installed systemd-44-8.fc17, /usr/bin/reboot reboots immediately
> :-)

Crud: running "yum update" after systemd update brings back the bug! I don't know which component is the culprit:

Packages updated:

May 03 06:54:04 Updated: libgcc-4.7.0-3.fc17.x86_64
May 03 06:54:17 Updated: glibc-common-2.15-35.fc17.x86_64
May 03 06:54:21 Updated: glibc-2.15-35.fc17.x86_64
May 03 06:54:21 Updated: wine-common-1.5.3-1.fc17.noarch
May 03 06:54:31 Updated: wine-core-1.5.3-1.fc17.x86_64
May 03 06:54:32 Updated: libstdc++-4.7.0-3.fc17.x86_64
May 03 06:54:32 Updated: libgomp-4.7.0-3.fc17.x86_64
May 03 06:54:33 Updated: nss-3.13.4-3.fc17.x86_64
May 03 06:54:34 Updated: nss-sysinit-3.13.4-3.fc17.x86_64
May 03 06:54:34 Updated: 1:NetworkManager-glib-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:54:40 Updated: libgcj-4.7.0-3.fc17.x86_64
May 03 06:54:53 Updated: libgcj-devel-4.7.0-3.fc17.x86_64
May 03 06:54:53 Updated: 1:NetworkManager-gtk-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:54:56 Updated: libstdc++-devel-4.7.0-3.fc17.x86_64
May 03 06:54:57 Updated: wine-capi-1.5.3-1.fc17.x86_64
May 03 06:54:57 Updated: wine-ldap-1.5.3-1.fc17.x86_64
May 03 06:54:58 Updated: wine-twain-1.5.3-1.fc17.x86_64
May 03 06:54:59 Updated: wine-openal-1.5.3-1.fc17.x86_64
May 03 06:54:59 Updated: wine-cms-1.5.3-1.fc17.x86_64
May 03 06:55:00 Updated: wine-wow-1.5.3-1.fc17.x86_64
May 03 06:55:00 Updated: wine-alsa-1.5.3-1.fc17.x86_64
May 03 06:55:00 Updated: wine-pulseaudio-1.5.3-1.fc17.x86_64
May 03 06:55:01 Updated: 1:wpa_supplicant-1.0-0.4.fc17.x86_64
May 03 06:55:05 Updated: 1:NetworkManager-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:55:06 Updated: cpp-4.7.0-3.fc17.x86_64
May 03 06:55:09 Updated: glibc-headers-2.15-35.fc17.x86_64
May 03 06:55:10 Updated: glibc-devel-2.15-35.fc17.x86_64
May 03 06:55:13 Updated: gcc-4.7.0-3.fc17.x86_64
May 03 06:55:13 Updated: libquadmath-4.7.0-3.fc17.x86_64
May 03 06:55:14 Updated: wine-ms-sans-serif-fonts-1.5.3-1.fc17.noarch
May 03 06:55:14 Updated: wine-systemd-1.5.3-1.fc17.noarch
May 03 06:55:15 Updated: wine-small-fonts-1.5.3-1.fc17.noarch
May 03 06:55:15 Updated: wine-courier-fonts-1.5.3-1.fc17.noarch
May 03 06:55:16 Updated: wine-marlett-fonts-1.5.3-1.fc17.noarch
May 03 06:55:17 Updated: libwacom-data-0.5-1.fc17.noarch
May 03 06:55:17 Updated: wine-system-fonts-1.5.3-1.fc17.noarch
May 03 06:55:18 Updated: wine-symbol-fonts-1.5.3-1.fc17.noarch
May 03 06:55:18 Updated: wine-tahoma-fonts-1.5.3-1.fc17.noarch
May 03 06:55:19 Updated: wine-fonts-1.5.3-1.fc17.noarch
May 03 06:55:20 Updated: hplip-common-3.12.4-2.fc17.x86_64
May 03 06:55:21 Updated: hplip-libs-3.12.4-2.fc17.x86_64
May 03 06:55:24 Updated: 1:hpijs-3.12.4-2.fc17.x86_64
May 03 06:55:27 Updated: hplip-3.12.4-2.fc17.x86_64
May 03 06:55:27 Updated: libsane-hpaio-3.12.4-2.fc17.x86_64
May 03 06:55:28 Updated: libwacom-0.5-1.fc17.x86_64
May 03 06:55:29 Updated: libgfortran-4.7.0-3.fc17.x86_64
May 03 06:55:30 Updated: gcc-java-4.7.0-3.fc17.x86_64
May 03 06:55:32 Updated: gcc-c++-4.7.0-3.fc17.x86_64
May 03 06:55:44 Updated: 1:NetworkManager-gnome-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:55:45 Updated: nss-tools-3.13.4-3.fc17.x86_64
May 03 06:55:47 Updated: shotwell-0.12.2-2.fc17.x86_64
May 03 06:55:52 Updated: wine-devel-1.5.3-1.fc17.x86_64
May 03 06:55:53 Updated: mdadm-3.2.3-9.fc17.x86_64
May 03 06:56:08 Updated: man-db-2.6.0.2-6.fc17.x86_64
May 03 06:56:09 Updated: fedora-logos-17.0.1-1.fc17.noarch
May 03 06:56:10 Updated: wine-sysvinit-1.5.3-1.fc17.noarch
May 03 06:56:13 Updated: glibc-2.15-35.fc17.i686
May 03 06:56:14 Updated: libgcc-4.7.0-3.fc17.i686
May 03 06:56:24 Updated: wine-core-1.5.3-1.fc17.i686
May 03 06:56:25 Updated: wine-desktop-1.5.3-1.fc17.noarch
May 03 06:56:25 Updated: wine-ldap-1.5.3-1.fc17.i686
May 03 06:56:26 Updated: wine-cms-1.5.3-1.fc17.i686
May 03 06:56:26 Updated: wine-openal-1.5.3-1.fc17.i686
May 03 06:56:27 Updated: wine-alsa-1.5.3-1.fc17.i686
May 03 06:56:28 Updated: wine-pulseaudio-1.5.3-1.fc17.i686
May 03 06:56:29 Updated: wine-capi-1.5.3-1.fc17.i686
May 03 06:56:29 Updated: wine-twain-1.5.3-1.fc17.i686
May 03 06:56:29 Updated: wine-1.5.3-1.fc17.x86_64
May 03 06:56:30 Updated: libstdc++-4.7.0-3.fc17.i686
May 03 06:56:31 Updated: nss-3.13.4-3.fc17.i686
Comment 21 Joachim Backes 2012-05-03 01:55:52 EDT
Additionally, I can say that the problem has deteriorated: Even going from runlevel 5 into runlevel 3 and then rebooting (was previously a workaround) no more helps. 

Even if booting directly to runlevel 3 and the trying to reboot shows the delay!
Comment 22 Jacek Pliszka 2012-05-03 03:04:41 EDT
I confirm Joachim's comment - something in the recent updates extended the problem to runlevel 3 as well.
Comment 23 Michal Schmidt 2012-05-03 03:38:40 EDT
I blame the NetworkManager update. I already noted that in the systemd update description:
https://admin.fedoraproject.org/updates/systemd-44-8.fc17
See also https://bugzilla.redhat.com/show_bug.cgi?id=739836#c61
Comment 24 Fedora Update System 2012-05-03 11:59:07 EDT
Package systemd-44-8.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-44-8.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-7193/systemd-44-8.fc17
then log in and leave karma (feedback).
Comment 25 Michal Schmidt 2012-05-04 07:11:26 EDT
*** Bug 817182 has been marked as a duplicate of this bug. ***
Comment 26 Adam Williamson 2012-05-04 13:10:26 EDT
Discussed at 2012-05-03 blocker review meeting. Accepted as a blocker per criterion "All release-blocking desktops' offered mechanisms (if any) for shutting down, logging out and rebooting must work". Note that 44-8 seems to fix it.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 27 Fedora Update System 2012-05-04 19:04:20 EDT
systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 28 Steve Gage 2012-05-06 08:02:53 EDT
(In reply to comment #27)
> systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If
> problems still persist, please make note of it in this bug report.

systemd-44-8.fc17 did not fix the problem for me. 

I am running FC17 32-bit fully updated as of 1130 UTC May 6.
Comment 29 Michal Schmidt 2012-05-06 10:38:16 EDT
(In reply to comment #28)
> systemd-44-8.fc17 did not fix the problem for me. 
> 
> I am running FC17 32-bit fully updated as of 1130 UTC May 6.

Make sure you try "yum distro-sync" as well to perform downgrades for the cases where packages were removed from updates-testing.
If the problem persists, please file a new bug.
Comment 30 Sergio Monteiro Basto 2012-05-06 11:18:51 EDT
(In reply to comment #28)
> (In reply to comment #27)
> > systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If
> > problems still persist, please make note of it in this bug report.
> 
> systemd-44-8.fc17 did not fix the problem for me. 
> 
> I am running FC17 32-bit fully updated as of 1130 UTC May 6.

have you got kernel 3.3.4-3.fc17  also fixes some stuck on kernel 3.3
Comment 31 Sergio Monteiro Basto 2012-05-06 11:27:19 EDT
(In reply to comment #30)
> (In reply to comment #28)
> > (In reply to comment #27)
> > > systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If
> > > problems still persist, please make note of it in this bug report.
> > 
> > systemd-44-8.fc17 did not fix the problem for me. 
> > 
> > I am running FC17 32-bit fully updated as of 1130 UTC May 6.
> 
> have you got kernel 3.3.4-3.fc17  also fixes some stuck on kernel 3.3

also take a look at bug 806548 and bug 739836

Note You need to log in before you can comment on or make changes to this bug.