Bug 816842

Summary: reboot hangs with systemd-44-7.fc17
Product: [Fedora] Fedora Reporter: Joachim Backes <joachim.backes>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 17CC: alekcejk, antonio.montagnani, aros, awilliam, chepioq, ed.greshko, horsley1953, Jacek.Pliszka, johannbg, mefoster, metherid, mishu, mschmidt, notting, pedrogfrancisco, piotrdrag, plautrba, rdieter, robatino, rolle.hoffmann, satellit, sergio, sgage, sven, systemd-maint, vpvainio, w_pirker
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedBlocker
Fixed In Version: systemd-44-8.fc17 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 818381 (view as bug list) Environment:
Last Closed: 2012-05-04 23:04:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 752650, 817857, 818381    
Attachments:
Description Flags
systemctl dump output
none
Output of "systemctl dump" none

Description Joachim Backes 2012-04-27 07:25:56 UTC 
Description of problem:

after updating to systemd-44-7.fc17 (and systemd-sysv-44-7.fc17), 
rebooting hangs up. Pushing the reset button is needed :-(

I had to downgrade to systemd-44-6.fc17 to get rid from this bug.
Version-Release number of selected component (if applicable):
systemd-44-7.fc17

How reproducible:
always

Steps to Reproduce:
1./usr/bin/reboot
2.
3.
  
Actual results:
Reboot hangs up, no shutdown done

Expected results:
reboot proceeds

Additional info:
Comment 1 Ed Greshko 2012-04-27 07:42:24 UTC 
I am basically seeing the same thing.  In my case the system appears hung for about 2 minutes but then will shutdown or restart.  During the "hang" the screen is blank but I do see some disk activity.
Comment 2 Pedro Francisco 2012-04-27 07:45:39 UTC 
Using the Magic Keys + R,E,I,S,U,B does not force the reboot.
Comment 3 Pedro Francisco 2012-04-27 08:21:28 UTC 
Sorry, my mistake. Here is what I meant: on random previous hangups previously Magic Keys + R,E,I would force the processes to terminate and then the reboot would happen without needing the S,U,B part.

Now, Magic Keys R,E,I do nothing -- they only close the processes, but the system does not reboot after (at least not without 2 mins going by).

The reboot does happen with Magic Keys R,E,I,S,U,B.
Comment 4 Joachim Backes 2012-04-27 15:56:58 UTC 
I found out that the problem is dependent from the current runlevel: If I boot into runlevel 3, then a subsequent reboot is performed flawlessly. The reboot/shutdown hangs up if performed in runlevel 5.
Comment 5 Michal Schmidt 2012-04-27 23:00:28 UTC 
Joachim,

have you tried waiting 2 minutes to confirm Ed's observation from comment #1?

Is it perfectly repeatable? I.e. does 44-6 always work and does 44-7 always hang?

Could you attach the output of "systemctl dump" saved before a failed reboot attempt?
Comment 6 Ed Greshko 2012-04-27 23:37:39 UTC 
Created attachment 580891 [details]
systemctl dump output

Here is the output from systemctl dump just prior to a "shutdown" which took 2 minutes to take effect.
Comment 7 Joachim Backes 2012-04-28 05:45:49 UTC 
(In reply to comment #5)
> Joachim,
> 
> have you tried waiting 2 minutes to confirm Ed's observation from comment #1?

Yes, I can confirm this.

> 
> Is it perfectly repeatable? I.e. does 44-6 always work and does 44-7 always
> hang?

In the meantime, it happens with 44-6 too (after performing "yum update" yesterday evening; this update included:

Apr 27 21:36:05 Updated: poppler-0.18.4-2.fc17.x86_64
Apr 27 21:36:06 Updated: poppler-glib-0.18.4-2.fc17.x86_64
Apr 27 21:36:06 Updated: libwpd-0.9.4-3.fc17.x86_64
Apr 27 21:36:07 Updated: qjson-0.7.1-6.fc17.x86_64
Apr 27 21:36:08 Updated: libmygpo-qt-1.0.5-3.fc17.x86_64
Apr 27 21:36:08 Updated: OpenEXR-libs-1.7.0-4.fc17.x86_64
Apr 27 21:36:12 Updated: qtscriptbindings-0.1.0-18.fc17.x86_64
Apr 27 21:36:13 Updated: exiv2-libs-0.22-5.fc17.x86_64
Apr 27 21:36:15 Updated: transmission-common-2.50-2.fc17.x86_64
Apr 27 21:36:16 Updated: wxBase-2.8.12-4.fc17.x86_64
Apr 27 21:36:16 Updated: 1:gmp-5.0.2-6.fc17.x86_64
Apr 27 21:36:17 Updated: ppl-pwl-0.11.2-8.fc17.x86_64
Apr 27 21:36:18 Updated: libopenraw-0.0.8-7.fc17.x86_64
Apr 27 21:36:19 Updated: amarok-utils-2.5.0-7.fc17.x86_64
Apr 27 21:36:20 Updated: amarok-libs-2.5.0-7.fc17.x86_64
Apr 27 21:36:25 Updated: amarok-2.5.0-7.fc17.x86_64
Apr 27 21:36:26 Updated: 1:aiksaurus-1.2.1-25.fc17.x86_64
Apr 27 21:36:26 Updated: 1:aiksaurus-gtk-1.2.1-25.fc17.x86_64
Apr 27 21:36:27 Updated: libopenraw-gnome-0.0.8-7.fc17.x86_64
Apr 27 21:36:28 Updated: ppl-0.11.2-8.fc17.x86_64
Apr 27 21:36:29 Updated: wxGTK-2.8.12-4.fc17.x86_64
Apr 27 21:36:32 Updated: transmission-gtk-2.50-2.fc17.x86_64
Apr 27 21:36:33 Updated: strigi-libs-0.7.7-3.fc17.x86_64
Apr 27 21:36:35 Updated: opencv-2.3.1-8.fc17.x86_64
Apr 27 21:36:36 Updated: libwps-0.2.4-3.fc17.x86_64
Apr 27 21:36:36 Updated: libwpg-0.2.0-4.fc17.x86_64
Apr 27 21:36:38 Updated: tracker-0.14.0-2.fc17.x86_64
Apr 27 21:36:39 Updated: poppler-utils-0.18.4-2.fc17.x86_64
Apr 27 21:36:40 Updated: cryptopp-5.6.1-7.fc17.x86_64
Apr 27 21:36:40 Updated: xorg-x11-drv-vesa-2.3.1-1.fc17.x86_64
Apr 27 21:36:41 Updated: xorg-x11-drv-cirrus-1.4.0-1.fc17.x86_64
Apr 27 21:36:41 Updated: libmodman-2.0.1-4.fc17.x86_64
Apr 27 21:36:42 Updated: libssh2-1.4.1-2.fc17.x86_64
Apr 27 21:36:44 Updated: source-highlight-3.1.4-10.fc17.x86_64
Apr 27 21:36:46 Updated: iptstate-2.2.2-7.fc17.x86_64
Apr 27 21:36:47 Updated: libsidplay-1.36.60-5.fc17.x86_64
Apr 27 21:36:47 Updated: cairomm-1.10.0-4.fc17.x86_64
Apr 27 21:36:48 Updated: libxml++-2.34.2-3.fc17.x86_64
Apr 27 21:36:51 Updated: libprojectM-2.0.1-14.fc17.x86_64
Apr 27 21:36:53 Updated: 4:htdig-3.2.0-0.13.b6.fc17.x86_64
Apr 27 21:36:54 Updated: vamp-plugin-sdk-2.2.1-3.fc17.x86_64
Apr 27 21:36:55 Updated: xorg-x11-drv-synaptics-1.5.99.904-1.fc17.x86_64
Apr 27 21:36:55 Updated: xorg-x11-xinit-1.3.2-5.fc17.x86_64
Apr 27 21:36:57 Updated: libfreebob-1.0.11-11.fc17.x86_64
Apr 27 21:36:57 Updated: cdrdao-1.2.3-13.fc17.x86_64
Apr 27 21:36:58 Updated: kasumi-2.5-7.fc17.x86_64
Apr 27 21:36:58 Updated: gupnp-0.18.2-2.fc17.x86_64
Apr 27 21:36:59 Updated: dirac-libs-1.0.2-8.fc17.x86_64
Apr 27 21:37:01 Updated: setroubleshoot-plugins-3.0.38-1.fc17.noarch
Apr 27 21:37:01 Updated: LibRaw-0.14.3-4.fc17.x86_64
Apr 27 21:37:03 Updated: ttmkfdir-3.0.9-35.fc17.x86_64
Apr 27 21:37:04 Updated: libcmis-0.1.0-2.fc17.x86_64
Apr 27 21:37:05 Updated: compat-libstdc++-33-3.2.3-68.3.i686
)
> 
> Could you attach the output of "systemctl dump" saved before a failed reboot
> attempt?

See the following attchment!
Comment 8 Joachim Backes 2012-04-28 05:47:39 UTC 
Created attachment 580932 [details]
Output of "systemctl dump"
Comment 9 Joachim Backes 2012-04-28 17:30:59 UTC 
I (In reply to comment #3)
> Sorry, my mistake. Here is what I meant: on random previous hangups previously
> Magic Keys + R,E,I would force the processes to terminate and then the reboot
> would happen without needing the S,U,B part.
> 
> Now, Magic Keys R,E,I do nothing -- they only close the processes, but the
> system does not reboot after (at least not without 2 mins going by).
> 
> The reboot does happen with Magic Keys R,E,I,S,U,B.

By pressing Ctrl+Alt+Del inside a gnome session and then pressing the "Restart" pushbutton (and not /usr/bin/reboot inside a graphical terminal window (gnome-terminal for example)), my box boots immediately :-)
Comment 10 Josh Boyer 2012-04-30 15:04:10 UTC 
*** Bug 817475 has been marked as a duplicate of this bug. ***
Comment 11 Jacek Pliszka 2012-05-01 07:35:00 UTC 
In my case Joachim's way hangs as well. (Ctrl-Alt-Del  an then poweroff).

However the following way shuts down fast:

telinit 3
poweroff

while poweroff directly from runlevel 5 hangs.

Is there an easy way to compare shut down sequence in both cases ?
Comment 12 Michal Schmidt 2012-05-01 16:49:18 UTC 
I have an explanation for the long shutdown. It is a regression caused by commit ecedd90 "service: place control command in subcgroup control/" that I backported to systemd-44-6.fc17 as a fix for blocker bug 805942.

To reproduce it, in the system there must be a service with these properties:
 - It has at least one ExecStartPre command defined (to cause the "control/"
   subgroup to be created).
 - Its main process spawns at least one child process.

When stopping such a service, the following can happen:
1. systemd sends SIGTERM to all the processes of the service.
2. The main process exits first (it is a race).
3. systemd checks the status of the cgroup. It still sees some live processes.
4. The remaining processes exit.
5. At this point systemd expects to receive a notification from
   systemd-cgroups-agent. The notification never arrives though, because the
   cgroup is not really empty - the existing "control/" subdirectory (with no
   tasks in it) is enough to make it non-empty.

dbus.service is often the actual unit that triggers it. For testing a simpler unit can be used - shutdownproblem.service:


[Unit]
Description=shutdown problem

[Service]
ExecStartPre=/bin/true
ExecStart=/bin/sh -c 'a(){ trap "sleep 3; exit 0" TERM; sleep 3600; }; a & sleep 3600'
Comment 13 Michal Schmidt 2012-05-01 16:53:54 UTC 
*** Bug 816714 has been marked as a duplicate of this bug. ***
Comment 14 Tom Horsley 2012-05-01 23:11:14 UTC 
I'm seeing this as well in my f17 partition. In my case if I first stop the
"network" service (I'm not using NetworkManager), I can shut down as fast
as I'd expect to (about 7 seconds for me). If I leave the network service
running and try to reboot, there is about an extra 90 second delay:

http://lists.fedoraproject.org/pipermail/test/2012-May/107663.html
Comment 15 Rex Dieter 2012-05-02 12:06:31 UTC 
nominating as blocker per,
* All release-blocking desktops' offered mechanisms (if any) for shutting down, logging out and rebooting must work
Comment 16 Rex Dieter 2012-05-02 12:11:27 UTC 
*** Bug 817857 has been marked as a duplicate of this bug. ***
Comment 17 Michal Schmidt 2012-05-02 13:04:49 UTC 
I agree with this being a blocker.

I'll let Lennart come up with a proper fix.

In the meantime (for F17 GA) I'll revert commit ecedd90 "service: place control command in subcgroup control/" from F17.
To avoid bug 805942 I will also apply a revert of 8f53a7b "service: brutally slaughter processes that are running in the cgroup when we enter START_PRE and START".
Comment 18 Fedora Update System 2012-05-02 22:08:19 UTC 
systemd-44-8.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/systemd-44-8.fc17
Comment 19 Joachim Backes 2012-05-03 03:58:01 UTC 
(In reply to comment #18)
> systemd-44-8.fc17 has been submitted as an update for Fedora 17.
> https://admin.fedoraproject.org/updates/systemd-44-8.fc17

After having installed systemd-44-8.fc17, /usr/bin/reboot reboots immediately :-(
Comment 20 Joachim Backes 2012-05-03 05:34:44 UTC 
(In reply to comment #19)
> (In reply to comment #18)
> > systemd-44-8.fc17 has been submitted as an update for Fedora 17.
> > https://admin.fedoraproject.org/updates/systemd-44-8.fc17
> 
> After having installed systemd-44-8.fc17, /usr/bin/reboot reboots immediately
> :-)

Crud: running "yum update" after systemd update brings back the bug! I don't know which component is the culprit:

Packages updated:

May 03 06:54:04 Updated: libgcc-4.7.0-3.fc17.x86_64
May 03 06:54:17 Updated: glibc-common-2.15-35.fc17.x86_64
May 03 06:54:21 Updated: glibc-2.15-35.fc17.x86_64
May 03 06:54:21 Updated: wine-common-1.5.3-1.fc17.noarch
May 03 06:54:31 Updated: wine-core-1.5.3-1.fc17.x86_64
May 03 06:54:32 Updated: libstdc++-4.7.0-3.fc17.x86_64
May 03 06:54:32 Updated: libgomp-4.7.0-3.fc17.x86_64
May 03 06:54:33 Updated: nss-3.13.4-3.fc17.x86_64
May 03 06:54:34 Updated: nss-sysinit-3.13.4-3.fc17.x86_64
May 03 06:54:34 Updated: 1:NetworkManager-glib-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:54:40 Updated: libgcj-4.7.0-3.fc17.x86_64
May 03 06:54:53 Updated: libgcj-devel-4.7.0-3.fc17.x86_64
May 03 06:54:53 Updated: 1:NetworkManager-gtk-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:54:56 Updated: libstdc++-devel-4.7.0-3.fc17.x86_64
May 03 06:54:57 Updated: wine-capi-1.5.3-1.fc17.x86_64
May 03 06:54:57 Updated: wine-ldap-1.5.3-1.fc17.x86_64
May 03 06:54:58 Updated: wine-twain-1.5.3-1.fc17.x86_64
May 03 06:54:59 Updated: wine-openal-1.5.3-1.fc17.x86_64
May 03 06:54:59 Updated: wine-cms-1.5.3-1.fc17.x86_64
May 03 06:55:00 Updated: wine-wow-1.5.3-1.fc17.x86_64
May 03 06:55:00 Updated: wine-alsa-1.5.3-1.fc17.x86_64
May 03 06:55:00 Updated: wine-pulseaudio-1.5.3-1.fc17.x86_64
May 03 06:55:01 Updated: 1:wpa_supplicant-1.0-0.4.fc17.x86_64
May 03 06:55:05 Updated: 1:NetworkManager-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:55:06 Updated: cpp-4.7.0-3.fc17.x86_64
May 03 06:55:09 Updated: glibc-headers-2.15-35.fc17.x86_64
May 03 06:55:10 Updated: glibc-devel-2.15-35.fc17.x86_64
May 03 06:55:13 Updated: gcc-4.7.0-3.fc17.x86_64
May 03 06:55:13 Updated: libquadmath-4.7.0-3.fc17.x86_64
May 03 06:55:14 Updated: wine-ms-sans-serif-fonts-1.5.3-1.fc17.noarch
May 03 06:55:14 Updated: wine-systemd-1.5.3-1.fc17.noarch
May 03 06:55:15 Updated: wine-small-fonts-1.5.3-1.fc17.noarch
May 03 06:55:15 Updated: wine-courier-fonts-1.5.3-1.fc17.noarch
May 03 06:55:16 Updated: wine-marlett-fonts-1.5.3-1.fc17.noarch
May 03 06:55:17 Updated: libwacom-data-0.5-1.fc17.noarch
May 03 06:55:17 Updated: wine-system-fonts-1.5.3-1.fc17.noarch
May 03 06:55:18 Updated: wine-symbol-fonts-1.5.3-1.fc17.noarch
May 03 06:55:18 Updated: wine-tahoma-fonts-1.5.3-1.fc17.noarch
May 03 06:55:19 Updated: wine-fonts-1.5.3-1.fc17.noarch
May 03 06:55:20 Updated: hplip-common-3.12.4-2.fc17.x86_64
May 03 06:55:21 Updated: hplip-libs-3.12.4-2.fc17.x86_64
May 03 06:55:24 Updated: 1:hpijs-3.12.4-2.fc17.x86_64
May 03 06:55:27 Updated: hplip-3.12.4-2.fc17.x86_64
May 03 06:55:27 Updated: libsane-hpaio-3.12.4-2.fc17.x86_64
May 03 06:55:28 Updated: libwacom-0.5-1.fc17.x86_64
May 03 06:55:29 Updated: libgfortran-4.7.0-3.fc17.x86_64
May 03 06:55:30 Updated: gcc-java-4.7.0-3.fc17.x86_64
May 03 06:55:32 Updated: gcc-c++-4.7.0-3.fc17.x86_64
May 03 06:55:44 Updated: 1:NetworkManager-gnome-0.9.4.0-8.git20120502.fc17.x86_64
May 03 06:55:45 Updated: nss-tools-3.13.4-3.fc17.x86_64
May 03 06:55:47 Updated: shotwell-0.12.2-2.fc17.x86_64
May 03 06:55:52 Updated: wine-devel-1.5.3-1.fc17.x86_64
May 03 06:55:53 Updated: mdadm-3.2.3-9.fc17.x86_64
May 03 06:56:08 Updated: man-db-2.6.0.2-6.fc17.x86_64
May 03 06:56:09 Updated: fedora-logos-17.0.1-1.fc17.noarch
May 03 06:56:10 Updated: wine-sysvinit-1.5.3-1.fc17.noarch
May 03 06:56:13 Updated: glibc-2.15-35.fc17.i686
May 03 06:56:14 Updated: libgcc-4.7.0-3.fc17.i686
May 03 06:56:24 Updated: wine-core-1.5.3-1.fc17.i686
May 03 06:56:25 Updated: wine-desktop-1.5.3-1.fc17.noarch
May 03 06:56:25 Updated: wine-ldap-1.5.3-1.fc17.i686
May 03 06:56:26 Updated: wine-cms-1.5.3-1.fc17.i686
May 03 06:56:26 Updated: wine-openal-1.5.3-1.fc17.i686
May 03 06:56:27 Updated: wine-alsa-1.5.3-1.fc17.i686
May 03 06:56:28 Updated: wine-pulseaudio-1.5.3-1.fc17.i686
May 03 06:56:29 Updated: wine-capi-1.5.3-1.fc17.i686
May 03 06:56:29 Updated: wine-twain-1.5.3-1.fc17.i686
May 03 06:56:29 Updated: wine-1.5.3-1.fc17.x86_64
May 03 06:56:30 Updated: libstdc++-4.7.0-3.fc17.i686
May 03 06:56:31 Updated: nss-3.13.4-3.fc17.i686
Comment 21 Joachim Backes 2012-05-03 05:55:52 UTC 
Additionally, I can say that the problem has deteriorated: Even going from runlevel 5 into runlevel 3 and then rebooting (was previously a workaround) no more helps. 

Even if booting directly to runlevel 3 and the trying to reboot shows the delay!
Comment 22 Jacek Pliszka 2012-05-03 07:04:41 UTC 
I confirm Joachim's comment - something in the recent updates extended the problem to runlevel 3 as well.
Comment 23 Michal Schmidt 2012-05-03 07:38:40 UTC 
I blame the NetworkManager update. I already noted that in the systemd update description:
https://admin.fedoraproject.org/updates/systemd-44-8.fc17
See also https://bugzilla.redhat.com/show_bug.cgi?id=739836#c61
Comment 24 Fedora Update System 2012-05-03 15:59:07 UTC 
Package systemd-44-8.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-44-8.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-7193/systemd-44-8.fc17
then log in and leave karma (feedback).
Comment 25 Michal Schmidt 2012-05-04 11:11:26 UTC 
*** Bug 817182 has been marked as a duplicate of this bug. ***
Comment 26 Adam Williamson 2012-05-04 17:10:26 UTC 
Discussed at 2012-05-03 blocker review meeting. Accepted as a blocker per criterion "All release-blocking desktops' offered mechanisms (if any) for shutting down, logging out and rebooting must work". Note that 44-8 seems to fix it.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 27 Fedora Update System 2012-05-04 23:04:20 UTC 
systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 28 Steve Gage 2012-05-06 12:02:53 UTC 
(In reply to comment #27)
> systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If
> problems still persist, please make note of it in this bug report.

systemd-44-8.fc17 did not fix the problem for me. 

I am running FC17 32-bit fully updated as of 1130 UTC May 6.
Comment 29 Michal Schmidt 2012-05-06 14:38:16 UTC 
(In reply to comment #28)
> systemd-44-8.fc17 did not fix the problem for me. 
> 
> I am running FC17 32-bit fully updated as of 1130 UTC May 6.

Make sure you try "yum distro-sync" as well to perform downgrades for the cases where packages were removed from updates-testing.
If the problem persists, please file a new bug.
Comment 30 Sergio Basto 2012-05-06 15:18:51 UTC 
(In reply to comment #28)
> (In reply to comment #27)
> > systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If
> > problems still persist, please make note of it in this bug report.
> 
> systemd-44-8.fc17 did not fix the problem for me. 
> 
> I am running FC17 32-bit fully updated as of 1130 UTC May 6.

have you got kernel 3.3.4-3.fc17  also fixes some stuck on kernel 3.3
Comment 31 Sergio Basto 2012-05-06 15:27:19 UTC 
(In reply to comment #30)
> (In reply to comment #28)
> > (In reply to comment #27)
> > > systemd-44-8.fc17 has been pushed to the Fedora 17 stable repository.  If
> > > problems still persist, please make note of it in this bug report.
> > 
> > systemd-44-8.fc17 did not fix the problem for me. 
> > 
> > I am running FC17 32-bit fully updated as of 1130 UTC May 6.
> 
> have you got kernel 3.3.4-3.fc17  also fixes some stuck on kernel 3.3

also take a look at bug 806548 and bug 739836