Bug 1023788

Summary: reboot/shutdown does not close ssh client connectiony
Product: [Fedora] Fedora Reporter: Harald Reindl <h.reindl>
Component: systemdAssignee: systemd-maint
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: collura, georgmueller, johannbg, kajtzu, lars, lnykryn, mattias.ellert, mgrepl, mihai, mkelly, msekleta, ncrubel, plautrba, pwouters, rwahl, samuel-rhbugs, stephan.wiesand, systemd-maint, tmraz, ville.skytta, vpavlin, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-23 16:59:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Harald Reindl 2013-10-27 21:13:46 UTC
i assume it has something to do with systemd, however, on my F20 test-VM reboot/shutdown let hang the client for a long time with no way to kill the session with STRG+C and the only way to release the terminal is to have a different one for "kill -s SIGKILL" the ssh-client

Comment 1 Nathan Crubel 2013-11-18 15:26:31 UTC
I am seeing the same issue.

Thanks

nathan

Comment 2 Petr Lautrbach 2013-11-19 14:35:39 UTC
I can reproduce this issue using network.service instead of NetworkManager.service. And it incidentally works with nm.service because nm doesn't shut a network interface down on stop. The problem with ssh session is that should be killed only on shutdown/reboot and they shouldn't be killed at sshd stop, network stop and so.

Following unit works for me as a workaround:

# cat /etc/systemd/system/sshd-shutdown.service 
[Unit]
Description=kill all sshd sessions
Requires=mutil-user.target

[Service]
ExecStart=/usr/bin/killall sshd
Type=oneshot

[Install]
WantedBy=shutdown.target reboot.target poweroff.target

Comment 3 Harald Reindl 2013-11-19 14:41:32 UTC
i know that they should not be killed and sshd restart/stop and network restart but that doe snot change the fact that it is a regression because on F18/F19 there is no problem

Comment 4 Petr Lautrbach 2013-11-19 14:58:57 UTC
sshd.service are same on f18 and f19. So, as you wrote, it's probably related to something else than openssh itself and I'm still looking for it. I started with the sshd-shutdown.service unit as it might be one of solutions how to workaround random future changes in the systemd or network services.

Comment 5 Harald Reindl 2013-12-13 13:51:59 UTC
pretty sure systemd, the same happens on RHEL7 public-beta

Comment 6 Madison Kelly 2014-01-02 14:06:50 UTC
I am seeing this on RHEL 7 beta as well, filed https://bugzilla.redhat.com/show_bug.cgi?id=1047614.

Comment 7 Georg Müller 2014-02-04 12:45:24 UTC
this seems to be the same as https://bugzilla.redhat.com/show_bug.cgi?id=626477

To cancel the ssh connection on the client side, type "~" followed by ".", this way, you don't need to kill the ssh client. If it does not work in the first place, press enter and repeat the sequence "~.".

Comment 8 Harald Reindl 2014-02-06 20:10:06 UTC
and *when* will we see a fix for that?

* this was reported *long* before F20 final release
* this can be reproduced with *every* reboot and shutdown

this did not happen in *any* fedora systemd-release and some magic sequences are only a dirty workaround for things which have to work out-of-the-box unquestinable and never made it in a final release or even through a beta-QA

Comment 9 Paul Wouters 2014-02-06 21:02:11 UTC
(In reply to Petr Lautrbach from comment #2)
> I can reproduce this issue using network.service instead of
> NetworkManager.service. And it incidentally works with nm.service because nm
> doesn't shut a network interface down on stop. The problem with ssh session
> is that should be killed only on shutdown/reboot and they shouldn't be
> killed at sshd stop, network stop and so.
> 
> Following unit works for me as a workaround:
> 
> # cat /etc/systemd/system/sshd-shutdown.service 
> [Unit]
> Description=kill all sshd sessions
> Requires=mutil-user.target
> 
> [Service]
> ExecStart=/usr/bin/killall sshd
> Type=oneshot
> 
> [Install]
> WantedBy=shutdown.target reboot.target poweroff.target

This workaround fixed my problem on our testing VMs that are rebooted per test-case and were hanging continiously and failing to reboot using virsh shutdown and virsh start.

Thanks Petr and Harald!

I do hope this bug will get resolved properly soon.

Comment 10 Harald Reindl 2014-02-06 21:14:32 UTC
yes, the workaround does what it is supposed to do
frankly i will continue to have it installed forever

why? because i do not trust systemd and sshd in combination any longer after that and some fedora-releases before where "yum distro-sync" and complete-transaction each time killed teh current ssh-session (the opposite of now) while ssh, reboots and ssh and dist-upgrades *never* was a problem before F15/F17

Comment 11 Madison Kelly 2014-02-06 21:17:51 UTC
As per #c2 and #c9, I created:

vim /etc/systemd/system/sshd-shutdown.service 
====
[Unit]
Description=kill all sshd sessions
Requires=mutil-user.target

[Service]
ExecStart=/usr/bin/killall sshd
Type=oneshot

[Install]
WantedBy=shutdown.target reboot.target poweroff.target
====

Then:

systemctl daemon-reload
systemctl enable sshd-shutdown.service
====
ln -s '/etc/systemd/system/sshd-shutdown.service' '/etc/systemd/system/shutdown.target.wants/sshd-shutdown.service'
ln -s '/etc/systemd/system/sshd-shutdown.service' '/etc/systemd/system/reboot.target.wants/sshd-shutdown.service'
ln -s '/etc/systemd/system/sshd-shutdown.service' '/etc/systemd/system/poweroff.target.wants/sshd-shutdown.service'
====

After this, I shut down and the terminal closed properly.

Comment 12 Ville Skyttä 2014-02-07 21:33:45 UTC
(In reply to digimer from comment #11)
> Requires=mutil-user.target

BTW at least for me, this "mutil" typo is essential for this hack to work. Do the s/mutil/multi/ fix or remove the whole line and it no longer does. Too lazy at the moment to figure out why, just did s/mutil/doesnt-exist/ locally to make it more clear and moving on...

Comment 13 Harald Reindl 2014-02-07 21:43:11 UTC
confirmed and makes systemd of F20/RHEL7 even more questionable because what crazy bahvior is it to rely on a typo? normally it should do the same without this line as with a non-existing target

what about the systemd guys waking up to fix this odd behavior at the root-cause and while the are at it debug how this rely-on-error can be possible at all?!?!

Comment 14 Harald Reindl 2014-02-12 19:03:53 UTC
in the meantime https://bugzilla.redhat.com/show_bug.cgi?id=1047614 for RHEL7 became "urgent" and "critical" - but who am i to qualify that such bugs make it in a release even if they are reported before GA and not fixed within 3 months...

Comment 15 Lennart Poettering 2014-02-23 16:59:04 UTC

*** This bug has been marked as a duplicate of bug 626477 ***

Comment 16 Madison Kelly 2014-02-23 17:29:55 UTC
This is marked as a duplicate of a ~3.5 year old bug... Does that not complicate resolving it today? Will this (regression?) be fixed?

Comment 17 Harald Reindl 2014-02-23 17:33:16 UTC
and the main question: why doe sthe problem re-appear
this is *not* a duplicate
this is an new regression in F20/RHEL7 as long the old
fix did not get removed completly for whatever reason

Comment 18 Harald Reindl 2014-02-26 11:13:11 UTC
systemd-210-2.fc21.x86_64 is fixing that behavior, well, hopefully F20 will get systemd-210 soon - unsure how to verify if it also fixes the "rely-on-typo-behavior" from https://bugzilla.redhat.com/show_bug.cgi?id=1023788#c12