Bug 963268 - add socket activated sshd units to the package
Summary: add socket activated sshd units to the package
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: openssh
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Petr Lautrbach
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: WhyWeBootSoSlow
TreeView+ depends on / blocked
 
Reported: 2013-05-15 14:19 UTC by Lennart Poettering
Modified: 2014-04-15 22:45 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-05-21 16:48:22 UTC
Type: Bug


Attachments (Terms of Use)

Description Lennart Poettering 2013-05-15 14:19:49 UTC
The sshd service is a really good candidate for lazy activation when the first connection comes in, since it is not frequently used (even the most busy servers probably won't serve more than a few connections per minute).

In order to save resources and minimize boot-times it would great if sshd could be run in socket-activation mode by default, but to keep the unit files for standalone mode around, so that administrators can easily change from socket-activated to standalone mode.

I'd also like to ask you to split out sshd-keygen from the sshd unit file, so that it can be conditionalized based on whether a key already exists, and be used by the per-connection sshd units, too.

I'd like to suggest units like the following:

/usr/lib/systemd/system/sshd.service:

<snip>
[Unit]
Description=Stand-Alone OpenSSH Server
Conflicts=sshd.socket
Wants=sshd-keygen.service
After=sshd-keygen.service

[Service]
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=/usr/sbin/sshd $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
Type=forking

[Install]
WantedBy=multi-user.target
</snip>

/usr/lib/systemd/system/sshd.socket:

<snip>
[Unit]
Description=SSH Socket for Per-Connection Servers

[Socket]
ListenStream=22
Accept=yes

[Install]
WantedBy=sockets.target
</snip>

/usr/lib/systemd/system/sshd@.service:
<snip>
[Unit]
Description=Per-Connection OpenSSH Server
Wants=sshd-keygen.service
After=sshd-keygen.service

[Service]
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=-/usr/sbin/sshd -i $OPTIONS
StandardInput=socket
</snip>

/usr/lib/systemd/system/sshd-keygen.service:
<snip>
[Unit]
Description=OpenSSH Server Key Generation
ConditionPathExists=!/etc/ssh/ssh_host_rsa_key

[Service]
ExecStart=/usr/sbin/sshd-keygen
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
</snip>

And in the spec file:

<snip>
%post
%systemd_post sshd.service sshd.socket sshd-keygen.service

%preun
%systemd_preun sshd.service sshd.socket sshd-keygen.service

%postun
%systemd_postun
</snip>

We'd then change the default preset file to only enable sshd.socket by default. People may then manually enable sshd.service or sshd-keygen.service on the local machine, so that they can run the standalone version instead, or to pull the key generation into the boot process (rather than lazily doing that as I'd propose). Site administrators could also ship their own preset file to make this change across a big number of machines.

Thank you for considering this!

Comment 1 Tomas Mraz 2013-05-15 14:32:23 UTC
Given the F19 beta freeze, moving to rawhide.

Comment 2 Lennart Poettering 2013-05-15 14:46:47 UTC
Here's a longer explanation BTW, what these unit files would do:

http://0pointer.de/blog/projects/inetd.html

Comment 3 Jóhann B. Guðmundsson 2013-05-15 15:41:12 UTC
This units lack Documentation and Tomaz has anything changes now since bug 697698 ?

Comment 4 Petr Lautrbach 2013-05-15 16:08:43 UTC
I don't think it's a good idea to have this kind of service socket activated by default.

It's not only about performance. There is  a slight startup delay for every connection
caused by exec()ing /usr/sbin/sshd process and regeneration of server key.

It's even liable to DOS attack against user's system using unauthenticated connections - MaxStartups 
(man sshd_config) cannot be applied in this case.

Further, it's harder to audit using conventional tools, e.g. nestat links socket to systemd without any
connection to /usr/sbin/sshd:

tcp6       0      0 [::]:22                 [::]:*                  LISTEN      1/systemd           
tcp6       0      0 192.168.122.51:22       192.168.122.1:36354     ESTABLISHED 1/systemd           
tcp6       0      0 192.168.122.51:22       192.168.122.1:36407     ESTABLISHED 1/systemd

Additionally, there's unavoidable downtime during reconfiguration with this configuration.
You need to stop socket first, reconfigure your sshd, check configuration and start the socket again.
With a traditional service, you don't have to stop the service during reconfiguration.

But I think that we can provide the openssh-server-ondemand subpackage with socket unit for
users who want use this. So they would be able to install and enable sshd.socket via kickstart or manually.

Comment 5 Petr Lautrbach 2013-05-15 16:13:39 UTC
(In reply to comment #0)
> /usr/lib/systemd/system/sshd@.service:
> <snip>
> [Unit]
> Description=Per-Connection OpenSSH Server
> Wants=sshd-keygen.service
> After=sshd-keygen.service

The sshd-keygen.service must be run for every ssh connection.

> 
> [Service]
> EnvironmentFile=-/etc/sysconfig/sshd
> ExecStart=-/usr/sbin/sshd -i $OPTIONS
> StandardInput=socket
> </snip>
> 
> /usr/lib/systemd/system/sshd-keygen.service:
> <snip>
> [Unit]
> Description=OpenSSH Server Key Generation
> ConditionPathExists=!/etc/ssh/ssh_host_rsa_key

There's also /etc/ssh/ssh_host_dsa_key and both filenames can be configured
via /etc/sysconfig/sshd

> 
> [Service]
> ExecStart=/usr/sbin/sshd-keygen
> Type=oneshot
> RemainAfterExit=yes
> 
> [Install]
> WantedBy=multi-user.target
> </snip>

Again, sshd_keygen must be run before every /usr/sbin/sshd fork(). It's a part
of sshd incialization.

Comment 6 Lennart Poettering 2013-05-15 18:04:15 UTC
(In reply to comment #5)
> (In reply to comment #0)
> > /usr/lib/systemd/system/sshd@.service:
> > <snip>
> > [Unit]
> > Description=Per-Connection OpenSSH Server
> > Wants=sshd-keygen.service
> > After=sshd-keygen.service
> 
> The sshd-keygen.service must be run for every ssh connection.

How so? The key is written to the fs, why would this needed to be started multiple times? Can you explain?

> > [Unit]
> > Description=OpenSSH Server Key Generation
> > ConditionPathExists=!/etc/ssh/ssh_host_rsa_key
> 
> There's also /etc/ssh/ssh_host_dsa_key and both filenames can be configured
> via /etc/sysconfig/sshd

Well, if the user changes the configuration, then he changes the configuration and he can edit the unit files, too.

You can easily check for both keys, with something like this:

ConditionGlobExists=!/etc/ssh/ssh_host_*_key

Comment 7 Lennart Poettering 2013-05-15 18:10:01 UTC
(In reply to comment #4)
> I don't think it's a good idea to have this kind of service socket activated
> by default.
> 
> It's not only about performance. There is  a slight startup delay for every
> connection
> caused by exec()ing /usr/sbin/sshd process and regeneration of server key.

Well, that's pretty niglible on *any* modern hardware. And for the systems where this matters we'd provide the stand-alone version too.

> It's even liable to DOS attack against user's system using unauthenticated
> connections - MaxStartups 
> (man sshd_config) cannot be applied in this case.

Hmm? systemd will enforce a limit on non-authenticated connections, so where's the problem?

> Further, it's harder to audit using conventional tools, e.g. nestat links
> socket to systemd without any
> connection to /usr/sbin/sshd:
> 
> tcp6       0      0 [::]:22                 [::]:*                  LISTEN  
> 1/systemd           
> tcp6       0      0 192.168.122.51:22       192.168.122.1:36354    
> ESTABLISHED 1/systemd           
> tcp6       0      0 192.168.122.51:22       192.168.122.1:36407    
> ESTABLISHED 1/systemd

Well, that's (if at all) a problem that needs to be fixed in netstat (it should look for all open fds to this, and list the processes). This is also not a new problem, and certainly nothing specific to ssh, as inetd did the same thing.

> Additionally, there's unavoidable downtime during reconfiguration with this
> configuration.
> You need to stop socket first, reconfigure your sshd, check configuration
> and start the socket again.
> With a traditional service, you don't have to stop the service during
> reconfiguration.

Not following here... You don't have to restart the socket, unless you change configuration of the socket itself -- which is exactly the same of a stand-aline sshd instance and its socket.
> 
> But I think that we can provide the openssh-server-ondemand subpackage with
> socket unit for
> users who want use this. So they would be able to install and enable
> sshd.socket via kickstart or manually.

This would be an awful solution. Placing these two additional unit files in sshd rpm should not bother anybody, they are tiny, and make the life of everybody easier.

Comment 8 Tomas Mraz 2013-05-15 18:27:28 UTC
(In reply to comment #7)
> > Further, it's harder to audit using conventional tools, e.g. nestat links
> > socket to systemd without any
> > connection to /usr/sbin/sshd:
> > 
> > tcp6       0      0 [::]:22                 [::]:*                  LISTEN  
> > 1/systemd           
> > tcp6       0      0 192.168.122.51:22       192.168.122.1:36354    
> > ESTABLISHED 1/systemd           
> > tcp6       0      0 192.168.122.51:22       192.168.122.1:36407    
> > ESTABLISHED 1/systemd
> 
> Well, that's (if at all) a problem that needs to be fixed in netstat (it
> should look for all open fds to this, and list the processes). This is also
> not a new problem, and certainly nothing specific to ssh, as inetd did the
> same thing.

Nobody really used sshd activated by inetd.

There is another thing to consider which is not relevant to desktop installs but very much to server ones. We really want to in case of emergency ssh login to do as little as possible during the login and starting up sshd might not be a good idea.

As it is not clear whether Fedora is primarily a desktop or server OS I'd prefer to be on the safe side in expense of probably wasting some resources and some seconds on system boot.

Comment 9 Petr Lautrbach 2013-05-15 18:38:32 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #0)
> > > /usr/lib/systemd/system/sshd@.service:
> > > <snip>
> > > [Unit]
> > > Description=Per-Connection OpenSSH Server
> > > Wants=sshd-keygen.service
> > > After=sshd-keygen.service
> > 
> > The sshd-keygen.service must be run for every ssh connection.
> 
> How so? The key is written to the fs, why would this needed to be started
> multiple times? Can you explain?

rephrase - The sshd-keygen.service must be run for every ssh connection if sshd keys doesn't exist otherwise sshd doesn't start. See #810419 and #805338

> 
> > > [Unit]
> > > Description=OpenSSH Server Key Generation
> > > ConditionPathExists=!/etc/ssh/ssh_host_rsa_key
> > 
> > There's also /etc/ssh/ssh_host_dsa_key and both filenames can be configured
> > via /etc/sysconfig/sshd
> 
> Well, if the user changes the configuration, then he changes the
> configuration and he can edit the unit files, too.

Honestly, I don't prefer changing unit files (even with all the .include and .d stuff) over changing configuration file.

> You can easily check for both keys, with something like this:
> 
> ConditionGlobExists=!/etc/ssh/ssh_host_*_key

So if random file /etc/ssh/ssh_host_BAK_key doesn't exists it will run sshd_keygen.service?


(In reply to comment #7)
> (In reply to comment #4)
> > I don't think it's a good idea to have this kind of service socket activated
> > by default.
> > 
> > It's not only about performance. There is  a slight startup delay for every
> > connection
> > caused by exec()ing /usr/sbin/sshd process and regeneration of server key.
> 
> Well, that's pretty niglible on *any* modern hardware. And for the systems
> where this matters we'd provide the stand-alone version too.
> 
> > It's even liable to DOS attack against user's system using unauthenticated
> > connections - MaxStartups 
> > (man sshd_config) cannot be applied in this case.
> 
> Hmm? systemd will enforce a limit on non-authenticated connections, so
> where's the problem?

But it will limit authenticated connections at the same time given that systemd can't distinguish connections.
 
> > Further, it's harder to audit using conventional tools, e.g. nestat links
> > socket to systemd without any
> > connection to /usr/sbin/sshd:
> > 
> > tcp6       0      0 [::]:22                 [::]:*                  LISTEN  
> > 1/systemd           
> > tcp6       0      0 192.168.122.51:22       192.168.122.1:36354    
> > ESTABLISHED 1/systemd           
> > tcp6       0      0 192.168.122.51:22       192.168.122.1:36407    
> > ESTABLISHED 1/systemd
> 
> Well, that's (if at all) a problem that needs to be fixed in netstat (it
> should look for all open fds to this, and list the processes). This is also
> not a new problem, and certainly nothing specific to ssh, as inetd did the
> same thing.

Or there's a new solution which  has some issues and doesn't provide any considerable improvement over the existing one.

> 
> > Additionally, there's unavoidable downtime during reconfiguration with this
> > configuration.
> > You need to stop socket first, reconfigure your sshd, check configuration
> > and start the socket again.
> > With a traditional service, you don't have to stop the service during
> > reconfiguration.
> 
> Not following here... You don't have to restart the socket, unless you
> change configuration of the socket itself -- which is exactly the same of a
> stand-aline sshd instance and its socket.

Every time a new connection come, a new sshd@.service will be run so sshd will read sshd_config file. So you need to stop it first if you want to edit it inplace.

Comment 10 Jóhann B. Guðmundsson 2013-05-15 18:41:44 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > > Further, it's harder to audit using conventional tools, e.g. nestat links
> > > socket to systemd without any
> > > connection to /usr/sbin/sshd:
> > > 
> > > tcp6       0      0 [::]:22                 [::]:*                  LISTEN  
> > > 1/systemd           
> > > tcp6       0      0 192.168.122.51:22       192.168.122.1:36354    
> > > ESTABLISHED 1/systemd           
> > > tcp6       0      0 192.168.122.51:22       192.168.122.1:36407    
> > > ESTABLISHED 1/systemd
> > 
> > Well, that's (if at all) a problem that needs to be fixed in netstat (it
> > should look for all open fds to this, and list the processes). This is also
> > not a new problem, and certainly nothing specific to ssh, as inetd did the
> > same thing.
> 
> Nobody really used sshd activated by inetd.
> 
> There is another thing to consider which is not relevant to desktop installs
> but very much to server ones. We really want to in case of emergency ssh
> login to do as little as possible during the login and starting up sshd
> might not be a good idea.
> 
> As it is not clear whether Fedora is primarily a desktop or server OS I'd
> prefer to be on the safe side in expense of probably wasting some resources
> and some seconds on system boot.

Afaik sshd is not installed/enabled on live spins so this only affects the releng dvd but the argument can be made that we should drop it and admins should use ks to do server installs while the desktop users should install of live

Comment 11 Tomas Mraz 2013-05-15 18:52:29 UTC
(In reply to comment #10)
> Afaik sshd is not installed/enabled on live spins so this only affects the
> releng dvd but the argument can be made that we should drop it and admins
> should use ks to do server installs while the desktop users should install
> of live

This makes the change even less needed.

Comment 12 Lennart Poettering 2013-05-15 19:13:22 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > > Further, it's harder to audit using conventional tools, e.g. nestat links
> > > socket to systemd without any
> > > connection to /usr/sbin/sshd:
> > > 
> > > tcp6       0      0 [::]:22                 [::]:*                  LISTEN  
> > > 1/systemd           
> > > tcp6       0      0 192.168.122.51:22       192.168.122.1:36354    
> > > ESTABLISHED 1/systemd           
> > > tcp6       0      0 192.168.122.51:22       192.168.122.1:36407    
> > > ESTABLISHED 1/systemd
> > 
> > Well, that's (if at all) a problem that needs to be fixed in netstat (it
> > should look for all open fds to this, and list the processes). This is also
> > not a new problem, and certainly nothing specific to ssh, as inetd did the
> > same thing.
> 
> Nobody really used sshd activated by inetd.

Well, but inetd has been used for many daemons, and it's all the same, sshd is not special....

> There is another thing to consider which is not relevant to desktop installs
> but very much to server ones. We really want to in case of emergency ssh
> login to do as little as possible during the login and starting up sshd
> might not be a good idea.
> 
> As it is not clear whether Fedora is primarily a desktop or server OS I'd
> prefer to be on the safe side in expense of probably wasting some resources
> and some seconds on system boot.

This is relevant for servers in the cloud where you want to run many VMs on the same hosts, and want their footprint and boot-times to be minimal. Starting sshd only when needed drastically improves both since footprint/boot-time is multiple by the number of VMs.

It's a question of scalability, of commit resources to customers, and that's highly relevant for servers.

Comment 13 Jóhann B. Guðmundsson 2013-05-15 19:15:30 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > Afaik sshd is not installed/enabled on live spins so this only affects the
> > releng dvd but the argument can be made that we should drop it and admins
> > should use ks to do server installs while the desktop users should install
> > of live
> 
> This makes the change even less needed.

The implementation of SSH in Fedora should be done with the best server integration in mind. 

The majority of our desktop user base will never use ssh and those that do surely are capably of installing and configuring it manually afterwards but there is no harm done shipping socket activated units along the ones we already do so the admin/user can simply chose which one he would like to use. 

For those that dont know ssh being installed/enable by default of dvd leaves users exposed to brute force attacks out of the box with no means to notify the unsuspecting desktop user about when someone is doing it. 

So the argument can be made that it should also be disabled on the releng dvd by default and only installed and enabled by admins via ks, puppet,chef cobbler spacewalk and what not when they setup the server.

Comment 14 Tomas Mraz 2013-05-15 19:26:04 UTC
(In reply to comment #12)
> This is relevant for servers in the cloud where you want to run many VMs on
> the same hosts, and want their footprint and boot-times to be minimal.
> Starting sshd only when needed drastically improves both since
> footprint/boot-time is multiple by the number of VMs.
> 
> It's a question of scalability, of commit resources to customers, and that's
> highly relevant for servers.

Sure, then for cloud where there might be different preferences - resource sparing over reliability it might make sense to use the activation. I'd still prefer regular sshd server for servers.

As bugzilla is not a discussion forum (as we could see argued about on many bugzillas everywhere) I'd say we should stop discussing here now.

Comment 15 Lennart Poettering 2013-05-15 19:26:31 UTC
(In reply to comment #9)
> (In reply to comment #6)
> > (In reply to comment #5)
> > > (In reply to comment #0)
> > > > /usr/lib/systemd/system/sshd@.service:
> > > > <snip>
> > > > [Unit]
> > > > Description=Per-Connection OpenSSH Server
> > > > Wants=sshd-keygen.service
> > > > After=sshd-keygen.service
> > > 
> > > The sshd-keygen.service must be run for every ssh connection.
> > 
> > How so? The key is written to the fs, why would this needed to be started
> > multiple times? Can you explain?
> 
> rephrase - The sshd-keygen.service must be run for every ssh connection if
> sshd keys doesn't exist otherwise sshd doesn't start. See #810419 and #805338

I don't grok this. Why mus this be run for every ssh connection? It should be started and finished before the first sshd starts up, that's all, and that's what the unit file above does. That's really all that's needed. What am i missing here? I cannot parse what you say, nor the two bug reports you cite.

> > Well, if the user changes the configuration, then he changes the
> > configuration and he can edit the unit files, too.
> 
> Honestly, I don't prefer changing unit files (even with all the .include and
> .d stuff) over changing configuration file.

Unit files are configuration files.
 
> > You can easily check for both keys, with something like this:
> > 
> > ConditionGlobExists=!/etc/ssh/ssh_host_*_key
> 
> So if random file /etc/ssh/ssh_host_BAK_key doesn't exists it will run
> sshd_keygen.service?

Oh god. Sure, then here you go:

ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key

With these to conditions the unit will be started when at least one of the two is missing.

> > Hmm? systemd will enforce a limit on non-authenticated connections, so
> > where's the problem?
> 
> But it will limit authenticated connections at the same time given that
> systemd can't distinguish connections.

Hmm? That makes no sense. As every connection starts out as unauthenticated authentication and only later becomes an authenticated systemd's MacConnections and sshd's MaxStartups do pretty much the same thing?

>  
> > > Further, it's harder to audit using conventional tools, e.g. nestat links
> > > socket to systemd without any
> > > connection to /usr/sbin/sshd:
> > > 
> > > tcp6       0      0 [::]:22                 [::]:*                  LISTEN  
> > > 1/systemd           
> > > tcp6       0      0 192.168.122.51:22       192.168.122.1:36354    
> > > ESTABLISHED 1/systemd           
> > > tcp6       0      0 192.168.122.51:22       192.168.122.1:36407    
> > > ESTABLISHED 1/systemd
> > 
> > Well, that's (if at all) a problem that needs to be fixed in netstat (it
> > should look for all open fds to this, and list the processes). This is also
> > not a new problem, and certainly nothing specific to ssh, as inetd did the
> > same thing.
> 
> Or there's a new solution which  has some issues and doesn't provide any
> considerable improvement over the existing one.

It makes things more efficient, more reliable and uses less resources. That all together means a lot, on servers, and especially in the cloud where the benefits multiply.

But anyway, could you at least add the socket-activation units to the package? Pretty please? Then admins can at least enable the socket-activation mode if they want to.

Comment 16 Petr Lautrbach 2013-05-16 10:52:50 UTC
(In reply to comment #15)
> > rephrase - The sshd-keygen.service must be run for every ssh connection if
> > sshd keys doesn't exist otherwise sshd doesn't start. See #810419 and #805338
> 
> I don't grok this. Why mus this be run for every ssh connection? It should
> be started and finished before the first sshd starts up, that's all, and
> that's what the unit file above does. That's really all that's needed. What
> am i missing here? I cannot parse what you say, nor the two bug reports you
> cite.

[root@rawhide-devel ~]# systemctl stop sshd.socket
[68727.494914] systemd[1]: Closed SSH Socket for Per-Connection Servers
.
[root@rawhide-devel ~]# mv /etc/ssh/*key ssh

[root@rawhide-devel ~]# systemctl start sshd.socket
[68746.297478] systemd[1]: Starting SSH Socket for Per-Connection Servers.
[68746.332132] systemd[1]: Listening on SSH Socket for Per-Connection Servers.

[root@rawhide-devel ~]# ssh localhost
[68752.141817] systemd[1]: Starting Per-Connection OpenSSH Server...
[68752.150345] systemd[1]: Started Per-Connection OpenSSH Server.
Read from socket failed: Connection reset by peer

sshd-keygen.service hasn't been restarted since it was run during first ssh connection and sshd can't start without its keys. This is a regression.

> > But it will limit authenticated connections at the same time given that
> > systemd can't distinguish connections.
> 
> Hmm? That makes no sense. As every connection starts out as unauthenticated
> authentication and only later becomes an authenticated systemd's
> MacConnections and sshd's MaxStartups do pretty much the same thing?

It's ok to have thousands authenticated connections - it's auditable, you can restrict number of logins or you can always disable the bad user. But it's not ok if an arbitrary client can block all new incoming connections using many new connections periodically  - CVE-2010-5107
 
 
> But anyway, could you at least add the socket-activation units to the
> package? Pretty please? Then admins can at least enable the
> socket-activation mode if they want to.

It needs to be fixed first.

Comment 17 Lennart Poettering 2013-05-16 11:08:10 UTC
(In reply to comment #16)
> (In reply to comment #15)
> > > rephrase - The sshd-keygen.service must be run for every ssh connection if
> > > sshd keys doesn't exist otherwise sshd doesn't start. See #810419 and #805338
> > 
> > I don't grok this. Why mus this be run for every ssh connection? It should
> > be started and finished before the first sshd starts up, that's all, and
> > that's what the unit file above does. That's really all that's needed. What
> > am i missing here? I cannot parse what you say, nor the two bug reports you
> > cite.
> 
> [root@rawhide-devel ~]# systemctl stop sshd.socket
> [68727.494914] systemd[1]: Closed SSH Socket for Per-Connection Servers
> .
> [root@rawhide-devel ~]# mv /etc/ssh/*key ssh
> 
> [root@rawhide-devel ~]# systemctl start sshd.socket
> [68746.297478] systemd[1]: Starting SSH Socket for Per-Connection Servers.
> [68746.332132] systemd[1]: Listening on SSH Socket for Per-Connection
> Servers.
> 
> [root@rawhide-devel ~]# ssh localhost
> [68752.141817] systemd[1]: Starting Per-Connection OpenSSH Server...
> [68752.150345] systemd[1]: Started Per-Connection OpenSSH Server.
> Read from socket failed: Connection reset by peer
> 
> sshd-keygen.service hasn't been restarted since it was run during first ssh
> connection and sshd can't start without its keys. This is a regression.

So let me get this right. The user actively moved away the key, and then the key generation wasn't restarted. 

Well, if the user deliberately changed configuration like this he should also know what to do about it.

The sshd-keygen service should take of the usual post-installation setup, but should not trample over the admin's configuration manipulations all the time.

> > > But it will limit authenticated connections at the same time given that
> > > systemd can't distinguish connections.
> > 
> > Hmm? That makes no sense. As every connection starts out as unauthenticated
> > authentication and only later becomes an authenticated systemd's
> > MacConnections and sshd's MaxStartups do pretty much the same thing?
> 
> It's ok to have thousands authenticated connections - it's auditable, you
> can restrict number of logins or you can always disable the bad user. But
> it's not ok if an arbitrary client can block all new incoming connections
> using many new connections periodically  - CVE-2010-5107

The guy himself didn't even think this really was an issue...

http://www.openwall.com/lists/oss-security/2013/02/06/5

But the fix, i.e. randomized login grace times also applies to connections done via systemd socket activation the same way, as we immediately pass the connection off to the new sshd instance.

I am really not seeing what the problem is supposed to be here, that was different for socket activation than for stand-alone sshd. Please, can your try to explain this in your own words?

> > But anyway, could you at least add the socket-activation units to the
> > package? Pretty please? Then admins can at least enable the
> > socket-activation mode if they want to.
> 
> It needs to be fixed first.

Fixed? What needs to be fixed? What is broken?

Comment 18 Petr Lautrbach 2013-05-17 08:43:05 UTC
(In reply to comment #17)
> The guy himself didn't even think this really was an issue...
> 
> http://www.openwall.com/lists/oss-security/2013/02/06/5

Please read it again. There's even a program attached demonstrating the attack.

> 
> But the fix, i.e. randomized login grace times also applies to connections
> done via systemd socket activation the same way, as we immediately pass the
> connection off to the new sshd instance.
> 
> I am really not seeing what the problem is supposed to be here, that was
> different for socket activation than for stand-alone sshd. Please, can your
> try to explain this in your own words?

It's not the login grace time (LoginGraceTime) but number of concurrent unauthenticated connections (MaxStartups). With a .socket unit you can specify only number of all connections MaxConections. 

The MaxStartups can't be used with the socket activation since each connection has its own server without any awareness of other connections.
 
Using stand-alone sshd and so MaxStartups makes the DOS attacks harder given that it is set to do an early _random_ connection drop by default (see again SSHD_CONFIG(5) please) and it's not applied to authenticated connections.

Comment 19 Michal Schmidt 2013-05-17 09:34:48 UTC
Interesting. A generic R.E.D. for instantiated socket activated services could be implemented by systemd provided that the service is made systemd-aware and communicates the change of the connection status from unauthenticated to authenticated somehow (sd_notify() looks like an obvious candidate).

Comment 20 Michael S. 2013-05-18 13:02:25 UTC
For the record, dropbear package in Fedora already implement the system of having 2 services ( 1 for ssh, 1 for the key )

Comment 21 Petr Lautrbach 2013-05-20 11:34:53 UTC
The proposed sshd-keygen.service has some issues compared to the current status:

- it's not automatically started every time the sshd.service, or the sshd@.service, is started AND host key files are missing - remove RemainAfterExit=yes 

- standalone enabled sshd-keygen.service doesn't make much sense without sshd.{service,socket} enabled - remove [Install] section.

- it brings another level of configuration. There are already two places where user can change the host key location, one is intended for sshd-keygen and sshd.service - /etc/sysconfig/sshd and second for sshd itself - /etc/ssh/sshd_config. With the proposed sshd-keygen.service unit, a third configuration place would be added.

I can see only one reason for using sshd-keygen.service instead of ExecPreStart and it's some optimization based on ConditionPathExists=. So my proposal is:

- don't change sshd.service and keep it default

- let informed users to use the socket activated sshd using following units:

sshd.socket:
[Unit]
Description=SSH Socket for Per-Connection Servers
Conflicts=sshd.service

[Socket]
ListenStream=22
Accept=yes

[Install]
WantedBy=sockets.target


sshd@.service:
[Unit]
Description=Per-Connection OpenSSH Server
Wants=sshd-keygen.service
After=sshd-keygen.service

[Service]
EnvironmentFile=-/etc/sysconfig/sshd
ExecStart=-/usr/sbin/sshd -i $OPTIONS
StandardInput=socket


sshd-keygen.service
[Unit]
Description=OpenSSH Server Key Generation
ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key

[Service]
ExecStart=/usr/sbin/sshd-keygen
Type=oneshot


Note You need to log in before you can comment on or make changes to this bug.