Bug 676665

Summary: ssh pubkey auth and polyinstantiation
Product: Red Hat Enterprise Linux 6 Reporter: Stephan Mueller <smueller>
Component: opensshAssignee: Jan F. Chadima <jchadima>
Status: CLOSED ERRATA QA Contact: Miroslav Vadkerti <mvadkert>
Severity: high Docs Contact:
Priority: high    
Version: 6.0CC: ebenes, mgrepl, mvadkert, sgrubb, tmraz
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openssh-5.3p1-50.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 13:30:56 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 584498, 846801, 846802    

Description Stephan Mueller 2011-02-10 16:45:17 UTC
Description of problem:

ssh pubkey authentication together with polyinstantiation on an MLS system is 
currently not working as expected.

To be precise, pubkey auth works properly by sshd. But a user can NOT generate 
the authorized_keys file at the right place.

Note the following sequence considering that after initial configuration, 
there is no authorized_keys file yet:

1. user logs in

2. user generates his ~/.ssh/authorized_keys file

3. user logs out

4. user logs in using pubkey auth

5. sshd looks for $HOME/.ssh/authorized_keys file


In step 5, sshd can NOT find the authorized_keys file, because the file 
generated in step 2 is stored in the polyinstantiated backend dir in 
/home/home-inst/.... but the directory that sshd looks up is /home/<USER> as 
the sshd does not set up polyinstantiation at the time of looking up the 
user's key.

Therefore, sshd cannot find the users authorized_keys file. But the user 
cannot create the authorized_keys file in a spot where sshd can find it.

For test purposes, if root moves the .ssh/ directory from /home/home-inst/.../ 
to /home/<USER> and does a chown on them, pubkey auth works nicely.

So, pubkey auth in an MLS system must be considered broken.


Version-Release number of selected component (if applicable): 


How reproducible:

Always using the steps above.

Expected results:

key-based authentication should work in an MLS environment

Comment 9 Miroslav Vadkerti 2011-03-07 12:36:02 UTC
I was able to manually test the pubkey authentication with 

* enabled polyinstantiation of home directories
* MLS policy enabled 

The ssh-keycat works in this environment as expected I can login without issues.

Comment 10 Stephan Mueller 2011-03-07 12:49:39 UTC
> The ssh-keycat works in this environment as expected I can login without
> issues.

Do you have a working solution for the described problem? Can you please show us the code or design, because I fear easy MLS policy violations.

Comment 11 Miroslav Vadkerti 2011-03-07 13:13:05 UTC
Stephan, our developers came up with a solution for the problem. It consists of a authorized keys helper program (ssh-keycat) and a pam module for it.

This is the howto for the solution
======
The ssh-keycat retrieves the content of the ~/.ssh/authorized_keys
of an user in any environment. This includes environments with
polyinstantiation of home directories and SELinux MLS policy enabled.

To use ssh-keycat, set these options in /etc/ssh/sshd_config file:
        AuthorizedKeysCommand /usr/libexec/openssh/ssh-keycat
        AuthorizedKeysCommandRunAs root

Do not forget to enable public key authentication:
        PubkeyAuthentication yes
=====

We are currently contacting Irine to make the new packages available to you for testing.

Comment 12 Stephan Mueller 2011-03-07 13:27:24 UTC
> We are currently contacting Irine to make the new packages available to you for
> testing.

Not having seen any code, please note my concerns: polyinstantiation ensures proper separation of information generated by the same user but in different levels. There must not be any connections between these levels.

Any solution that copies the authorized keys file around naturally tries to circumvent this requirement. However, this circumvention must only be done for keys! Note that you can stuff anything in a file called authorized keys, even data not parsable by sshd. In addition, there may be comments and such.

Note that we must prevent information flow (that is any arbitrary, user-definable data) between the labels. That means we must only allow keys or commands to be copied around.

Comment 13 Jan F. Chadima 2011-03-07 18:49:27 UTC
> 
> Any solution that copies the authorized keys file around naturally tries to
> circumvent this requirement. However, this circumvention must only be done for
> keys! Note that you can stuff anything in a file called authorized keys, even
> data not parsable by sshd. In addition, there may be comments and such.

There are no data copying. The ssh-keycat program in the fly reads the data from the proper user/context's authorized keys and via stdout piped to sshd send it for verification.

Comment 14 Stephan Mueller 2011-03-07 19:36:17 UTC
> from the proper user/context's authorized keys and via stdout piped to sshd
> send it for verification.

That is good - this approach should not cause any problem with the MLS constraints.

Comment 15 Miroslav Vadkerti 2011-03-09 16:02:49 UTC
The howto is fixed now also. Moving back to ON_QA

Comment 16 Miroslav Vadkerti 2011-03-09 16:06:48 UTC
Stephan, we are still waiting for the refresh of the ftp to make the new bits available to you.

Comment 17 Stephan Mueller 2011-03-09 16:10:17 UTC
Thanks for the note - just ping me.

Comment 18 Miroslav Vadkerti 2011-03-23 08:40:06 UTC
Stephan, openssh-5.3p1-45.el6 should be available to you as it is included in the RHEL6.1 Beta compose. Could you please try to reproduce the problem using this package? The howto is here:
/usr/share/doc/openssh-keycat-5.3p1/HOWTO.ssh-keycat

Comment 19 Stephan Mueller 2011-03-23 09:38:00 UTC
Maybe I do not look at the right place, but the ISO image only contains the following files:

openssh-5.3p1-45.el6.x86_64.rpm
openssh-askpass-5.3p1-45.el6.x86_64.rpm
openssh-clients-5.3p1-45.el6.x86_64.rpm
openssh-server-5.3p1-45.el6.x86_64.rpm

Looking at the OpenSSH package spec file, I would need to expect some openssh-keycat package. Bottom line, when unpacking the above mentioned packages, I do not see any keycat related bits.

Where can I find the keycat package?

Comment 20 Stephan Mueller 2011-03-23 10:06:03 UTC
Some comments to the keycat patch:

+dump_keys():
+       if (!S_ISREG(st.st_mode) || 
+               (st.st_uid != pwd->pw_uid && st.st_uid != 0)) {
+               rv = ERR_FILE_MODE;
+               goto fail;
+       }

Shouldn't this check be consistent with the one from the sshd implementation (it also checks for permissions on .ssh/)? I would strongly suggest to apply the same security measures as in sshd for resolving .ssh and reading the authorized_keys file (see secure_filename function).

Does keycat have any SELinux privilege (I guess not due to the configuration of keycat in sshd_config - but I want to be sure)? I am worried about the following:

+       while ((len = fread(buf, 1, sizeof(buf), f)) > 0) {
+               rv = fwrite(buf, 1, len, stdout) != len ? ERR_WRITE : 0;
+       }

Another question: how can a user in a privsep environment generate the authorized_keys file in the non-polyinstantiated $HOME/.ssh directory?

Comment 21 Tomas Mraz 2011-03-23 10:19:16 UTC
(In reply to comment #20)
> Some comments to the keycat patch:
> 
> +dump_keys():
> +       if (!S_ISREG(st.st_mode) || 
> +               (st.st_uid != pwd->pw_uid && st.st_uid != 0)) {
> +               rv = ERR_FILE_MODE;
> +               goto fail;
> +       }
> 
> Shouldn't this check be consistent with the one from the sshd implementation
> (it also checks for permissions on .ssh/)? I would strongly suggest to apply
> the same security measures as in sshd for resolving .ssh and reading the
> authorized_keys file (see secure_filename function).
The problem is some of the secure_filename checks are only optional - you can switch them off in the sshd_config. ssh-keycat does not read the sshd configuration file as we wanted to keep it as small as possible and so it would be impossible to turn off the checks in it.

> Does keycat have any SELinux privilege (I guess not due to the configuration of
> keycat in sshd_config - but I want to be sure)? I am worried about the
> following:
> 
> +       while ((len = fread(buf, 1, sizeof(buf), f)) > 0) {
> +               rv = fwrite(buf, 1, len, stdout) != len ? ERR_WRITE : 0;
> +       }
What is the problem here? Can you be more specific?

> Another question: how can a user in a privsep environment generate the
> authorized_keys file in the non-polyinstantiated $HOME/.ssh directory?
I am sorry but I do not understand this question.

Comment 22 Miroslav Vadkerti 2011-03-23 10:20:54 UTC
Stephan, the package is in the optional tree. I'm checking with release engineering if this is available also to the partners. I though it is :(

Comment 23 Stephan Mueller 2011-03-23 10:29:58 UTC
(In reply to comment #22)
> Stephan, the package is in the optional tree. I'm checking with release
> engineering if this is available also to the partners. I though it is :(

Found it in the optional tree - thanks for the pointer. Are there plans to move it to the main ISO? I will need another day or two before I can test it.

Comment 24 Stephan Mueller 2011-03-23 10:37:54 UTC
(In reply to comment #21)

> The problem is some of the secure_filename checks are only optional - you can
> switch them off in the sshd_config. ssh-keycat does not read the sshd
> configuration file as we wanted to keep it as small as possible and so it would
> be impossible to turn off the checks in it.

Hm, ok. However, I have to think a bit longer about that check and whether it is sufficient. My problem is that a fully privileged application reads data from a path name that is under the full control of the user. Granted, the output is fed into sshd and sshd tries to interpret the bits as an authorized_keys file. But since sshd does some magic beyond just checking the keys (think of the potential commands in authorized_keys), I need to think whether there is a privilege escalation possible here (privilege escalation means: access something with a different UID/GID, different label or different capability).
> 

> What is the problem here? Can you be more specific?

See above.
> 
> > Another question: how can a user in a privsep environment generate the
> > authorized_keys file in the non-polyinstantiated $HOME/.ssh directory?
> I am sorry but I do not understand this question.

Skip that question for the moment in favor of the following: you use PAM with keycat. I guess the main intention here is to allow pam_namespace to be invoked, right? What is the expected/proposed PAM config for keycat?

Comment 25 Stephan Mueller 2011-03-23 10:42:29 UTC
> 
> Skip that question for the moment in favor of the following: you use PAM with
> keycat. I guess the main intention here is to allow pam_namespace to be
> invoked, right? What is the expected/proposed PAM config for keycat?

Maybe I should check the RPM before I ask as there is the answer for a PAM config. Sorry for the noise. :-(

Comment 26 Miroslav Vadkerti 2011-03-23 10:44:05 UTC
Stephan so do you already have access to the openssh-keycat package?

Comment 27 Stephan Mueller 2011-03-23 10:47:39 UTC
(In reply to comment #26)
> Stephan so do you already have access to the openssh-keycat package?

Yes, I do.

Comment 28 Tomas Mraz 2011-03-23 12:27:21 UTC
(In reply to comment #24)
> (In reply to comment #21)
> 
> > The problem is some of the secure_filename checks are only optional - you can
> > switch them off in the sshd_config. ssh-keycat does not read the sshd
> > configuration file as we wanted to keep it as small as possible and so it would
> > be impossible to turn off the checks in it.
> 
> Hm, ok. However, I have to think a bit longer about that check and whether it
> is sufficient. My problem is that a fully privileged application reads data
> from a path name that is under the full control of the user. Granted, the
> output is fed into sshd and sshd tries to interpret the bits as an
> authorized_keys file. But since sshd does some magic beyond just checking the
> keys (think of the potential commands in authorized_keys), I need to think
> whether there is a privilege escalation possible here (privilege escalation
> means: access something with a different UID/GID, different label or different
> capability).

Note that there is the temporarily_use_uid() call before the open() call. This means that the effective uid as well as the groups will be set to the user. This avoids any symlinking and other race attacks where the user could make ssh-keycat read files that he does not have an access to.

Also note that if ssh-keycat is not used the sshd directly reads/parses/uses the .ssh/authorized_keys file and this file can be fully manipulated by the user - so there is no change in possible attack vectors against the sshd. Of course in this case the code of ssh-keycat could be attacked as well but as you can see all the manipulation that it does with the file contents is extremely simple - it just reads it and write to stdout.

Comment 29 Eduard Benes 2011-03-23 12:32:25 UTC
(In reply to comment #23)
> (In reply to comment #22)
> > Stephan, the package is in the optional tree. I'm checking with release
> > engineering if this is available also to the partners. I though it is :(
> 
> Found it in the optional tree - thanks for the pointer. Are there plans to move
> it to the main ISO? I will need another day or two before I can test it.

Filed bug 690127 to track the requested change and move it out of optionals.

Comment 30 Stephan Mueller 2011-03-23 12:43:24 UTC
(In reply to comment #28)

> 
> Note that there is the temporarily_use_uid() call before the open() call. This

Thanks, that is a good hint. I overlooked that one.

> means that the effective uid as well as the groups will be set to the user.
> This avoids any symlinking and other race attacks where the user could make
> ssh-keycat read files that he does not have an access to.
> 
> Also note that if ssh-keycat is not used the sshd directly reads/parses/uses
> the .ssh/authorized_keys file and this file can be fully manipulated by the
> user - so there is no change in possible attack vectors against the sshd. Of
> course in this case the code of ssh-keycat could be attacked as well but as you
> can see all the manipulation that it does with the file contents is extremely
> simple - it just reads it and write to stdout.

With the set*uid you mentioned above I think we can call my concerns closed. :-)

Comment 32 Stephan Mueller 2011-03-28 10:08:44 UTC
With the version of OpenSSH packages that supposedly have the fixes, I still cannot log in using keys on an MLS system.

When starting sshd in debug mode, I see:

lookup of /home/eal/.ssh/authorized_keys and /home/eal/.ssh/authorized_keys2 which both fail. There is no indication that the ssh-keycat was spawned. Also the audit log does not indicate the starting of ssh-keycat.



Home directory of user eal:
[eal/staff_r/SystemLow@rhel ~]$ ls -la
total 32
drwx------. 4 eal  eal  4096 Mar 28 04:50 .
drwxr-xr-x. 5 root root 4096 Mar 28 04:26 ..
-rw-------. 1 eal  eal    18 Mar 28 04:50 .bash_history
-rw-r--r--. 1 eal  eal    18 Mar 28 04:35 .bash_logout
-rw-r--r--. 1 eal  eal   176 Mar 28 04:35 .bash_profile
-rw-r--r--. 1 eal  eal   124 Mar 28 04:35 .bashrc
drwx------. 2 eal  eal  4096 Mar 28 05:04 .screen
drwx------. 2 eal  eal  4096 Mar 28 04:48 .ssh
[eal/staff_r/SystemLow@rhel ~]$

Contents of ~eal/.ssh:
[eal/staff_r/SystemLow@rhel ~]$ ls -la .ssh/
total 12
drwx------. 2 eal eal 4096 Mar 28 04:48 .
drwx------. 4 eal eal 4096 Mar 28 04:50 ..
-rw-------. 1 eal eal 1115 Mar 28 04:48 authorized_keys
[eal/staff_r/SystemLow@rhel ~]$


Packages installed:
rpm -qa | grep openssh
openssh-server-5.3p1-45.el6.x86_64
openssh-5.3p1-45.el6.x86_64
openssh-clients-5.3p1-45.el6.x86_64
openssh-keycat-5.3p1-45.el6.x86_64

sshd_config (critical part):

# Configure password-based login. If password-based
# authentication is enabled, you MUST use the PAM
# library exclusively.
# You MAY disable UsePAM by setting both variables to "no"
UsePAM yes
ChallengeResponseAuthentication yes

# You MAY disable key-based authentication
PubkeyAuthentication yes

# The ssh-keycat retrieves the content of the ~/.ssh/authorized_keys
# of an user in any environment. This includes environments with
# polyinstantiation of home directories and SELinux MLS policy enabled.
# You MAY disable the following lines - however, key-based authentication
# will NOT work in an MLS environment.
AuthorizedKeysCommand "/usr/libexec/openssh/ssh-keycat"
AuthorizedKeysCommandRunAs root

Comment 33 Miroslav Vadkerti 2011-03-28 10:29:57 UTC
Strange, trying to reproduce again and will report back ASAP

Comment 34 Miroslav Vadkerti 2011-03-28 12:50:01 UTC
Stephan could you please try remove the double quotes around the command please?

AuthorizedKeysCommand /usr/libexec/openssh/ssh-keycat

Comment 35 Miroslav Vadkerti 2011-03-28 12:55:08 UTC
With the quotes this doesn't work, we will need to fix the docs, sorry for this.

Jan could you please remove the bogus double quotes from the README?

Comment 36 Stephan Mueller 2011-03-28 12:58:46 UTC
Yes, without the double quotes it works. I can log in using a key. Thanks for clarifying

Comment 39 errata-xmlrpc 2011-05-19 13:30:56 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0598.html