Bug 987792 - KRB5CCNAME is not set in PAM session with GSSAPI SSH auth
KRB5CCNAME is not set in PAM session with GSSAPI SSH auth
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: openssh (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Petr Lautrbach
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-24 03:44 EDT by Ken Dreyer
Modified: 2013-08-05 10:55 EDT (History)
10 users (show)

See Also:
Fixed In Version: openssh-6.2p2-5.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-05 10:55:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
IRC triage log (4.76 KB, text/plain)
2013-07-24 15:28 EDT, Stephen Gallagher
no flags Details

  None (edit)
Description Ken Dreyer 2013-07-24 03:44:07 EDT
I upgraded one of my Fedora 18 VMs to Fedora 19, and now pam_afs_session no longer auto-detects the Kerberos cache when I log into SSH with GSSAPI authentication.

The problem is that the $KRB5CCNAME variable is no longer available to PAM modules in the session stack.

I can demonstrate this apart from pam_afs_session, with a small shell script and pam_exec. On F19:

  # grep ^session /etc/pam.d/password-auth
  session     optional      pam_keyinit.so revoke
  session     required      pam_limits.so
  session     optional      pam_mkhomedir.so
  session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
  session     required      pam_unix.so
  session     optional      pam_sss.so
  session     optional      pam_exec.so seteuid /var/local/bin/logk
  session     optional      pam_afs_session.so debug

And the /var/local/bin/logk script is simply:

  $ cat /var/local/bin/logk
  #!/bin/bash
  output=${KRB5CCNAME-no cache found}
  logger $output

With this setup on Fedora 18, pam_exec creates a syslog entry with a $KRB5CCNAME:

  Jul 24 00:52:32 afsdb kdreyer: DIR::/tmp/krb5cc_1000_9ZXqY8w0hN/tkt

Whereas on Fedora 19, the cache is always missing on login.

  Jul 24 01:04:03 afs1 kdreyer: no cache found

I expected to see something like this on F19:

  Jul 24 01:04:00 afs1 kdreyer: DIR:/run/user/1000/krb5cc_XPTgiapnY8

(This is problematic for pam_afs_session because it will bail out if $KRB5CCNAME is missing.)

I don't know what exactly is supposed to set $KRB5CCNAME in PAM, so please feel free to re-assign this bug to the appropriate component if necessary.
Comment 1 Ken Dreyer 2013-07-24 15:13:57 EDT
I discussed this today with sgallagh in #sssd in IRC. This same problem occurs on my Rawhide VM (openssh-server-6.2p2-3.fc20) as well.
Comment 2 Stephen Gallagher 2013-07-24 15:28:27 EDT
Created attachment 777955 [details]
IRC triage log

Attaching a log from IRC where we identified that the problem is that the /run/user/UID directory is not available early enough in the process when logging in via GSSAPI.
Comment 3 Ken Dreyer 2013-07-25 02:24:48 EDT
I confirmed that downgrading my F19 box to openssh-server-6.2p2-2 allows pam_exec to print the KRB5CCNAME in the shell script, and pam_afs_session works again.
Comment 4 Stephen Gallagher 2013-07-25 07:39:04 EDT
(In reply to Ken Dreyer from comment #3)
> I confirmed that downgrading my F19 box to openssh-server-6.2p2-2 allows
> pam_exec to print the KRB5CCNAME in the shell script, and pam_afs_session
> works again.

Looking at the changelog:

* Mon Jun 17 2013 Petr Lautrbach <plautrba@redhat.com> 6.2p2-3 + 0.9.3-5 - move default gssapi cache to /run/user/<uid> (#848228)


So you're not fixing the problem, you're just changing it back to using /tmp for the credential cache (which is not really a fix). There are several reasons we are trying to move the credcache out of /tmp:

1) The location of the cache cannot be /tmp, because we are aiming to
make /tmp namespaced by default
2) Applications like AFS and GSSD need to be able to have a
predictable place to find valid credentials. The traditional behavior
of scanning through /tmp was slow, time-consuming and prone to bugs.
3) The default location of the cache should be on tmpfs in order to
ensure that the credentials cannot be used if someone steals a hard drive.

I'm pretty much entirely convinced at this point that this is just yet-another aspect of the continuing drama that XDG_RUNTIME_DIR is not available early enough. I'm currently engaged in discussions with the systemd/logind folks to resolve this. It's a blocker for many pieces of the Kerberos discussion.
Comment 5 Petr Lautrbach 2013-07-30 06:42:32 EDT
It seems to be really an issue of openssh-6.2p2-3 update. Unfortunately, I added a change which change the order of starting pam session and storing kerberos credentials. 

This change was needed if pam_namespace and polyinstantiated /tmp directories was used. Otherwise, a user can't see her credentials since it's created in original /tmp while the user has her own /tmp created during pam session initialization.

Please try following build with the change reverted
http://koji.fedoraproject.org/koji/taskinfo?taskID=5677270
Comment 6 Jan Pazdziora 2013-07-30 06:53:37 EDT
(In reply to Petr Lautrbach from comment #5)
> 
> This change was needed if pam_namespace and polyinstantiated /tmp
> directories was used. Otherwise, a user can't see her credentials since it's
> created in original /tmp while the user has her own /tmp created during pam
> session initialization.

Indeed. The reason is in bug 974096.

I wonder if it would be possible to accommodate both situations -- create the credentials before session and after session check if it still exists and if it does not exist, assume polyinstantiation and create it once more, in the new directory.
Comment 7 Stephen Gallagher 2013-07-30 08:14:25 EDT
It is worth noting that on F19 and later, the default place to create Kerberos credentials is no longer in /tmp, but is in /run/user/$UID/krb5cc, which should not be affected by the polyinstantiated /tmp problem (it is shared with the user session).

Note: we are currently considering adjusting the actual location of this directory, but as long as openssh is relying on the libkrb5 default, it should be transparent.
Comment 8 Petr Lautrbach 2013-07-30 08:39:31 EDT
(In reply to Stephen Gallagher from comment #7)
> It is worth noting that on F19 and later, the default place to create
> Kerberos credentials is no longer in /tmp, but is in /run/user/$UID/krb5cc,
> which should not be affected by the polyinstantiated /tmp problem (it is
> shared with the user session).

There is still a case when openssh can't use /run/user/$UID. This directory is created during pam session and the credentials need to be stored before. And openssh can't create this directory as the particular code is running unprivileged. There's now a fallback in openssh, which sets cache directory to /tmp if /run/user/$UID doesn't exist yet. 

> Note: we are currently considering adjusting the actual location of this
> directory, but as long as openssh is relying on the libkrb5 default, it
> should be transparent.

If openssh uses pam_krb5 for authentication then yes. But for GSSAPI authentication - gssapi-with-mic - it uses "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" resp "DIR:/tmp/krb5cc_%d_XXXXXXXXXX"
Comment 9 Stephen Gallagher 2013-07-30 08:49:40 EDT
(In reply to Petr Lautrbach from comment #8)
> (In reply to Stephen Gallagher from comment #7)
> > It is worth noting that on F19 and later, the default place to create
> > Kerberos credentials is no longer in /tmp, but is in /run/user/$UID/krb5cc,
> > which should not be affected by the polyinstantiated /tmp problem (it is
> > shared with the user session).
> 
> There is still a case when openssh can't use /run/user/$UID. This directory
> is created during pam session and the credentials need to be stored before.
> And openssh can't create this directory as the particular code is running
> unprivileged. There's now a fallback in openssh, which sets cache directory
> to /tmp if /run/user/$UID doesn't exist yet. 
> 

Please see: https://lists.fedoraproject.org/pipermail/devel/2013-July/186930.html

We are currently planning to solve this problem by moving to /run/kerberos/$UID and having this directory created by a helper triggered by libkrb5. This will solve the availability problem where /run/user/$UID wasn't created until session phase. Will that work for you?

> > Note: we are currently considering adjusting the actual location of this
> > directory, but as long as openssh is relying on the libkrb5 default, it
> > should be transparent.
> 
> If openssh uses pam_krb5 for authentication then yes. But for GSSAPI
> authentication - gssapi-with-mic - it uses
> "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" resp "DIR:/tmp/krb5cc_%d_XXXXXXXXXX"

pam_krb5 should be irrelevant. What I am asking is if you are *setting* the KRB5CCNAME variable to "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" before calling libkrb5? I suspect that you are, since you are apparently setting a mkstemp()-style random string at the end of it[*]. So please keep an eye on the thread I posted above, because it will probably be preferable to move that to use the new /run/kerberos/$UID directory.

[*] Why are you setting a random value here? My guess is that you want to address the case where two users SSH into the same machine and account from different remote machines and accounts and want to keep their caches separate? That's probably sensible, although if you have this situation occurring, it's absolutely trivial for either user to steal the other's credentials.
Comment 10 Petr Lautrbach 2013-07-30 11:50:01 EDT
(In reply to Stephen Gallagher from comment #9)
> We are currently planning to solve this problem by moving to
> /run/kerberos/$UID and having this directory created by a helper triggered
> by libkrb5. This will solve the availability problem where /run/user/$UID
> wasn't created until session phase. Will that work for you?

If the cache or its directory are created by libkrb5 during krb5_cc_resolve() or similar function then it will be ok.

> > If openssh uses pam_krb5 for authentication then yes. But for GSSAPI
> > authentication - gssapi-with-mic - it uses
> > "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" resp "DIR:/tmp/krb5cc_%d_XXXXXXXXXX"
> 
> pam_krb5 should be irrelevant. What I am asking is if you are *setting* the
> KRB5CCNAME variable to "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" before calling
> libkrb5? I suspect that you are, since you are apparently setting a
> mkstemp()-style random string at the end of it[*]. So please keep an eye on
> the thread I posted above, because it will probably be preferable to move
> that to use the new /run/kerberos/$UID directory.

The upstream's openssh creates ccname base on cctemplate 
"FILE:/tmp/krb5cc_%d_XXXXXXXXXX" and calls krb5_cc_resolve(ctx, ccname, ccache).

I changed that to use "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" in F19 according to
the feature page. 

Today, I found out that I could probably use krb5_cc_default_name(ctx) to generate
ccname based on the system configuration in /etc/krb5.conf. But the default value for default_ccache_name is DIR:/run/user/%{uid}/krb5cc where %{uid} stands for unix real uid. For sshd it means that  krb5_cc_default_name() will always return
"DIR:/run/user/0/krb5cc" . So it's not usable and it still needs to generate ccname using an effective uid.

> 
> [*] Why are you setting a random value here? My guess is that you want to
> address the case where two users SSH into the same machine and account from
> different remote machines and accounts and want to keep their caches
> separate? That's probably sensible, although if you have this situation
> occurring, it's absolutely trivial for either user to steal the other's
> credentials.

There were some concerns about accidentally cleaned caches by a second session
and also some concerns about using MLS SELinux policy. 
But if it's possible to configure the default ccname then it would be acceptable to use single cache by default with possibility to change it for the administrator.
Comment 11 Simo Sorce 2013-07-30 16:01:59 EDT
(In reply to Petr Lautrbach from comment #10)
> (In reply to Stephen Gallagher from comment #9)

> The upstream's openssh creates ccname base on cctemplate 
> "FILE:/tmp/krb5cc_%d_XXXXXXXXXX" and calls krb5_cc_resolve(ctx, ccname,
> ccache).
> 
> I changed that to use "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" in F19 according
> to
> the feature page. 
> 
> Today, I found out that I could probably use krb5_cc_default_name(ctx) to
> generate
> ccname based on the system configuration in /etc/krb5.conf. But the default
> value for default_ccache_name is DIR:/run/user/%{uid}/krb5cc where %{uid}
> stands for unix real uid. For sshd it means that  krb5_cc_default_name()
> will always return
> "DIR:/run/user/0/krb5cc" . So it's not usable and it still needs to generate
> ccname using an effective uid.

Any reason why you can't use the default credentials after ssh setuids to %{uid} ?
Is it because authentication is run as uid = 0 ?

> > 
> > [*] Why are you setting a random value here? My guess is that you want to
> > address the case where two users SSH into the same machine and account from
> > different remote machines and accounts and want to keep their caches
> > separate? That's probably sensible, although if you have this situation
> > occurring, it's absolutely trivial for either user to steal the other's
> > credentials.
> 
> There were some concerns about accidentally cleaned caches by a second
> session
> and also some concerns about using MLS SELinux policy. 
> But if it's possible to configure the default ccname then it would be
> acceptable to use single cache by default with possibility to change it for
> the administrator.

Yes, using the default cache by default would be desirable I think.
Comment 12 Petr Lautrbach 2013-07-31 05:59:40 EDT
(In reply to Simo Sorce from comment #11)
> Any reason why you can't use the default credentials after ssh setuids to
> %{uid} ?
> Is it because authentication is run as uid = 0 ?

pam session needs to be opened with effective uid sufficent to perform such tasks as creating or mounting the user's home directory and at this time the cache needs to be already prepared. So the cache is resolved using the user's effective id and root's real id.
Comment 13 Fedora Update System 2013-08-01 04:12:04 EDT
openssh-6.2p2-5.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/openssh-6.2p2-5.fc19
Comment 14 Ken Dreyer 2013-08-01 23:55:58 EDT
I've confirmed that openssh-6.2p2-5.fc19 fixes pam_afs_session. Thank you for this update.
Comment 15 Fedora Update System 2013-08-03 20:13:50 EDT
openssh-6.2p2-5.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.