I upgraded one of my Fedora 18 VMs to Fedora 19, and now pam_afs_session no longer auto-detects the Kerberos cache when I log into SSH with GSSAPI authentication. The problem is that the $KRB5CCNAME variable is no longer available to PAM modules in the session stack. I can demonstrate this apart from pam_afs_session, with a small shell script and pam_exec. On F19: # grep ^session /etc/pam.d/password-auth session optional pam_keyinit.so revoke session required pam_limits.so session optional pam_mkhomedir.so session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_sss.so session optional pam_exec.so seteuid /var/local/bin/logk session optional pam_afs_session.so debug And the /var/local/bin/logk script is simply: $ cat /var/local/bin/logk #!/bin/bash output=${KRB5CCNAME-no cache found} logger $output With this setup on Fedora 18, pam_exec creates a syslog entry with a $KRB5CCNAME: Jul 24 00:52:32 afsdb kdreyer: DIR::/tmp/krb5cc_1000_9ZXqY8w0hN/tkt Whereas on Fedora 19, the cache is always missing on login. Jul 24 01:04:03 afs1 kdreyer: no cache found I expected to see something like this on F19: Jul 24 01:04:00 afs1 kdreyer: DIR:/run/user/1000/krb5cc_XPTgiapnY8 (This is problematic for pam_afs_session because it will bail out if $KRB5CCNAME is missing.) I don't know what exactly is supposed to set $KRB5CCNAME in PAM, so please feel free to re-assign this bug to the appropriate component if necessary.
I discussed this today with sgallagh in #sssd in IRC. This same problem occurs on my Rawhide VM (openssh-server-6.2p2-3.fc20) as well.
Created attachment 777955 [details] IRC triage log Attaching a log from IRC where we identified that the problem is that the /run/user/UID directory is not available early enough in the process when logging in via GSSAPI.
I confirmed that downgrading my F19 box to openssh-server-6.2p2-2 allows pam_exec to print the KRB5CCNAME in the shell script, and pam_afs_session works again.
(In reply to Ken Dreyer from comment #3) > I confirmed that downgrading my F19 box to openssh-server-6.2p2-2 allows > pam_exec to print the KRB5CCNAME in the shell script, and pam_afs_session > works again. Looking at the changelog: * Mon Jun 17 2013 Petr Lautrbach <plautrba> 6.2p2-3 + 0.9.3-5 - move default gssapi cache to /run/user/<uid> (#848228) So you're not fixing the problem, you're just changing it back to using /tmp for the credential cache (which is not really a fix). There are several reasons we are trying to move the credcache out of /tmp: 1) The location of the cache cannot be /tmp, because we are aiming to make /tmp namespaced by default 2) Applications like AFS and GSSD need to be able to have a predictable place to find valid credentials. The traditional behavior of scanning through /tmp was slow, time-consuming and prone to bugs. 3) The default location of the cache should be on tmpfs in order to ensure that the credentials cannot be used if someone steals a hard drive. I'm pretty much entirely convinced at this point that this is just yet-another aspect of the continuing drama that XDG_RUNTIME_DIR is not available early enough. I'm currently engaged in discussions with the systemd/logind folks to resolve this. It's a blocker for many pieces of the Kerberos discussion.
It seems to be really an issue of openssh-6.2p2-3 update. Unfortunately, I added a change which change the order of starting pam session and storing kerberos credentials. This change was needed if pam_namespace and polyinstantiated /tmp directories was used. Otherwise, a user can't see her credentials since it's created in original /tmp while the user has her own /tmp created during pam session initialization. Please try following build with the change reverted http://koji.fedoraproject.org/koji/taskinfo?taskID=5677270
(In reply to Petr Lautrbach from comment #5) > > This change was needed if pam_namespace and polyinstantiated /tmp > directories was used. Otherwise, a user can't see her credentials since it's > created in original /tmp while the user has her own /tmp created during pam > session initialization. Indeed. The reason is in bug 974096. I wonder if it would be possible to accommodate both situations -- create the credentials before session and after session check if it still exists and if it does not exist, assume polyinstantiation and create it once more, in the new directory.
It is worth noting that on F19 and later, the default place to create Kerberos credentials is no longer in /tmp, but is in /run/user/$UID/krb5cc, which should not be affected by the polyinstantiated /tmp problem (it is shared with the user session). Note: we are currently considering adjusting the actual location of this directory, but as long as openssh is relying on the libkrb5 default, it should be transparent.
(In reply to Stephen Gallagher from comment #7) > It is worth noting that on F19 and later, the default place to create > Kerberos credentials is no longer in /tmp, but is in /run/user/$UID/krb5cc, > which should not be affected by the polyinstantiated /tmp problem (it is > shared with the user session). There is still a case when openssh can't use /run/user/$UID. This directory is created during pam session and the credentials need to be stored before. And openssh can't create this directory as the particular code is running unprivileged. There's now a fallback in openssh, which sets cache directory to /tmp if /run/user/$UID doesn't exist yet. > Note: we are currently considering adjusting the actual location of this > directory, but as long as openssh is relying on the libkrb5 default, it > should be transparent. If openssh uses pam_krb5 for authentication then yes. But for GSSAPI authentication - gssapi-with-mic - it uses "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" resp "DIR:/tmp/krb5cc_%d_XXXXXXXXXX"
(In reply to Petr Lautrbach from comment #8) > (In reply to Stephen Gallagher from comment #7) > > It is worth noting that on F19 and later, the default place to create > > Kerberos credentials is no longer in /tmp, but is in /run/user/$UID/krb5cc, > > which should not be affected by the polyinstantiated /tmp problem (it is > > shared with the user session). > > There is still a case when openssh can't use /run/user/$UID. This directory > is created during pam session and the credentials need to be stored before. > And openssh can't create this directory as the particular code is running > unprivileged. There's now a fallback in openssh, which sets cache directory > to /tmp if /run/user/$UID doesn't exist yet. > Please see: https://lists.fedoraproject.org/pipermail/devel/2013-July/186930.html We are currently planning to solve this problem by moving to /run/kerberos/$UID and having this directory created by a helper triggered by libkrb5. This will solve the availability problem where /run/user/$UID wasn't created until session phase. Will that work for you? > > Note: we are currently considering adjusting the actual location of this > > directory, but as long as openssh is relying on the libkrb5 default, it > > should be transparent. > > If openssh uses pam_krb5 for authentication then yes. But for GSSAPI > authentication - gssapi-with-mic - it uses > "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" resp "DIR:/tmp/krb5cc_%d_XXXXXXXXXX" pam_krb5 should be irrelevant. What I am asking is if you are *setting* the KRB5CCNAME variable to "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" before calling libkrb5? I suspect that you are, since you are apparently setting a mkstemp()-style random string at the end of it[*]. So please keep an eye on the thread I posted above, because it will probably be preferable to move that to use the new /run/kerberos/$UID directory. [*] Why are you setting a random value here? My guess is that you want to address the case where two users SSH into the same machine and account from different remote machines and accounts and want to keep their caches separate? That's probably sensible, although if you have this situation occurring, it's absolutely trivial for either user to steal the other's credentials.
(In reply to Stephen Gallagher from comment #9) > We are currently planning to solve this problem by moving to > /run/kerberos/$UID and having this directory created by a helper triggered > by libkrb5. This will solve the availability problem where /run/user/$UID > wasn't created until session phase. Will that work for you? If the cache or its directory are created by libkrb5 during krb5_cc_resolve() or similar function then it will be ok. > > If openssh uses pam_krb5 for authentication then yes. But for GSSAPI > > authentication - gssapi-with-mic - it uses > > "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" resp "DIR:/tmp/krb5cc_%d_XXXXXXXXXX" > > pam_krb5 should be irrelevant. What I am asking is if you are *setting* the > KRB5CCNAME variable to "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" before calling > libkrb5? I suspect that you are, since you are apparently setting a > mkstemp()-style random string at the end of it[*]. So please keep an eye on > the thread I posted above, because it will probably be preferable to move > that to use the new /run/kerberos/$UID directory. The upstream's openssh creates ccname base on cctemplate "FILE:/tmp/krb5cc_%d_XXXXXXXXXX" and calls krb5_cc_resolve(ctx, ccname, ccache). I changed that to use "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" in F19 according to the feature page. Today, I found out that I could probably use krb5_cc_default_name(ctx) to generate ccname based on the system configuration in /etc/krb5.conf. But the default value for default_ccache_name is DIR:/run/user/%{uid}/krb5cc where %{uid} stands for unix real uid. For sshd it means that krb5_cc_default_name() will always return "DIR:/run/user/0/krb5cc" . So it's not usable and it still needs to generate ccname using an effective uid. > > [*] Why are you setting a random value here? My guess is that you want to > address the case where two users SSH into the same machine and account from > different remote machines and accounts and want to keep their caches > separate? That's probably sensible, although if you have this situation > occurring, it's absolutely trivial for either user to steal the other's > credentials. There were some concerns about accidentally cleaned caches by a second session and also some concerns about using MLS SELinux policy. But if it's possible to configure the default ccname then it would be acceptable to use single cache by default with possibility to change it for the administrator.
(In reply to Petr Lautrbach from comment #10) > (In reply to Stephen Gallagher from comment #9) > The upstream's openssh creates ccname base on cctemplate > "FILE:/tmp/krb5cc_%d_XXXXXXXXXX" and calls krb5_cc_resolve(ctx, ccname, > ccache). > > I changed that to use "DIR:/run/user/%d/krb5cc_XXXXXXXXXX" in F19 according > to > the feature page. > > Today, I found out that I could probably use krb5_cc_default_name(ctx) to > generate > ccname based on the system configuration in /etc/krb5.conf. But the default > value for default_ccache_name is DIR:/run/user/%{uid}/krb5cc where %{uid} > stands for unix real uid. For sshd it means that krb5_cc_default_name() > will always return > "DIR:/run/user/0/krb5cc" . So it's not usable and it still needs to generate > ccname using an effective uid. Any reason why you can't use the default credentials after ssh setuids to %{uid} ? Is it because authentication is run as uid = 0 ? > > > > [*] Why are you setting a random value here? My guess is that you want to > > address the case where two users SSH into the same machine and account from > > different remote machines and accounts and want to keep their caches > > separate? That's probably sensible, although if you have this situation > > occurring, it's absolutely trivial for either user to steal the other's > > credentials. > > There were some concerns about accidentally cleaned caches by a second > session > and also some concerns about using MLS SELinux policy. > But if it's possible to configure the default ccname then it would be > acceptable to use single cache by default with possibility to change it for > the administrator. Yes, using the default cache by default would be desirable I think.
(In reply to Simo Sorce from comment #11) > Any reason why you can't use the default credentials after ssh setuids to > %{uid} ? > Is it because authentication is run as uid = 0 ? pam session needs to be opened with effective uid sufficent to perform such tasks as creating or mounting the user's home directory and at this time the cache needs to be already prepared. So the cache is resolved using the user's effective id and root's real id.
openssh-6.2p2-5.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/openssh-6.2p2-5.fc19
I've confirmed that openssh-6.2p2-5.fc19 fixes pam_afs_session. Thank you for this update.
openssh-6.2p2-5.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.