Bug 985016

+++ This bug was initially created as a clone of Bug #961235 +++

I'm trying to 'kinit -k' as root, and I get this message about my credential cache directory not existing. 

[stef@stef ~]$ sudo kinit Administrator.LAN
kinit: Credential cache directory /run/user/0/krb5cc does not exist while getting default ccache

Name        : krb5-workstation
Arch        : x86_64
Version     : 1.11.2
Release     : 4.fc19
Size        : 2.3 M

--- Additional comment from Patrik Kis on 2013-05-09 04:31:53 EDT ---

I hit this bug by doing kinit as non-root user:

$ kinit Bender.QE
kinit: Credential cache directory /run/user/1000/krb5cc does not exist while getting default ccache

Root works:

# kinit Bender.QE
Password for Bender.QE: 
# klist 
Ticket cache: DIR::/run/user/0/krb5cc/tktxJkaGT
Default principal: Bender.QE

Valid starting       Expires              Service principal
05/09/2013 10:30:40  05/09/2013 20:30:40  krbtgt/AD.BASEOS.QE.QE
	renew until 05/16/2013 10:30:40

Kerberos:
krb5-1.11.2-4.fc19

--- Additional comment from Aleš Mareček on 2013-05-09 05:44:11 EDT ---

Workaround:
mkdir /run/usr/1000/krb5cc
chown -R <user><user> /run/usr/1000/krb5cc
kinit <user>@<kdc>
<password>

--- Additional comment from Aleš Mareček on 2013-05-09 05:44:52 EDT ---

*chown -R <user>:<user> ...

--- Additional comment from Patrik Kis on 2013-05-09 07:46:27 EDT ---

[root@pkisf19 ~]# rm -rf /run/user/0/
[root@pkisf19 ~]# 
[root@pkisf19 ~]# kinit Administrator.QE
kinit: Credential cache directory /run/user/0/krb5cc does not exist while getting default ccache

--- Additional comment from Nalin Dahyabhai on 2013-05-09 11:43:14 EDT ---

(In reply to comment #1)
> I hit this bug by doing kinit as non-root user:
> 
> $ kinit Bender.QE
> kinit: Credential cache directory /run/user/1000/krb5cc does not exist while
> getting default ccache

How was your shell started?  Under most conditions I'd expect pam_systemd to have created /run/user/$UID as part of setting up the user's session.

(In reply to comment #4)
> [root@pkisf19 ~]# rm -rf /run/user/0/

$XDG_RUNTIME_DIR typically points to this directory.  If it does, it's expected to be there.

--- Additional comment from Nalin Dahyabhai on 2013-05-09 13:27:10 EDT ---

(In reply to comment #0)
> I'm trying to 'kinit -k' as root, and I get this message about my credential
> cache directory not existing. 
> 
> [stef@stef ~]$ sudo kinit Administrator.LAN
> kinit: Credential cache directory /run/user/0/krb5cc does not exist while
> getting default ccache

This one's thornier, as sudo doesn't open a session unless the -i flag is used, so pam_systemd wouldn't always be run if we added it to the PAM configuration for sudo.  If it did, the per-user temporary directory would also, by default, be cleaned up immediately after kinit exited.  But then pam_systemd also registers the login session using the process's loginuid, which is that of the invoking user (and which I'm guessing isn't supposed to be changed by sudo).

I don't see a general solution to this that makes this work for sudo along with other things; we'll probably have to patch sudo specifically.

--- Additional comment from Nalin Dahyabhai on 2013-05-09 14:04:03 EDT ---

(In reply to comment #6)
> sudo doesn't open a session unless the -i flag is used

Whoops, yes it does.

--- Additional comment from Simo Sorce on 2013-05-09 19:01:39 EDT ---

(In reply to comment #4)
> [root@pkisf19 ~]# rm -rf /run/user/0/
> [root@pkisf19 ~]# 
> [root@pkisf19 ~]# kinit Administrator.QE
> kinit: Credential cache directory /run/user/0/krb5cc does not exist while
> getting default ccache


krb5 1.10 will not create the ccdir for you, 1.11 should do it.

--- Additional comment from Nalin Dahyabhai on 2013-05-10 15:54:21 EDT ---

(In reply to comment #8)
> (In reply to comment #4)
> > [root@pkisf19 ~]# rm -rf /run/user/0/
> > [root@pkisf19 ~]# 
> > [root@pkisf19 ~]# kinit Administrator.QE
> > kinit: Credential cache directory /run/user/0/krb5cc does not exist while
> > getting default ccache
> 
> krb5 1.10 will not create the ccdir for you, 1.11 should do it.

Even 1.11 will only create the directory itself (/run/user/0/krb5cc).  For most users, though, removing /run/user/$UID isn't going to be allowed by filesystem permissions, so I'm less concerned about it being removed than it not being created in the first place.

--- Additional comment from Patrik Kis on 2013-05-14 11:10:10 EDT ---

Sorry for late response, I didn't realized that there was a progress in this bug.

(In reply to comment #5)
> (In reply to comment #1)
> > I hit this bug by doing kinit as non-root user:
> > 
> > $ kinit Bender.QE
> > kinit: Credential cache directory /run/user/1000/krb5cc does not exist while
> > getting default ccache
> 
> How was your shell started?  Under most conditions I'd expect pam_systemd to
> have created /run/user/$UID as part of setting up the user's session.

The shell was stared by su (the same with sudo); and it seems this is the problem:

[root@pkis ~]# su - test
Last login: Tue May 14 16:31:24 CEST 2013 on pts/1
[test@pkis ~]$ echo $XDG_RUNTIME_DIR
/run/user/0
[test@pkis ~]$ id
uid=1001(test) gid=1001(test) groups=1001(test) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[test@pkis ~]$ ll /run/user/1001
ls: cannot access /run/user/1001: No such file or directory

But when it is stared by ssh (or login from console) it's ok:
$ ssh test.100.19
test.100.19's password: 
Last login: Tue May 14 16:35:36 2013
[test@pkis ~]$ echo $XDG_RUNTIME_DIR
/run/user/1001
[test@pkis ~]$ ll /run/user/1001
total 0
[test@pkis ~]$ 

So it seems that a bugs for su/sudo have to be opened.

> 
> (In reply to comment #4)
> > [root@pkisf19 ~]# rm -rf /run/user/0/
> 
> $XDG_RUNTIME_DIR typically points to this directory.  If it does, it's
> expected to be there.

--- Additional comment from Patrik Kis on 2013-05-17 04:50:36 EDT ---

(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #4)
> > > [root@pkisf19 ~]# rm -rf /run/user/0/
> > > [root@pkisf19 ~]# 
> > > [root@pkisf19 ~]# kinit Administrator.QE
> > > kinit: Credential cache directory /run/user/0/krb5cc does not exist while
> > > getting default ccache
> > 
> > krb5 1.10 will not create the ccdir for you, 1.11 should do it.
> 
> Even 1.11 will only create the directory itself (/run/user/0/krb5cc).  For
> most users, though, removing /run/user/$UID isn't going to be allowed by
> filesystem permissions, so I'm less concerned about it being removed than it
> not being created in the first place.

I'm a bit confused. I'm using krb5-1.11.2-5.fc19 and the /run/user/$UID directory is not created by kerberos. I tried:

[root@pkis ~]# env |grep XDG_RUNTIME_DIR
XDG_RUNTIME_DIR=/run/user/0
[root@pkis ~]# ls /run/user/0
ls: cannot access /run/user/0: No such file or directory
(reverse-i-search)`j': ^Curnalctl -f
[root@pkis ~]# kinit Bender.QE
kinit: Credential cache directory /run/user/0/krb5cc does not exist while getting default ccache
[root@pkis ~]# 

Or when/how this should be created.

I see still two problems here and not quite sure where/how it should be fixed:

1/ After su/sudo login /run/user/$UID is not created. Who supposed to create it? sudo/su(pam) or kerberos

2/ Kerberos relays on value XDG_RUNTIME_DIR, which is not updated after su/sudo login. This seems to be clearly problem in su/sudo or maybe pam.

--- Additional comment from Patrik Kis on 2013-05-21 10:18:28 EDT ---

Hi Nalin,

Please ignore my previous comments; I was confused a bit.
It's clear that the $ID directory under /run/user should be create by the tool that manages the login process (su/sudo etc). The directory like /run/user/0/krb5cc/ again should be created by kerberos and it seems the kerberos tools handle this (I've tested most of command line tools, like kinit, kadmin, etc).

As far as the login tools concerns bug are filed against su and sudo and they will be fixed hopefully.
I have still one specific case though. Our automated tests are executed by daemons running on the system and like that there is no physical user logged in (particularly root). Therefore the directory /run/user/0 is missing and all our kerberos tests are failing when running kadmin, kinit, etc. The tests can be modified in a way that the required directory is created, but there is a question whether this directory shouldn't be created rather automatically (either by kinit,kadmin, etc for the calling user or precreated for every user that has running process (systemd?) ). The main question is if there is other use case than automated tests, if there may be other daemons that need to have /run/user/$ID created for user. Any thought, ideas?

So far it seems to me only kerberos command line tools require this directory, so that's the reason I'm asking you.

--- Additional comment from Simo Sorce on 2013-05-21 11:15:49 EDT ---

/run/user/0 should probably be precreated at system boot.
not sure what tool shoul do it, either systemd or tmpfiles.d script ?

--- Additional comment from Nalin Dahyabhai on 2013-05-21 14:24:40 EDT ---

I'm not overly familiar with processes which run as root and which operate as clients which would need credential caches -- my guess is that most of those cases, among things that we ship, are probably only concerned about filesystem access, and their needs are handled as special cases by rpc.gssd or gssproxy.

I don't think that a tmpfiles.d-based attempt would prevent logind from removing the directory if the root user logged in properly and then logged out.

--- Additional comment from Nalin Dahyabhai on 2013-05-21 17:45:49 EDT ---

Hmm, so based on a reading of bug #965419, the lifecycle /run/user/$LOGINUID is more or less bound to the audit session, and su and sudo have no effect on that.

--- Additional comment from Stef Walter on 2013-05-22 08:55:54 EDT ---

(In reply to Nalin Dahyabhai from comment #14)
> I'm not overly familiar with processes which run as root and which operate
> as clients which would need credential caches 

realmd does, for instance.

--- Additional comment from Simo Sorce on 2013-05-22 09:01:02 EDT ---

Stef, in general applications that run as root should be run by setting a specific KRB5CCNAME variable unless they are meant to modify the interactive root user ccache. (like kinit, klist, kvno).

Any other application should probably use a separate ccache.

Not sure what to make of "sudo kinit", it makes little sense to me. However sudo -i should create the /run/user/0/krb5cc dir if it is not already available IMO.

--- Additional comment from Stef Walter on 2013-05-22 09:18:39 EDT ---

(In reply to Simo Sorce from comment #17)
> Stef, in general applications that run as root should be run by setting a
> specific KRB5CCNAME variable unless they are meant to modify the interactive
> root user ccache. (like kinit, klist, kvno).
> 
> Any other application should probably use a separate ccache.

Yes, that's what realmd 'the daemon' does.

> Not sure what to make of "sudo kinit", it makes little sense to me. However
> sudo -i should create the /run/user/0/krb5cc dir if it is not already
> available IMO.

It totally makes sense to do things like this:

$ sudo -s
# kinit admin
# realm join blah.com

Or the similar:

sudo -- sh -c 'kinit admin; realm join blah.com'

Especially if you want to use non-password credentials to do a domain join.

The samba equivalent is:

$ sudo -s
# kinit admin
# net join ads -k blah.com

Not being able to use kinit within a normal sudo command is pretty broken for lots of use cases.

--- Additional comment from Simo Sorce on 2013-05-22 13:13:42 EDT ---

There is a way to tell systemd to not destroy the user directory at session logout but I forgot what it is.
Maybe we can special case root this way and create the dir with tmpfiles.d, it would allow all the sudo scenarios to work properly.

--- Additional comment from Nalin Dahyabhai on 2013-05-22 17:55:30 EDT ---

(In reply to Simo Sorce from comment #19)
> There is a way to tell systemd to not destroy the user directory at session
> logout but I forgot what it is.

That'd be "loginctl enable-linger root".  I think you lose out on it ever being pruned, though.

--- Additional comment from Simo Sorce on 2013-05-23 16:45:41 EDT ---

Isn't it pruned at reboot given /run is a tmpfs ?
(reason why I say we also need tmpfiles.d)

--- Additional comment from Nalin Dahyabhai on 2013-05-23 17:55:53 EDT ---

Yes.  The linger option also results in it being re-created at boot, though, which is handy because tmpfiles.d configuration appears to be handled before /run is mounted on my test box.

--- Additional comment from Simo Sorce on 2013-05-24 09:40:49 EDT ---

(In reply to Nalin Dahyabhai from comment #22)
> Yes.  The linger option also results in it being re-created at boot, though,

Great, even better.

> which is handy because tmpfiles.d configuration appears to be handled before
> /run is mounted on my test box.

Sounds like a bug, I recall other programs creating directories there, heck tmpfiles.d reason of existence is to recreate directories on tmpfs filesystems, especially /run, so this makes no sense ... is this a bug in f19/rawhide ?

--- Additional comment from Patrik Kis on 2013-05-27 06:45:08 EDT ---

(In reply to Simo Sorce from comment #23)
> (In reply to Nalin Dahyabhai from comment #22)
> > Yes.  The linger option also results in it being re-created at boot, though,
> 
> Great, even better.
> 
> > which is handy because tmpfiles.d configuration appears to be handled before
> > /run is mounted on my test box.
> 
> Sounds like a bug, I recall other programs creating directories there, heck
> tmpfiles.d reason of existence is to recreate directories on tmpfs
> filesystems, especially /run, so this makes no sense ... is this a bug in
> f19/rawhide ?

On my F19 machine I was able to create directories with tmpfiles.d in /run itself but not in /run/user, so I would say that /run/user is created later, but I don't know what creates it.
Anyhow, it seems all configuration I've checked in tmpfiles.d creates application specific files/directories and no user specific, what /run/user/0 is. So it is really better to create it by the linger option of systemd.
I verified it and it works as expected; thanks to figuring this out. It just should be enabled by default for user root, so the automated test can be executed without changes. This would also help for sudo case, as you wrote (not the sudo executed by root and want to run task as non-root user, like # sudo -u dumyuser CMD).

--- Additional comment from Patrik Kis on 2013-05-27 06:46:51 EDT ---

Filed bug for RHEL-7.0 to precreate the /run/user/0 directory during the first boot: bug 967509

--- Additional comment from Alexander Bokovoy on 2013-06-05 08:48:48 EDT ---

(In reply to Simo Sorce from comment #17)
> Stef, in general applications that run as root should be run by setting a
> specific KRB5CCNAME variable unless they are meant to modify the interactive
> root user ccache. (like kinit, klist, kvno).
> 
> Any other application should probably use a separate ccache.
In FreeIPA case we need to modify the interactive root user ccache for 'ipa-adtrust-install'. Since ipa-adtrust-install must be run as root (it modifies Samba configuration via databases which only accessible to root), and end result of the ipa-adtrust-install is that KDC starts attaching MS-PAC to tickets, and those tickets required by smbd for trust setup, admin will have to  re-init the ticket to have MS-PAC in it.

That is why we don't use private ccache in ipa-adtrust-install.

--- Additional comment from Stef Walter on 2013-06-12 12:05:17 EDT ---

So how are we going to fix this in Fedora 19?

--- Additional comment from Nalin Dahyabhai on 2013-06-13 13:47:33 EDT ---

For the general case, the only complete solution is to enable lingering for accounts which are accessed using su and sudo, but we can't do that within this package.

Anything else (for example, having libkrb5 create the directory if it's /run/user/0, which is the only case where it would have permissions to do so) leaves us open to surprises if a such a user opens a full login session and then logs off, at which point logind will wipe the directory.

--- Additional comment from Patrik Kis on 2013-06-14 03:00:20 EDT ---

I have opened this systemd bug 967509 for RHEL-7 to enalble lingering for root by default. I'm not sure if systemd is the right component, maybe it should be done directly in anaconda, but the bug can forwarded any time.
This will enable root lingering only. Maybe we should consider to enable it to all user by default when user accounts are created; the main question is whether it has any drawbacks.

--- Additional comment from Martin Kosek on 2013-06-26 02:48:35 EDT ---