Bug 986610

Summary: sssd[krb5_child[PID]: Permission denied
Product: [Fedora] Fedora Reporter: John Florian <trailtotale>
Component: sssdAssignee: Jakub Hrozek <jhrozek>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: jhrozek, lslebodn, okos, pbrezina, sbose, sgallagh, ssorce
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sssd-1.10.1-1.fc19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-15 15:20:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description John Florian 2013-07-21 00:13:20 UTC
Description of problem:
I have a Fedora 19 host configured to use sssd with id via OpenLDAP and auth via Kerberos.  Having just migrated my Kerberos server from F16 to F19, I was running through various tests.  Among these was a ssh connection to the new Kerberos server where I observed the following:

Via journalctl -f:
sshd[3563]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=from_host.mydomain  user=myuser
[sssd[krb5_child[3565]: Can't find client principal myuser@MYREALM in cache collection
[sssd[krb5_child[3565]: Permission denied
[sssd[krb5_child[3565]: Permission denied
sshd[3563]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=from_host.mydomain user=myuser
sshd[3563]: pam_sss(sshd:auth): received for user myuser: 4 (System error)
sshd[3563]: Failed password for myuser from A.B.C.D port 46164 ssh2

That prompted me to examine /var/log/sssd/krb5_child.log where I'd captured:
[[sssd[krb5_child[3565]]]] [unpack_buffer] (0x0100): cmd [241] uid [1000] gid [1000] validate [false] enterprise principal [false] offline [false] UPN [myuser@MYREALM]
[[sssd[krb5_child[3565]]]] [unpack_buffer] (0x0100): ccname: [DIR:/run/user/1000/krb5cc] keytab: [/etc/krb5.keytab]
[[sssd[krb5_child[3565]]]] [k5c_setup] (0x0100): Cannot read [SSSD_KRB5_RENEWABLE_LIFETIME] from environment.
[[sssd[krb5_child[3565]]]] [k5c_setup] (0x0100): Cannot read [SSSD_KRB5_LIFETIME] from environment.
[[sssd[krb5_child[3565]]]] [krb5_set_canonicalize] (0x0100): SSSD_KRB5_CANONICALIZE is set to [false]
[[sssd[krb5_child[3565]]]] [k5c_setup] (0x0100): Not using FAST.
[[sssd[krb5_child[3565]]]] [get_and_save_tgt] (0x0100): TGT validation is disabled.
[[sssd[krb5_child[3565]]]] [create_ccdir] (0x0020): The directory /run/user/1000/krb5cc is owned by 0/0, expected 1000/1000
[[sssd[krb5_child[3565]]]] [create_ccache_in_dir] (0x0040): Cannot create directory /run/user/1000/krb5cc
[[sssd[krb5_child[3565]]]] [get_and_save_tgt] (0x0020): 1263: [13][Permission denied]
[[sssd[krb5_child[3565]]]] [map_krb5_error] (0x0020): 1283: [13][Permission denied]

I can confirm that indeed /run/user/1000/krb5cc/ is owned by root (while the parent directory is owned by me (myuser).  As root, I 'rm -rf /run/user/1000' and retried but things were created this way again.  My next step was to 'chown -R myuser. /run/user/1000' but then I see:

[[sssd[krb5_child[3634]]]] [create_ccdir] (0x0020): The directory /run/user/1000/krb5cc has wrong permissions 600, expected 0700
[[sssd[krb5_child[3634]]]] [create_ccache_in_dir] (0x0040): Cannot create directory /run/user/1000/krb5cc

I then 'chmod 0700 /run/user/1000/krb5cc' and finally got successful looking logs.  Of course none of my "fixes" lasts.

Version-Release number of selected component (if applicable):
sssd-1.10.0-16.fc19.x86_64
sssd-krb5-common-1.10.0-16.fc19.x86_64

How reproducible:
always

Comment 1 Lukas Slebodnik 2013-07-22 07:14:56 UTC
This is well known bug and it has already been fixed in upstream.
https://fedorahosted.org/sssd/ticket/2002

Bug fix is not included in sssd-1.10.0-16.fc19,
but you can try update sssd from Fedora 19 testing repository
sssd-1.10.1-1.fc19 contains lot of bug fixes.

sudo yum update --enablerepo=updates-testing sssd sssd-krb5-common

If you use sssd only with OpenLDAP and Kerberos (and not with freeipa or AD)
you can install only sssd-ldap and sssd-krb5. In this case, less dependencies will be installed.

Comment 2 Jakub Hrozek 2013-07-22 08:16:06 UTC
Please also consider adding karma to the update if it indeed solves the problem.

Comment 3 John Florian 2013-07-22 22:31:12 UTC
(In reply to Lukas Slebodnik from comment #1)
> This is well known bug and it has already been fixed in upstream.
> https://fedorahosted.org/sssd/ticket/2002

I thought I remembered reading about this on fedora-devel, but it seemed long enough ago I figured the fix must have made it in before F19 was released.

> Bug fix is not included in sssd-1.10.0-16.fc19,
> but you can try update sssd from Fedora 19 testing repository
> sssd-1.10.1-1.fc19 contains lot of bug fixes.

Ah, thanks for the pointer.

> If you use sssd only with OpenLDAP and Kerberos (and not with freeipa or AD)
> you can install only sssd-ldap and sssd-krb5. In this case, less
> dependencies will be installed.

That's good to know!  I guess I'll go revise my puppet class now.  :-)

Thank you so much for exceeding my expectations on this BZ.


(In reply to Jakub Hrozek from comment #2)
> Please also consider adding karma to the update if it indeed solves the
> problem.

Done.  BTW, should this BZ# be listed in the Bugs Fixed section for that update?

Comment 4 Fedora Update System 2013-07-23 08:07:09 UTC
sssd-1.10.1-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/FEDORA-2013-13278/sssd-1.10.1-1.fc19

Comment 5 Jakub Hrozek 2013-07-23 08:08:26 UTC
(In reply to John Florian from comment #3)
> Done.  BTW, should this BZ# be listed in the Bugs Fixed section for that
> update?

It probably should now, when I created the update we only had an upstream Trac ticket, not a bugzilla. The only reason I didn't include the bugzilla to the update was that adding/removing a bug to an updates-testing update might move it back to pending. (Or at least I thought it would happen..)

That said we should include all bugs fixed in the update. It's important for the users to see what had been fixed.

Comment 6 Fedora Update System 2013-07-30 17:51:57 UTC
sssd-1.10.1-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 7 Michael Cronenworth 2013-08-14 16:22:47 UTC
I'm still seeing this problem on F19.

sssd-1.10.1-1.fc19.x86_64

The krb5cc directory is being created sometimes with 0600 and sometimes with 0700. I have to remove that directory to allow SSH logins again to let things recreate.

Comment 8 Lukas Slebodnik 2013-08-14 16:36:04 UTC
This bug should be fixed, but it can be another problem non related to sssd.
Can you provide more informations? (Steps to reproduce) 
Do you authenticate with ssh keys?

Comment 9 Michael Cronenworth 2013-08-14 16:45:09 UTC
I do not use ssh keys.

The machine was just upgraded from Fedora 18 last week and logins were working normally. After the upgrade logins still worked but after a few days people reported that they were not able to login. I deleted the /run/user/$UID/krb5cc directory and the user who reported the problem was able to log in. I've had to do this for 3 different users now. Today I had to do it a second time for a user who reported the problem before, so I went bug hunting.

The users perform a normal SSH login with PuTTY or SFTP in with Filezilla. When it fails this message appears in /var/log/messages:
Aug 14 10:05:21 localhost [sssd[krb5_child[18793]]]: Can't find client principal username in cache collection
(credentials have been obfuscated)
I have to remove the krb5cc directory for their user before they are able to login with SSH or SFTP in.

SSSD is setup to talk to a Active Directory server for authentication. Here is my sssd.conf:
http://fpaste.org/32106/98676137/

Comment 10 Lukas Slebodnik 2013-08-15 07:27:18 UTC
The main problem is in libkrb5. If library find out, that directory does not exist librkrb5 will create directory itself, but usually with wrong owner(root).
In sssd, we create directory with right permissions and owner before we call any krb5 functions. You can see in description of this bug 986610#c0 some error messages in log file /var/log/sssd/krb5_child.log. It was a special case, where we forgot create directory itself before calling krb5 functions. This is a reason why I wanted to know more information about your problem.

But it is not solved in another packages. You can read detailed information about this problem here: https://lists.fedoraproject.org/pipermail/devel/2013-July/186930.html . There are approximately 40 mails and people are trying to solve this.

Possible workaround is to call command "loginctl enable-linger $USER".
With this command, directory "/run/user/$UID/" will not be removed after logout.

I hope it helps you.

Comment 11 Michael Cronenworth 2013-08-15 15:20:26 UTC
Sorry. It looks like bug 961235 is what I should be tracking.

I found reference in bug 967509 that the Red Hat KRB team found some solution but it links to a Red Hat private bug 991169.

I will try your work around in the mean time. Thanks.

Comment 12 Lukas Slebodnik 2013-08-15 15:45:01 UTC
Bug 991169 is private, but it has not been solved yet. As I wrote earlier fundamental solution of this issue is a work in progress.