Bug 690778

+++ This bug was initially created as a clone of Bug #689989 +++

Description of problem:
Today when I resumed my system and unlocked Gnome, my Kerberos ticket did not get renewed, not even after some time passed since the login. I have krb5_store_password_if_offline = true in /etc/sssd/sssd.conf. I have noticed a similar problem on another system, which is authenticating against AD Kerberos.

Version-Release number of selected component (if applicable):
1.5.3-3.fc14

How reproducible:
Sometimes.

Steps to Reproduce:
1. Suspend/resume.
2. Unlock Gnome.
3. Check ticket with klist.
  
Actual results:
Ticket doesn't get renewed.

Expected results:
Ticket should always get renewed.

Additional info:
This system is authenticating against a MIT Kerberos KDC running on F-14.

The other system, one authenticating against AD, never suspends. I access it via xrdp and occasionally the ticket doesn't get renewed on it either when I login the next day.

--- Additional comment from bojan on 2011-03-22 18:34:32 EDT ---

One more detail here. I think I may have seen this with 1.5.1-9 and 1.5.1-3 as well, but not 100% sure on this.

--- Additional comment from bojan on 2011-03-22 18:36:32 EDT ---

Just locked/unlocked Gnome and the ticket got renewed. So, it may have something to do with network being unavailable on resume or something like that.

Not sure how this would affect the system that runs xrdp, because there the network must be available before one can even attempt a login.

--- Additional comment from sbose on 2011-03-23 04:30:28 EDT ---

It would be nice if you can attach log files, preferably with debug_level = 9, because I cannot reproduce the issue with F14.

--- Additional comment from bojan on 2011-03-23 08:28:15 EDT ---

(In reply to comment #3)
> It would be nice if you can attach log files, preferably with debug_level = 9,
> because I cannot reproduce the issue with F14.

Will try. It's kinda intermittent, so it may take a while.

--- Additional comment from dpal on 2011-03-23 13:30:13 EDT ---

I am running 1.5.1 for a while now and have seen a glitch only once.
But I was under the assumption that this is somehow related to the fact that you can renew ticket in couple different ways. 
1) You can re-authenticate using a credential (password) - this will be a full blown authentication and new TGT will be issued. This happens when you authenticate online. 
2) Renew ticket like Kinit -R does. But:
       -R     requests renewal of the ticket-granting ticket.   Note  that  an
              expired  ticket  cannot  be renewed, even if the ticket is still
              within its renewable life.

I have not checked the code but I was under the assumption that in the VPN case we have to do 2) since we do not have password in clear - only hash.
So if you authenticated online and got TGT and then for several days suspended the machine and then try to get online but without access to the KDC first the authentication will happen offline, you will start VPN and then SSSD will try to renew the TGT but since time expired it would fail.

Can we confirm that this is the case here? 

My RHEL machine in this case produced a bubble that the credentials were not renewed. I do not remember any more details.

It should be an easy thing to test as the TGT renewal policies can be adjusted for the purpose of this test. Some details can be found here: http://www.freeipa.org/page/Automatic_Ticket_Renewal

--- Additional comment from sgallagh on 2011-03-23 13:56:16 EDT ---

(In reply to comment #5)
> I am running 1.5.1 for a while now and have seen a glitch only once.
> But I was under the assumption that this is somehow related to the fact that
> you can renew ticket in couple different ways. 
> 1) You can re-authenticate using a credential (password) - this will be a full
> blown authentication and new TGT will be issued. This happens when you
> authenticate online. 

This is also what happens if SSSD gets the TGT on your behalf with 'krb5_store_password_if_offline = true'.

> 2) Renew ticket like Kinit -R does. But:
>        -R     requests renewal of the ticket-granting ticket.   Note  that  an
>               expired  ticket  cannot  be renewed, even if the ticket is still
>               within its renewable life.
> 
> I have not checked the code but I was under the assumption that in the VPN case
> we have to do 2) since we do not have password in clear - only hash.

This is wrong. When setting 'krb5_store_password_if_offline = true', it means that you are giving us permission to store the plaintext password in the kernel keyring until the SSSD goes online. We will then kinit on your behalf with the password and then destroy the plaintext.


> So if you authenticated online and got TGT and then for several days suspended
> the machine and then try to get online but without access to the KDC first the
> authentication will happen offline, you will start VPN and then SSSD will try
> to renew the TGT but since time expired it would fail.

This would not be the case even if we WERE using the TGT to renew (which as previously stated we are not). The renewal would always be a new request with a timestamp of *right now*.

> 
> Can we confirm that this is the case here? 
> 

It's not.

> My RHEL machine in this case produced a bubble that the credentials were not
> renewed. I do not remember any more details.
> 

If you saw a notification that SPECIFICALLY said your credentials were not renewed, it means something happened during the kinit that caused the remote server to deny you. This could have been a clock skew issue or your password may have changed on the KDC since you logged in locally offline to the laptop.

> It should be an easy thing to test as the TGT renewal policies can be adjusted
> for the purpose of this test. Some details can be found here:
> http://www.freeipa.org/page/Automatic_Ticket_Renewal

Automatic Ticket Renewal is a separate and unrelated feature.

--- Additional comment from bojan on 2011-03-23 18:21:04 EDT ---

(In reply to comment #4)
 
> Will try. It's kinda intermittent, so it may take a while.

I happened again this morning when I resumed, so it seems to be somewhat consistent. Are we having a repeat of bug #658689 in some way here?

Let me turn debugging on...

--- Additional comment from bojan on 2011-03-23 18:33:09 EDT ---

(In reply to comment #7)
 
> Let me turn debugging on...

Of course, the moment I did this, the system issued a new ticket on resume. I guess we'll have to wait a while for this to occur again.

--- Additional comment from bojan on 2011-03-23 18:51:21 EDT ---

(In reply to comment #8)

> Of course, the moment I did this, the system issued a new ticket on resume. I
> guess we'll have to wait a while for this to occur again.

Here we go. Last time I resumed from suspend was 9:34 am. This is when the ticket was renewed (i.e. the failed attempt to reproduce the issue). So, I suspended once again and resumed around 9:46 am. At this time, the old ticket was kept, although I had to type the password in at the unlock screen.

Will attach the logs.

--- Additional comment from bojan on 2011-03-23 18:52:04 EDT ---

Created attachment 487160 [details]
Logs of one successful and one unsuccessful ticket issue on resume

--- Additional comment from sgallagh on 2011-03-24 11:18:53 EDT ---

From the logs you posted:

(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [krb5_find_ccache_step] (9): Preparing for offline operation.
(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [krb5_find_ccache_step] (9): Valid TGT available or ccache file is already in use.
...
(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [sysdb_cache_auth] (9): Offline credentials expiration is [0] days.
(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [check_failed_login_attempts] (9): Failed login attempts [0], allowed failed login attempts [0], failed login delay [5].
(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [sysdb_cache_auth] (4): Hashes do match!
(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [ldb] (9): commit ldb transaction (nesting: 0)
(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [add_user_to_delayed_online_authentication] (1): add_key fialed [122][Disk quota exceeded].
(Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [krb5_pam_handler_cache_auth_step] (1): add_user_to_delayed_online_authentication failed.


So what happened here is that your system attempted to perform offline authentication (the KDC may have been temporarily unreachable while you were unsuspending) and it attempted to perform an offline authentication and then queue up a kinit for when SSSD got back online.

However, you got an error from our syscall to save your password into the kernel keyring "Disk quota exceeded".

From the manpage for add_key():
EDQUOT

The key quota for this user would be exceeded by creating this key or linking it to the keyring.


So I'd say that this bug is being caused by a tunable parameter set too low on your system. We're behaving properly here.

--- Additional comment from sgallagh on 2011-03-24 12:10:37 EDT ---

Reopening this bug.

On further investigation, we identified a possibility that we may be at fault for this. Under certain circumstances, we may be saving multiple keyring values and then not removing them later.

We need to add a check that the user does not already have a pending delayed authentication before we save the password to the keyring.

--- Additional comment from sbose on 2011-03-24 13:41:49 EDT ---

It would be also good to know if maybe other applications of the user are using the kernel keyring. You can check as the user /proc/keys and /proc/key-users about usage and limits for the user. See e.g. http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/keys.txt;h=e4dbbdb1bd961e7a7a582c3dbb3ada1008218218;hb=521cb40b0c44418a4fd36dc633f575813d59a43d

for details

--- Additional comment from bojan on 2011-03-24 19:00:00 EDT ---

Here is how many keys are stored:
-------------------
[root@shrek ~]# cat /proc/keys  | wc -l
282
-------------------

And this is what uses them:
-------------------
[root@shrek ~]# cat /proc/key-users 
    0:   285 284/284 282/200 6822/20000
  500:     5 5/5 5/200 117/20000
  501:    11 11/11 11/200 119/20000
-------------------

Note, user 501 is the other person using this system, but there are no processes running for this user right now.

The keys actually stored are below, if it helps:
-------------------
[root@shrek ~]# cat /proc/keys 
000c27bf I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
00285071 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0053f956 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
00588387 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
00b5e2f3 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
00b6214c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
00e334e5 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0117293f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0130a81c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
018b6456 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
018c93a1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
018fd74e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
01f10b3b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
02294765 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
024f11bc I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0278d7f4 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
02a21917 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
02b097a3 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
02c4f5f2 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
03069c70 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
030ca501 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
03172d9f IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
03309a79 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
034bdedd I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
03608404 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
044a465e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
04d1723d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
05080eb1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
056b699a I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
061af6f1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
062ff355 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0635f84e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
06da5539 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
07412273 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
07e15583 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
07e93c69 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
0809fea0 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
081bdcab I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
088e542c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
097aa67f IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
098bb034 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
09c3621c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
09ddc23b IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
0a931c92 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0ab9319d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0ae2f8a8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0ae494c5 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0b28a0e1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0b2b4534 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0b625f73 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0b6dbf7d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0b808cd1 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
0bbcd96d IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
0beb8e6f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0c664e07 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0cf910da I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0d07e439 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0d0a1e6c IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
0d2ae984 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0d5f1ad8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0db578d5 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0e07b545 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0e15b4b5 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
0e467947 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0e551c93 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0e61874b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0e75e6d0 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
0effa0a0 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0f3cedf8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0f5ec6f7 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0f65706b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0fbba5b6 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
0fc8bf77 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
101c6d15 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1079f4e0 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
108b9daa IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1140a299 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
114b7d83 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
114eab76 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
114fd3af IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
11eccb25 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
121d798a I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
122e24a8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1248fedf I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
12844ea3 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
128a723f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
12b26e86 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
12c23616 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
137b575d IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
146f4187 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
14ca43b8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
14cfe541 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
14d42e02 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
15018ce0 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
15d0556f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
15e1db6f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
15ee8bb5 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
16925f9c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
16a15e8f IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
17a53bac I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
17a9b101 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
17ae4b01 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
17b5bf8c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
180de34b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
18303afb IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
18afa044 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
18cb5073 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1988a86b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
19a842c9 I--Q--     2 perm 1f3f0000     0    -1 keyring   _uid.0: empty
19aae242 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1a0889ec IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1a55c6d3 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1aba689d IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1ade5a5f I--Q--     1 perm 1f3f0000     0     0 keyring   _tid: empty
1ae1a9af IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1b11c8ed I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1b48f20a I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1b513a29 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1b5420b9 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1b81e8f4 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1bb055a9 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1bd18844 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1c0960ad I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1d08b39d IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1d138771 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1d4953e4 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1d81660f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1da4d8bd IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1db75240 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1dee79d8 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1df8a5f8 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
1dfbcc35 IR-Q--     3 expd 1f3f0000     0     0 keyring   _ses: empty
1e4c169c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1ec8bf0e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1f4c9277 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
1fc62a20 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
200de391 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
201ce9eb I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
20204e12 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
202b01b0 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
203e1d14 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
2080c6af I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
20f27c5c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
20f49269 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
211f11dc I--Q--     1 perm 1f3f0000     0     0 keyring   _tid: empty
21345c61 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
21d10a3c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
225f034b IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
2267432a I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
22805f41 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
22af5fc2 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
22b8a6f7 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
22f43b14 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
230277ef I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
23709cf9 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
23b93021 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
241d5aad I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
24515976 I--Q--     1 perm 1f3f0000     0     0 keyring   _tid: empty
24a0f578 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
24e3155c IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
24fc4357 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
24fc70e5 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
250f8c01 I--Q--     1 perm 1f3f0000     0    -1 keyring   _uid_ses.0: 1/4
2570b9d8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
25cd073e IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
25dd307c I--Q--     1 perm 1f3f0000     0     0 keyring   _tid: empty
25e520b1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
25f6e8cf I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
26033c17 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2690692e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
26a6474f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
26d70df8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
26f5b4d1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
27019896 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
270f3b2a I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
272f0739 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
275bd826 IR-Q--     1 expd 1f3f0000     0     0 keyring   _ses: empty
2794d680 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
279ba68d IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
27be52d1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
27cc6e02 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
282e90bb I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
28747ade I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
289abc02 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
28a04e64 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
28abf6fc I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
28c92e36 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
28e2e0ff I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
292579d0 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
294f0b5d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2a05ef1b IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
2a26fed8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2a3343fc I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2b15bb7b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2b38023e IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
2b5015a3 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2b7068ba IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
2b84ceb6 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2b8c1984 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2bfa061e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2cddf18b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2cf2a627 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2d44d938 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
2d7823c9 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2db08df0 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2dda46fd I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2e2174a7 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2e445ba8 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
2e4ed4cb I--Q--     1 perm 1f3f0000     0     0 keyring   _tid: empty
2ebe7586 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2efb8c47 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2f148eb3 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2fdd2a52 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
2ff37f4d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
30768307 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
30d1b5cc I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
31498844 IR-Q--     1 expd 1f3f0000     0     0 keyring   _ses: empty
31953da6 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
319e4b5d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
31a93f24 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
31afd915 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
31f0c993 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
320e892f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
32153150 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
32321ca0 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3260a5db I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
32a35521 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
32cb3fae I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
330f264b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
33187f5e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
335ed7bd IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
3368fe36 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
33985a8c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
33cacc56 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
34166904 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
34276759 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
343f886f IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
35454ccf IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
35462950 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
355380e3 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
357cfb5a I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
35cfc2d8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
35ead540 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
362d9f85 I--Q--     4 perm 1f3f0000     0     0 keyring   _ses: empty
363b1797 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
365a23c6 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
365e426d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3699d549 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
36a5db5e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
36e001ed I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3753e43c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
38217525 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
38c4b67e I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
392f904f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
39329aae I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
39d35ea2 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
39fc8732 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3a0cb29f I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3a7dfef8 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3a954004 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
3b2e5878 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3be75bd6 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3bebd991 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3c388932 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3c8add1b I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3c9718ed I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3ceb5e16 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3d532674 IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
3d640757 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3dbef489 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3ddf352e IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
3dfafc3d IR-Q--     2 expd 1f3f0000     0     0 keyring   _ses: empty
3e04c911 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3e294939 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3e5d2709 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3ec36fc7 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3edc2e1c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3f003a1a I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3f15881d I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3f99cdad IR-Q--   114 expd 1f3f0000     0     0 keyring   _ses: empty
3f9bd13c I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
3febb8f1 I--Q--     1 perm 1f3f0000     0     0 keyring   _ses: empty
-------------------

Limits (which are default - haven't touched anything):
-------------------
[root@shrek keys]# cat maxbytes maxkeys root_maxbytes root_maxkeys gc_delay 
20000
200
20000
200
300
-------------------

So, yeah, looks like we are hitting the limit, but I'm not sure why. I try to keep this box as stock F-14 as possible.

--- Additional comment from bojan on 2011-03-24 19:06:09 EDT ---

On that other system I mentioned (the one that runs xrdp and is a VM), I see:
-------------
[root@XXXXXXXX ~]# cat /proc/key-users 
    0:    81 80/80 77/200 1585/20000
nnnnn:     3 3/3 3/200 83/20000
[root@XXXXXXXX ~]# cat /proc/keys | wc -l
78
-------------

So, if there was a problem there, it may have been due to the same thing. I rebooted that VM 2 days ago. I did not experience any problems with sssd after the reboot.

On the other hand, my laptop, which is having problems with this, has been up for 28 days (I always suspend or hibernate it).

--- Additional comment from bojan on 2011-03-25 02:39:01 EDT ---

(In reply to comment #15)

> [root@XXXXXXXX ~]# cat /proc/keys | wc -l
> 78

This has gone up to 85 while I was working today. I login to the VM a few times after I've been disconnected, if it matters.

--- Additional comment from sgallagh on 2011-03-25 07:23:19 EDT ---

Hmm, we're definitely doing something wrong here. I just ran a simple experiment on my machine and we're increasing the number of keys in the kernel keyring every time we perform an authentication through Kerberos.

Note: this is EVERY time, not just offline auth. However, when we're online, no side-effects are experienced.

We need to investigate why we're adding keys unnecessarily (and not deleting them)