Description of problem: Today when I resumed my system and unlocked Gnome, my Kerberos ticket did not get renewed, not even after some time passed since the login. I have krb5_store_password_if_offline = true in /etc/sssd/sssd.conf. I have noticed a similar problem on another system, which is authenticating against AD Kerberos. Version-Release number of selected component (if applicable): 1.5.3-3.fc14 How reproducible: Sometimes. Steps to Reproduce: 1. Suspend/resume. 2. Unlock Gnome. 3. Check ticket with klist. Actual results: Ticket doesn't get renewed. Expected results: Ticket should always get renewed. Additional info: This system is authenticating against a MIT Kerberos KDC running on F-14. The other system, one authenticating against AD, never suspends. I access it via xrdp and occasionally the ticket doesn't get renewed on it either when I login the next day.
One more detail here. I think I may have seen this with 1.5.1-9 and 1.5.1-3 as well, but not 100% sure on this.
Just locked/unlocked Gnome and the ticket got renewed. So, it may have something to do with network being unavailable on resume or something like that. Not sure how this would affect the system that runs xrdp, because there the network must be available before one can even attempt a login.
It would be nice if you can attach log files, preferably with debug_level = 9, because I cannot reproduce the issue with F14.
(In reply to comment #3) > It would be nice if you can attach log files, preferably with debug_level = 9, > because I cannot reproduce the issue with F14. Will try. It's kinda intermittent, so it may take a while.
I am running 1.5.1 for a while now and have seen a glitch only once. But I was under the assumption that this is somehow related to the fact that you can renew ticket in couple different ways. 1) You can re-authenticate using a credential (password) - this will be a full blown authentication and new TGT will be issued. This happens when you authenticate online. 2) Renew ticket like Kinit -R does. But: -R requests renewal of the ticket-granting ticket. Note that an expired ticket cannot be renewed, even if the ticket is still within its renewable life. I have not checked the code but I was under the assumption that in the VPN case we have to do 2) since we do not have password in clear - only hash. So if you authenticated online and got TGT and then for several days suspended the machine and then try to get online but without access to the KDC first the authentication will happen offline, you will start VPN and then SSSD will try to renew the TGT but since time expired it would fail. Can we confirm that this is the case here? My RHEL machine in this case produced a bubble that the credentials were not renewed. I do not remember any more details. It should be an easy thing to test as the TGT renewal policies can be adjusted for the purpose of this test. Some details can be found here: http://www.freeipa.org/page/Automatic_Ticket_Renewal
(In reply to comment #5) > I am running 1.5.1 for a while now and have seen a glitch only once. > But I was under the assumption that this is somehow related to the fact that > you can renew ticket in couple different ways. > 1) You can re-authenticate using a credential (password) - this will be a full > blown authentication and new TGT will be issued. This happens when you > authenticate online. This is also what happens if SSSD gets the TGT on your behalf with 'krb5_store_password_if_offline = true'. > 2) Renew ticket like Kinit -R does. But: > -R requests renewal of the ticket-granting ticket. Note that an > expired ticket cannot be renewed, even if the ticket is still > within its renewable life. > > I have not checked the code but I was under the assumption that in the VPN case > we have to do 2) since we do not have password in clear - only hash. This is wrong. When setting 'krb5_store_password_if_offline = true', it means that you are giving us permission to store the plaintext password in the kernel keyring until the SSSD goes online. We will then kinit on your behalf with the password and then destroy the plaintext. > So if you authenticated online and got TGT and then for several days suspended > the machine and then try to get online but without access to the KDC first the > authentication will happen offline, you will start VPN and then SSSD will try > to renew the TGT but since time expired it would fail. This would not be the case even if we WERE using the TGT to renew (which as previously stated we are not). The renewal would always be a new request with a timestamp of *right now*. > > Can we confirm that this is the case here? > It's not. > My RHEL machine in this case produced a bubble that the credentials were not > renewed. I do not remember any more details. > If you saw a notification that SPECIFICALLY said your credentials were not renewed, it means something happened during the kinit that caused the remote server to deny you. This could have been a clock skew issue or your password may have changed on the KDC since you logged in locally offline to the laptop. > It should be an easy thing to test as the TGT renewal policies can be adjusted > for the purpose of this test. Some details can be found here: > http://www.freeipa.org/page/Automatic_Ticket_Renewal Automatic Ticket Renewal is a separate and unrelated feature.
(In reply to comment #4) > Will try. It's kinda intermittent, so it may take a while. I happened again this morning when I resumed, so it seems to be somewhat consistent. Are we having a repeat of bug #658689 in some way here? Let me turn debugging on...
(In reply to comment #7) > Let me turn debugging on... Of course, the moment I did this, the system issued a new ticket on resume. I guess we'll have to wait a while for this to occur again.
(In reply to comment #8) > Of course, the moment I did this, the system issued a new ticket on resume. I > guess we'll have to wait a while for this to occur again. Here we go. Last time I resumed from suspend was 9:34 am. This is when the ticket was renewed (i.e. the failed attempt to reproduce the issue). So, I suspended once again and resumed around 9:46 am. At this time, the old ticket was kept, although I had to type the password in at the unlock screen. Will attach the logs.
Created attachment 487160 [details] Logs of one successful and one unsuccessful ticket issue on resume
From the logs you posted: (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [krb5_find_ccache_step] (9): Preparing for offline operation. (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [krb5_find_ccache_step] (9): Valid TGT available or ccache file is already in use. ... (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [sysdb_cache_auth] (9): Offline credentials expiration is [0] days. (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [check_failed_login_attempts] (9): Failed login attempts [0], allowed failed login attempts [0], failed login delay [5]. (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [sysdb_cache_auth] (4): Hashes do match! (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [ldb] (9): commit ldb transaction (nesting: 0) (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [add_user_to_delayed_online_authentication] (1): add_key fialed [122][Disk quota exceeded]. (Thu Mar 24 09:45:22 2011) [sssd[be[default]]] [krb5_pam_handler_cache_auth_step] (1): add_user_to_delayed_online_authentication failed. So what happened here is that your system attempted to perform offline authentication (the KDC may have been temporarily unreachable while you were unsuspending) and it attempted to perform an offline authentication and then queue up a kinit for when SSSD got back online. However, you got an error from our syscall to save your password into the kernel keyring "Disk quota exceeded". From the manpage for add_key(): EDQUOT The key quota for this user would be exceeded by creating this key or linking it to the keyring. So I'd say that this bug is being caused by a tunable parameter set too low on your system. We're behaving properly here.
Reopening this bug. On further investigation, we identified a possibility that we may be at fault for this. Under certain circumstances, we may be saving multiple keyring values and then not removing them later. We need to add a check that the user does not already have a pending delayed authentication before we save the password to the keyring.
It would be also good to know if maybe other applications of the user are using the kernel keyring. You can check as the user /proc/keys and /proc/key-users about usage and limits for the user. See e.g. http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/keys.txt;h=e4dbbdb1bd961e7a7a582c3dbb3ada1008218218;hb=521cb40b0c44418a4fd36dc633f575813d59a43d for details
Here is how many keys are stored: ------------------- [root@shrek ~]# cat /proc/keys | wc -l 282 ------------------- And this is what uses them: ------------------- [root@shrek ~]# cat /proc/key-users 0: 285 284/284 282/200 6822/20000 500: 5 5/5 5/200 117/20000 501: 11 11/11 11/200 119/20000 ------------------- Note, user 501 is the other person using this system, but there are no processes running for this user right now. The keys actually stored are below, if it helps: ------------------- [root@shrek ~]# cat /proc/keys 000c27bf I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 00285071 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0053f956 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 00588387 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 00b5e2f3 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 00b6214c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 00e334e5 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0117293f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0130a81c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 018b6456 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 018c93a1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 018fd74e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 01f10b3b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 02294765 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 024f11bc I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0278d7f4 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 02a21917 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 02b097a3 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 02c4f5f2 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 03069c70 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 030ca501 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 03172d9f IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 03309a79 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 034bdedd I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 03608404 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 044a465e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 04d1723d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 05080eb1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 056b699a I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 061af6f1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 062ff355 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0635f84e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 06da5539 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 07412273 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 07e15583 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 07e93c69 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 0809fea0 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 081bdcab I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 088e542c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 097aa67f IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 098bb034 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 09c3621c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 09ddc23b IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 0a931c92 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0ab9319d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0ae2f8a8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0ae494c5 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0b28a0e1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0b2b4534 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0b625f73 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0b6dbf7d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0b808cd1 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 0bbcd96d IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 0beb8e6f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0c664e07 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0cf910da I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0d07e439 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0d0a1e6c IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 0d2ae984 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0d5f1ad8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0db578d5 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0e07b545 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0e15b4b5 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 0e467947 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0e551c93 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0e61874b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0e75e6d0 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 0effa0a0 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0f3cedf8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0f5ec6f7 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0f65706b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0fbba5b6 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 0fc8bf77 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 101c6d15 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1079f4e0 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 108b9daa IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1140a299 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 114b7d83 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 114eab76 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 114fd3af IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 11eccb25 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 121d798a I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 122e24a8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1248fedf I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 12844ea3 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 128a723f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 12b26e86 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 12c23616 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 137b575d IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 146f4187 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 14ca43b8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 14cfe541 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 14d42e02 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 15018ce0 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 15d0556f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 15e1db6f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 15ee8bb5 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 16925f9c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 16a15e8f IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 17a53bac I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 17a9b101 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 17ae4b01 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 17b5bf8c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 180de34b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 18303afb IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 18afa044 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 18cb5073 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1988a86b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 19a842c9 I--Q-- 2 perm 1f3f0000 0 -1 keyring _uid.0: empty 19aae242 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1a0889ec IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1a55c6d3 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1aba689d IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1ade5a5f I--Q-- 1 perm 1f3f0000 0 0 keyring _tid: empty 1ae1a9af IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1b11c8ed I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1b48f20a I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1b513a29 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1b5420b9 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1b81e8f4 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1bb055a9 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1bd18844 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1c0960ad I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1d08b39d IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1d138771 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1d4953e4 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1d81660f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1da4d8bd IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1db75240 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1dee79d8 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1df8a5f8 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 1dfbcc35 IR-Q-- 3 expd 1f3f0000 0 0 keyring _ses: empty 1e4c169c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1ec8bf0e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1f4c9277 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 1fc62a20 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 200de391 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 201ce9eb I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 20204e12 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 202b01b0 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 203e1d14 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 2080c6af I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 20f27c5c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 20f49269 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 211f11dc I--Q-- 1 perm 1f3f0000 0 0 keyring _tid: empty 21345c61 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 21d10a3c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 225f034b IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 2267432a I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 22805f41 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 22af5fc2 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 22b8a6f7 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 22f43b14 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 230277ef I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 23709cf9 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 23b93021 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 241d5aad I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 24515976 I--Q-- 1 perm 1f3f0000 0 0 keyring _tid: empty 24a0f578 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 24e3155c IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 24fc4357 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 24fc70e5 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 250f8c01 I--Q-- 1 perm 1f3f0000 0 -1 keyring _uid_ses.0: 1/4 2570b9d8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 25cd073e IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 25dd307c I--Q-- 1 perm 1f3f0000 0 0 keyring _tid: empty 25e520b1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 25f6e8cf I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 26033c17 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2690692e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 26a6474f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 26d70df8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 26f5b4d1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 27019896 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 270f3b2a I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 272f0739 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 275bd826 IR-Q-- 1 expd 1f3f0000 0 0 keyring _ses: empty 2794d680 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 279ba68d IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 27be52d1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 27cc6e02 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 282e90bb I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 28747ade I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 289abc02 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 28a04e64 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 28abf6fc I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 28c92e36 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 28e2e0ff I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 292579d0 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 294f0b5d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2a05ef1b IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 2a26fed8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2a3343fc I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2b15bb7b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2b38023e IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 2b5015a3 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2b7068ba IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 2b84ceb6 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2b8c1984 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2bfa061e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2cddf18b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2cf2a627 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2d44d938 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 2d7823c9 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2db08df0 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2dda46fd I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2e2174a7 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2e445ba8 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 2e4ed4cb I--Q-- 1 perm 1f3f0000 0 0 keyring _tid: empty 2ebe7586 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2efb8c47 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2f148eb3 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2fdd2a52 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 2ff37f4d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 30768307 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 30d1b5cc I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 31498844 IR-Q-- 1 expd 1f3f0000 0 0 keyring _ses: empty 31953da6 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 319e4b5d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 31a93f24 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 31afd915 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 31f0c993 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 320e892f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 32153150 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 32321ca0 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3260a5db I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 32a35521 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 32cb3fae I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 330f264b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 33187f5e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 335ed7bd IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 3368fe36 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 33985a8c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 33cacc56 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 34166904 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 34276759 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 343f886f IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 35454ccf IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 35462950 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 355380e3 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 357cfb5a I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 35cfc2d8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 35ead540 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 362d9f85 I--Q-- 4 perm 1f3f0000 0 0 keyring _ses: empty 363b1797 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 365a23c6 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 365e426d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3699d549 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 36a5db5e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 36e001ed I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3753e43c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 38217525 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 38c4b67e I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 392f904f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 39329aae I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 39d35ea2 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 39fc8732 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3a0cb29f I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3a7dfef8 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3a954004 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 3b2e5878 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3be75bd6 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3bebd991 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3c388932 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3c8add1b I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3c9718ed I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3ceb5e16 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3d532674 IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 3d640757 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3dbef489 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3ddf352e IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 3dfafc3d IR-Q-- 2 expd 1f3f0000 0 0 keyring _ses: empty 3e04c911 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3e294939 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3e5d2709 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3ec36fc7 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3edc2e1c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3f003a1a I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3f15881d I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3f99cdad IR-Q-- 114 expd 1f3f0000 0 0 keyring _ses: empty 3f9bd13c I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty 3febb8f1 I--Q-- 1 perm 1f3f0000 0 0 keyring _ses: empty ------------------- Limits (which are default - haven't touched anything): ------------------- [root@shrek keys]# cat maxbytes maxkeys root_maxbytes root_maxkeys gc_delay 20000 200 20000 200 300 ------------------- So, yeah, looks like we are hitting the limit, but I'm not sure why. I try to keep this box as stock F-14 as possible.
On that other system I mentioned (the one that runs xrdp and is a VM), I see: ------------- [root@XXXXXXXX ~]# cat /proc/key-users 0: 81 80/80 77/200 1585/20000 nnnnn: 3 3/3 3/200 83/20000 [root@XXXXXXXX ~]# cat /proc/keys | wc -l 78 ------------- So, if there was a problem there, it may have been due to the same thing. I rebooted that VM 2 days ago. I did not experience any problems with sssd after the reboot. On the other hand, my laptop, which is having problems with this, has been up for 28 days (I always suspend or hibernate it).
(In reply to comment #15) > [root@XXXXXXXX ~]# cat /proc/keys | wc -l > 78 This has gone up to 85 while I was working today. I login to the VM a few times after I've been disconnected, if it matters.
Hmm, we're definitely doing something wrong here. I just ran a simple experiment on my machine and we're increasing the number of keys in the kernel keyring every time we perform an authentication through Kerberos. Note: this is EVERY time, not just offline auth. However, when we're online, no side-effects are experienced. We need to investigate why we're adding keys unnecessarily (and not deleting them)
So I dug deeper into this problem and found that we're not at fault. As I noticed in Comment #17 above, the keyring is expanding on every login, not just our offline logins. However, when I put SSSD in the debugger, I proved that we're not the ones creating these keys (I strongly suspect that pam_keyinit.so is the source) However, the real problem here is that the kernel is not garbage-collecting keys marked as 'expd' (expired) in the keyring. According to the kernel keyring documentation: Similarly, revoked and expired keys will be garbage collected, but only after a certain amount of time has passed. This time is set as a number of seconds in: /proc/sys/kernel/keys/gc_delay By default on Fedora, this is 300 seconds. I performed the following experiment: [sgallagh@sgallagh ~]$ cat /proc/keys 0d430a98 IR-Q-- 1 expd 1f3f0000 13041 13041 keyring _ses: empty 158f8929 IR-Q-- 1 expd 1f3f0000 13041 13041 keyring _ses: empty 15bd971c IR-Q-- 1 expd 1f3f0000 13041 13041 keyring _ses: empty 162a79f7 I--Q-- 3 perm 1f3f0000 13041 13041 keyring _ses: 1/4 1b501319 IR-Q-- 1 expd 1f3f0000 13041 13041 keyring _ses: empty 21020256 IR-Q-- 1 expd 1f3f0000 13041 13041 keyring _ses: empty 243536e9 IR-Q-- 1 expd 1f3f0000 13041 13041 keyring _ses: empty 28eef73e I--Q-- 170 perm 1f3f0000 13041 13041 keyring _ses: 1/4 2c5ae653 IR-Q-- 2 expd 1f3f0000 13041 13041 keyring _ses: empty 2edc26e8 IR-Q-- 1 expd 1f3f0000 13041 13041 keyring _ses: empty 2f747f5f I--Q-- 4 perm 1f3f0000 13041 -1 keyring _uid.13041: empty 3f4bc234 I--Q-- 1 perm 1f3f0000 13041 -1 keyring _uid_ses.13041: 1/4 [sgallagh@sgallagh ~]$ cat /proc/sys/kernel/keys/gc_delay 300 [sgallagh@sgallagh ~]$ cat /proc/keys |wc -l; sleep 301; cat /proc/keys |wc -l 12 12 As you can see, nine of the keys in my keyring above were marked 'expd'. After 300 seconds, they should have been cleaned up by the garbage collector, but they were not.
Can you try the 2.6.38 kernel from F15?
Everything appears to be working properly on kernel-2.6.38-1.fc15.x86_64
(In reply to comment #20) > Everything appears to be working properly on kernel-2.6.38-1.fc15.x86_64 Sooo, we gettin' .38 in F-14? Nice :-)
(In reply to comment #17) > Hmm, we're definitely doing something wrong here. I just ran a simple > experiment on my machine and we're increasing the number of keys in the kernel > keyring every time we perform an authentication through Kerberos. > > Note: this is EVERY time, not just offline auth. However, when we're online, no > side-effects are experienced. > > We need to investigate why we're adding keys unnecessarily (and not deleting > them) My KDC, which has been up for 45 days now and is also F-14 (same kernel, same arch), shows this: # wc -l /proc/keys 37394 /proc/keys # grep exp /proc/keys | wc -l 305 So, the ratio of all keys v. expired keys seems 100:1 there. Looks like a serious leak.
(In reply to comment #19) > Can you try the 2.6.38 kernel from F15? So, is this the official fix for F-14?
(In reply to comment #23) > (In reply to comment #19) > > Can you try the 2.6.38 kernel from F15? > > So, is this the official fix for F-14? Ping... Is 2.6.35.12-88 bringing anything new here?
(In reply to comment #24) > Is 2.6.35.12-88 bringing anything new here? Ha, fodder! commit 04d450668aa58e6202916ad870cdfc73621dee26
This looks fixed in kernel-2.6.35.12-88.fc14.x86_64. Thanks!
Hi Bojan, how it get fixed. I am still seeing this error. Please let me know, how i can fix this problem by changing in kernel. Thanks Priti