Hide Forgot
Description of problem: On CentOS 8 Stream kernel 4.18.0-348.el8.x86_64 (originally discovered on CentOS 8.3), I'm seeing multiple users exceed their kernel keyring keys quota, and checking the output of `/proc/keys` for the affected users almost all keys are expired/revoked with a refcount of 1 or higher. Being revoked, these cannot be interacted with using keyctl. It appears that either garbage collection is not running, or these expired/revoked keys are not being cleaned up when GC does run. I have verified that downgrading to an 8.1 kernel (leaving the userspace on 8.3), the GC does run correctly and and expired keys do not hang around indefinitely. Version-Release number of selected component (if applicable): CentOS 8.3 userspace with CentOS 8 Stream 4.18.0-348.el8.x86_64 kernel How reproducible: Always on affected kernel versions Kernel versions tested: - 8.1 4.18.0-147.5.1.el8.x86_64 NOT affected - 8.3 4.18.0-240.1.1.el8.x86_64 affected - 8.3 4.18.0-240.22.1.el8.x86_64 affected - 8-stream 4.18.0-348.el8.x86_64 affected Steps to Reproduce: - Boot system with 8 Stream, or CentOS 8.3/8.4 kernel - SSH in to the host, logout, repeat a few times - Observe from /proc/key-users that the number of keys in use is increasing - Observe from /proc/keys that keys relating to the logged out SSH connections are revoked/expired - Wait > 5 minutes for the GC routine to run - Observe that expired/revoked keys have not been cleaned up - On a system with sufficient activity, observe that the per-user quota limit will be reached and subsequently new keys cannot be created. Actual results: - /proc/keys fills with expired/revoked keys - /proc/key-users shows the per-user quota limit is reached - Attempts to create new keyrings fail with quota error Expected results: - Revoked/expired keys to be garbage collected in a timely fashion Additional info: Example list of keys when the host is in this situation. ``` $ cat /proc/keys 00757a86 IR-Q--- 2 expd 3f030000 15171 5000 keyring _ses: empty 00aebcec IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 00d874cc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 00f354b4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 01317f10 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 01fc9be6 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0274d916 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 02fa88a1 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0301f42c IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0307418a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0337fe9a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0363e414 I--Q--- 3 perm 1f3f0000 15171 65534 keyring _uid.15171: empty 03a58bbc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 03ded0ca IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 03e00b58 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 047825a4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 049eb199 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 04d34978 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 052f1971 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 065b1604 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 065df0d3 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 066f89d9 I--Q--- 1 perm 1f3f0000 15171 65534 keyring _uid_ses.15171: 1 06749a56 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0680f7be IR-Q--- 18 expd 3f030000 15171 5000 keyring _ses: empty # ... ```