Bug 2021088 - Expired/Revoked kernel keyring keys not garbage collected in 8.3, 8-stream kernels
Summary: Expired/Revoked kernel keyring keys not garbage collected in 8.3, 8-stream ke...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: kernel
Version: CentOS Stream
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: David Howells
QA Contact: Linqing Lu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-08 10:37 UTC by Ben Roberts
Modified: 2021-11-09 21:25 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
CentOS 18343 0 None None None 2021-11-08 10:37:23 UTC
Red Hat Issue Tracker RHELPLAN-101952 0 None None None 2021-11-08 10:38:28 UTC

Description Ben Roberts 2021-11-08 10:37:24 UTC
Description of problem:

On CentOS 8 Stream kernel 4.18.0-348.el8.x86_64 (originally discovered on CentOS 8.3), I'm seeing multiple users exceed their kernel keyring keys quota, and checking the output of `/proc/keys` for the affected users almost all keys are expired/revoked with a refcount of 1 or higher. Being revoked, these cannot be interacted with using keyctl. It appears that either garbage collection is not running, or these expired/revoked keys are not being cleaned up when GC does run.

I have verified that downgrading to an 8.1 kernel (leaving the userspace on 8.3), the GC does run correctly and and expired keys do not hang around indefinitely.

Version-Release number of selected component (if applicable):
CentOS 8.3 userspace with CentOS 8 Stream 4.18.0-348.el8.x86_64 kernel

How reproducible:
Always on affected kernel versions

Kernel versions tested:
- 8.1 4.18.0-147.5.1.el8.x86_64 NOT affected
- 8.3 4.18.0-240.1.1.el8.x86_64 affected
- 8.3 4.18.0-240.22.1.el8.x86_64 affected
- 8-stream 4.18.0-348.el8.x86_64 affected


Steps to Reproduce:
- Boot system with 8 Stream, or CentOS 8.3/8.4 kernel
- SSH in to the host, logout, repeat a few times
- Observe from /proc/key-users that the number of keys in use is increasing
- Observe from /proc/keys that keys relating to the logged out SSH connections are revoked/expired
- Wait > 5 minutes for the GC routine to run
- Observe that expired/revoked keys have not been cleaned up
- On a system with sufficient activity, observe that the per-user quota limit will be reached and subsequently new keys cannot be created.

Actual results:

- /proc/keys fills with expired/revoked keys
- /proc/key-users shows the per-user quota limit is reached
- Attempts to create new keyrings fail with quota error

Expected results:
- Revoked/expired keys to be garbage collected in a timely fashion

Additional info:

Example list of keys when the host is in this situation.
```
$ cat /proc/keys
00757a86 IR-Q--- 2 expd 3f030000 15171 5000 keyring _ses: empty
00aebcec IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
00d874cc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
00f354b4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
01317f10 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
01fc9be6 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0274d916 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
02fa88a1 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0301f42c IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0307418a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0337fe9a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0363e414 I--Q--- 3 perm 1f3f0000 15171 65534 keyring _uid.15171: empty
03a58bbc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
03ded0ca IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
03e00b58 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
047825a4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
049eb199 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
04d34978 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
052f1971 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
065b1604 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
065df0d3 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
066f89d9 I--Q--- 1 perm 1f3f0000 15171 65534 keyring _uid_ses.15171: 1
06749a56 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0680f7be IR-Q--- 18 expd 3f030000 15171 5000 keyring _ses: empty
# ...
```


Note You need to log in before you can comment on or make changes to this bug.