Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
A customer experiences an issue while running "oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_C2S" on his system:
probe_file consumes all the memory available (up to 128GB) + all the swap space.
Stracing probe_file while we see memory increase, we can see that it is lstat()'ing /proc/XXX/tasks/YYY/fdinfo/ZZZ continuously:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
11586 12:25:34.497399 lstat("/proc/1434/task/11236/fdinfo/65", {st_dev=makedev(0, 3), st_ino=18613883, st_mode=S_IFREG|0400, st_nlink=1, st_uid=1085, st_gid=100, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_atime_nsec=451000000, st_mtime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_mtime_nsec=451000000, st_ctime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_ctime_nsec=451000000}) = 0 <0.000007>
11586 12:25:34.497444 getxattr("/proc/1434/task/11236/fdinfo/65", "system.posix_acl_access", NULL, 0) = -1 EOPNOTSUPP (Operation not supported) <0.000007>
11586 12:25:34.497525 lstat("/proc/1434/task/11236/fdinfo/66", {st_dev=makedev(0, 3), st_ino=18613884, st_mode=S_IFREG|0400, st_nlink=1, st_uid=1085, st_gid=100, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_atime_nsec=451000000, st_mtime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_mtime_nsec=451000000, st_ctime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_ctime_nsec=451000000}) = 0 <0.000008>
...
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
I cannot reproduce the exact same behaviour on my test system, but see something similar when having numerous processes opening numerous files and having numerous threads: on my test system, I can see swap space being consumed and the rule "xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned" eventually fails with message below:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Title Ensure All Files Are Owned by a Group
Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned
Ident CCE-80135-7
W: probe_file: Minimum free memory limit reached! limit=139723876073984, current=96
Result fail
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
The rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned should not browse /proc at all, but it does, which is likely root cause of the issue.
Last but not least, once the message above is printed, I see probe_file doesn't quit, nor oscap main process.
Using GDB, I can see that thread 1 is doing a pthread_join() on thread 2, which seems waiting on a select:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
(gdb) bt
#0 0x00007f14c3df2017 in pthread_join (threadid=139727059293952, thread_return=thread_return@entry=0x0)
at pthread_join.c:90
#1 0x000055fb44cabbe4 in main (argc=<optimized out>, argv=<optimized out>) at main.c:308
(gdb) info threads
Id Target Id Frame
3 Thread 0x7f14bebc3700 (LWP 11583) "probe_file" pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
2 Thread 0x7f14bdbc1700 (LWP 11585) "probe_file" 0x00007f14c3b10a33 in select ()
at ../sysdeps/unix/syscall-template.S:81
* 1 Thread 0x7f14c6812880 (LWP 11582) "probe_file" 0x00007f14c3df2017 in pthread_join (threadid=139727059293952,
thread_return=thread_return@entry=0x0) at pthread_join.c:90
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f14bdbc1700 (LWP 11585))]
#0 0x00007f14c3b10a33 in select () at ../sysdeps/unix/syscall-template.S:81
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) bt
#0 0x00007f14c3b10a33 in select () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f14c6174cf9 in sch_generic_select (desc=<optimized out>, ev=<optimized out>, timeout=0,
flags=<optimized out>) at sch_generic.c:165
#2 0x00007f14c6177226 in SEAP_packet_recv (ctx=ctx@entry=0x55fb45945060, sd=sd@entry=32,
packet=packet@entry=0x7f14bdbc0cd8) at seap-packet.c:634
#3 0x00007f14c617969d in SEAP_recvmsg (ctx=0x55fb45945060, sd=32, seap_msg=seap_msg@entry=0x7f14bdbc0d58)
at seap.c:359
#4 0x000055fb44cad576 in probe_input_handler (arg=0x7ffc9177e100) at input_handler.c:88
#5 0x00007f14c3df0ea5 in start_thread (arg=0x7f14bdbc1700) at pthread_create.c:307
#6 0x00007f14c3b199fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
oscap doesn't exit because it's waiting for probe_file to exit, but due to the pthread_join/select, nothing happens.
Version-Release number of selected component (if applicable):
openscap-1.2.17-13.el7_9.x86_64
How reproducible:
Always on customer system, somehow on my test system
Steps to Reproduce:
1. Create 100 users
# for i in $(seq 1 100); do useradd -N -g users user$i; echo "redhat" | passwd --stdin user$i; done
2. Compile generate_files in attachment to generate files for the users (group is set to unused group id 9999 on purpose)
# for i in $(seq 1 100); do ./generate_files 1000 $(id -u user$i); done
3. Compile many_files_and_threads in attachment to spawn many processes having many threads and opening many files
# for i in $(seq 1 100); do sudo -u user$i /usr/local/bin/many_files_and_threads 1000 100 & done
--> this will start 100 processes having 100 threads each, which are opening 1000 files each (shared between threads)
4. Run oscap
# /usr/bin/oscap xccdf eval --rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned --profile xccdf_org.ssgproject.content_profile_C2S --results-arf /tmp/oscap_results.xml /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml
5. While oscap runs, strace probe_file for some tim
# timeout 10s strace -fttTvyy -o oscap_10s.strace -s 64 -p <pid of probe_file>
Actual results:
Title Ensure All Files Are Owned by a Group
Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned
Ident CCE-80135-7
W: probe_file: Minimum free memory limit reached! limit=139723876073984, current=96
Result fail
11586 12:25:24.518108 lstat("/proc/1434/task/11211/fd/56", <unfinished ...>
:
11586 12:25:34.497525 lstat("/proc/1434/task/11236/fdinfo/66", {st_dev=makedev(0, 3), st_ino=18613884, st_mode=S_IFREG|0400, st_nlink=1, st_uid=1085, st_gid=100, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_atime_nsec=451000000, st_mtime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_mtime_nsec=451000000, st_ctime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_ctime_nsec=451000000}) = 0 <0.000008>
:
Expected results:
- Not that much memory consumed
- /proc browsing being avoided
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (openscap bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2021:4780
Description of problem: A customer experiences an issue while running "oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_C2S" on his system: probe_file consumes all the memory available (up to 128GB) + all the swap space. Stracing probe_file while we see memory increase, we can see that it is lstat()'ing /proc/XXX/tasks/YYY/fdinfo/ZZZ continuously: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- 11586 12:25:34.497399 lstat("/proc/1434/task/11236/fdinfo/65", {st_dev=makedev(0, 3), st_ino=18613883, st_mode=S_IFREG|0400, st_nlink=1, st_uid=1085, st_gid=100, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_atime_nsec=451000000, st_mtime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_mtime_nsec=451000000, st_ctime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_ctime_nsec=451000000}) = 0 <0.000007> 11586 12:25:34.497444 getxattr("/proc/1434/task/11236/fdinfo/65", "system.posix_acl_access", NULL, 0) = -1 EOPNOTSUPP (Operation not supported) <0.000007> 11586 12:25:34.497525 lstat("/proc/1434/task/11236/fdinfo/66", {st_dev=makedev(0, 3), st_ino=18613884, st_mode=S_IFREG|0400, st_nlink=1, st_uid=1085, st_gid=100, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_atime_nsec=451000000, st_mtime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_mtime_nsec=451000000, st_ctime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_ctime_nsec=451000000}) = 0 <0.000008> ... -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- I cannot reproduce the exact same behaviour on my test system, but see something similar when having numerous processes opening numerous files and having numerous threads: on my test system, I can see swap space being consumed and the rule "xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned" eventually fails with message below: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned Ident CCE-80135-7 W: probe_file: Minimum free memory limit reached! limit=139723876073984, current=96 Result fail -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- The rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned should not browse /proc at all, but it does, which is likely root cause of the issue. Last but not least, once the message above is printed, I see probe_file doesn't quit, nor oscap main process. Using GDB, I can see that thread 1 is doing a pthread_join() on thread 2, which seems waiting on a select: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- (gdb) bt #0 0x00007f14c3df2017 in pthread_join (threadid=139727059293952, thread_return=thread_return@entry=0x0) at pthread_join.c:90 #1 0x000055fb44cabbe4 in main (argc=<optimized out>, argv=<optimized out>) at main.c:308 (gdb) info threads Id Target Id Frame 3 Thread 0x7f14bebc3700 (LWP 11583) "probe_file" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 2 Thread 0x7f14bdbc1700 (LWP 11585) "probe_file" 0x00007f14c3b10a33 in select () at ../sysdeps/unix/syscall-template.S:81 * 1 Thread 0x7f14c6812880 (LWP 11582) "probe_file" 0x00007f14c3df2017 in pthread_join (threadid=139727059293952, thread_return=thread_return@entry=0x0) at pthread_join.c:90 (gdb) thread 2 [Switching to thread 2 (Thread 0x7f14bdbc1700 (LWP 11585))] #0 0x00007f14c3b10a33 in select () at ../sysdeps/unix/syscall-template.S:81 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x00007f14c3b10a33 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f14c6174cf9 in sch_generic_select (desc=<optimized out>, ev=<optimized out>, timeout=0, flags=<optimized out>) at sch_generic.c:165 #2 0x00007f14c6177226 in SEAP_packet_recv (ctx=ctx@entry=0x55fb45945060, sd=sd@entry=32, packet=packet@entry=0x7f14bdbc0cd8) at seap-packet.c:634 #3 0x00007f14c617969d in SEAP_recvmsg (ctx=0x55fb45945060, sd=32, seap_msg=seap_msg@entry=0x7f14bdbc0d58) at seap.c:359 #4 0x000055fb44cad576 in probe_input_handler (arg=0x7ffc9177e100) at input_handler.c:88 #5 0x00007f14c3df0ea5 in start_thread (arg=0x7f14bdbc1700) at pthread_create.c:307 #6 0x00007f14c3b199fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- oscap doesn't exit because it's waiting for probe_file to exit, but due to the pthread_join/select, nothing happens. Version-Release number of selected component (if applicable): openscap-1.2.17-13.el7_9.x86_64 How reproducible: Always on customer system, somehow on my test system Steps to Reproduce: 1. Create 100 users # for i in $(seq 1 100); do useradd -N -g users user$i; echo "redhat" | passwd --stdin user$i; done 2. Compile generate_files in attachment to generate files for the users (group is set to unused group id 9999 on purpose) # for i in $(seq 1 100); do ./generate_files 1000 $(id -u user$i); done 3. Compile many_files_and_threads in attachment to spawn many processes having many threads and opening many files # for i in $(seq 1 100); do sudo -u user$i /usr/local/bin/many_files_and_threads 1000 100 & done --> this will start 100 processes having 100 threads each, which are opening 1000 files each (shared between threads) 4. Run oscap # /usr/bin/oscap xccdf eval --rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned --profile xccdf_org.ssgproject.content_profile_C2S --results-arf /tmp/oscap_results.xml /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml 5. While oscap runs, strace probe_file for some tim # timeout 10s strace -fttTvyy -o oscap_10s.strace -s 64 -p <pid of probe_file> Actual results: Title Ensure All Files Are Owned by a Group Rule xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned Ident CCE-80135-7 W: probe_file: Minimum free memory limit reached! limit=139723876073984, current=96 Result fail 11586 12:25:24.518108 lstat("/proc/1434/task/11211/fd/56", <unfinished ...> : 11586 12:25:34.497525 lstat("/proc/1434/task/11236/fdinfo/66", {st_dev=makedev(0, 3), st_ino=18613884, st_mode=S_IFREG|0400, st_nlink=1, st_uid=1085, st_gid=100, st_blksize=1024, st_blocks=0, st_size=0, st_atime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_atime_nsec=451000000, st_mtime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_mtime_nsec=451000000, st_ctime=1614252334 /* 2021-02-25T12:25:34.451000000+0100 */, st_ctime_nsec=451000000}) = 0 <0.000008> : Expected results: - Not that much memory consumed - /proc browsing being avoided