Bug 2161499

Summary:	Insights compliance command fails with scan failed error
Product:	Red Hat Enterprise Linux 8	Reporter:	Nikhil Gupta <ngupta>
Component:	openscap	Assignee:	Evgeny Kolesnikov <ekolesni>
Status:	CLOSED WONTFIX	QA Contact:	BaseOS QE Security Team <qe-baseos-security>
Severity:	high	Docs Contact:	Petr Hybl <phybl>
Priority:	unspecified
Version:	8.6	CC:	ahumbe, bblasco, ekolesni, gfialova, jafiala, jcerny, mhaicman, mmarhefk, phybl
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	.OpenSCAP memory-consumption problems On systems with limited memory, the OpenSCAP scanner might stop prematurely or it might not generate the results files. To work around this problem, you can customize the scanning profile to deselect rules that involve recursion over the entire `/` file system: * `rpm_verify_hashes` * `rpm_verify_permissions` * `rpm_verify_ownership` * `file_permissions_unauthorized_world_writable` * `no_files_unowned_by_user` * `dir_perms_world_writable_system_owned` * `file_permissions_unauthorized_suid` * `file_permissions_unauthorized_sgid` * `file_permissions_ungroupowned` * `dir_perms_world_writable_sticky_bits` For more details and more workarounds, see the related link:https://access.redhat.com/articles/6999111[Knowledgebase article].	Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-03-27 08:56:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nikhil Gupta 2023-01-17 06:14:16 UTC

Description of problem:
While running the 'insights-client --compliance' command, getting the following error:
~~~
2022-11-30 13:38:27,611 INFO insights.client.apps.compliance Running scan for xccdf_org.ssgproject.content_profile_cis_workstation_l1... this may take a while
2022-11-30 13:38:27,612 DEBUG insights.util.subproc Executing: [['oscap', 'xccdf', 'eval', '--profile', 'xccdf_org.ssgproject.content_profile_cis_workstation_l1', '--results', '/var/tmp/insights-client/insights-archive-hi4pf70l/insights-dmzgrg06.gbc.local-20221130133822/oscap_results-xccdf_org.ssgproject.content_profile_cis_workstation_l1.xml', '/usr/share/xml/scap/ssg/content/ssg-rhel8-ds-1.2.xml']]
2022-11-30 13:42:41,702 ERROR insights.client.apps.compliance Scan failed
2022-11-30 13:42:41,882 ERROR insights.client.apps.compliance WARNING: This content points out to the remote resources. Use `--fetch-remote-resources' option to download them.
~~~

The oom-killer has been invoked while running compliance command:
~~~
Dec  1 11:28:48 dmzgrg06 kernel: nsrexecd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec  1 11:28:48 dmzgrg06 kernel: CPU: 1 PID: 1549 Comm: nsrexecd Kdump: loaded Not tainted 4.18.0-372.32.1.el8_6.x86_64 #1
Dec  1 11:28:48 dmzgrg06 kernel: Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.0 12/17/2019
Dec  1 11:28:48 dmzgrg06 kernel: Call Trace:
Dec  1 11:28:48 dmzgrg06 kernel: dump_stack+0x41/0x60
Dec  1 11:28:48 dmzgrg06 kernel: dump_header+0x4a/0x1df
Dec  1 11:28:48 dmzgrg06 kernel: oom_kill_process.cold.32+0xb/0x10
Dec  1 11:28:48 dmzgrg06 kernel: out_of_memory+0x1bd/0x4e0
Dec  1 11:28:48 dmzgrg06 kernel: __alloc_pages_slowpath+0xbe0/0xcc0
Dec  1 11:28:48 dmzgrg06 kernel: __alloc_pages_nodemask+0x2e2/0x320
Dec  1 11:28:48 dmzgrg06 kernel: pagecache_get_page+0xce/0x310
Dec  1 11:28:48 dmzgrg06 kernel: filemap_fault+0x78b/0xa10
Dec  1 11:28:48 dmzgrg06 kernel: ? __mod_lruvec_page_state+0x5e/0x80
...
Dec  1 11:28:49 dmzgrg06 kernel: [3353301]     0 3353301  3426205  1329185 22974464  1405758             0 oscap
Dec  1 11:28:49 dmzgrg06 kernel: [3362023]     0 3362023     1830       18    61440        0             0 sleep
Dec  1 11:28:49 dmzgrg06 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1012.slice/session-459.scope,task=oscap,pid=3353301,uid=0
Dec  1 11:28:49 dmzgrg06 kernel: Out of memory: Killed process 3353301 (oscap) total-vm:13704820kB, anon-rss:5316740kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:22436kB oom_score_adj:0
Dec  1 11:28:49 dmzgrg06 kernel: oom_reaper: reaped process 3353301 (oscap), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
~~~

Version-Release number of selected component (if applicable):
openscap-1.3.6-3.el8.x86_64
openscap-scanner-1.3.6-3.el8.x86_64
scap-security-guide-0.1.63-1.el8_6.noarch

How reproducible:
Always (On one of the hosts in the customer environment)

Steps to Reproduce:
1. Register host with insights
2. Assign 'cis_workstation_l1' compliance policy to the host
3. Run the 'insights-client --compliance' command on the host.

Actual results:
The compliance scan failed and the result is not uploaded to the cloud.

Expected results:
The scan should be successful and the result should be uploaded to the cloud.

Additional info:
The server has a mount /u01 for 750G, with 550G full.

Comment 1 Nikhil Gupta 2023-01-17 06:17:46 UTC

oscap --verbose DEVEL xccdf eval --profile cis_server_l1 --report cis_server_l1.html /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml

and after an hour or more it got me this:

==========================================================
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
Killed
[root@dmzgrg06 ~]#
===========================================================

I did not kill it, something inside did it.


Scrolling up, I get the error here:

=================================================
D: oscap:     Signaling `notfull' [oscap(1916230):icache_worker(7f79f6ffd700):icache.c:241:probe_icache_worker]
D: oscap:     Handling NOP [oscap(1916230):icache_worker(7f79f6ffd700):icache.c:265:probe_icache_worker]
D: oscap:     Sync [oscap(1916230):probe_worker(7f78e47f0700):icache.c:477:probe_icache_nop]
D: oscap:     old flag: 0, new flag: 2. [oscap(1916230):probe_worker(7f78e47f0700):probe-api.c:688:probe_cobj_set_flag]
D: oscap:     handler result = 0x7f79e8030ba0, return code = 0 [oscap(1916230):probe_worker(7f78e47f0700):worker.c:99:probe_worker_runfn]
D: oscap:     probe thread deleted [oscap(1916230):probe_worker(7f78e47f0700):worker.c:119:probe_worker_runfn]
D: oscap:     Sorting blocks & building iterator array [oscap(1916230):probe_worker(7f78e47f0700):sexp-manip.c:1408:SEXP_list_sort]
D: oscap:     Iterator count = 1 [oscap(1916230):probe_worker(7f78e47f0700):sexp-manip.c:1442:SEXP_list_sort]
D: oscap:     cnt = 0 [oscap(1916230):probe_worker(7f78e47f0700):seap-message.c:138:SEAP_msgattr_exists]
D: oscap:     no-reply not set: sending full reply [oscap(1916230):probe_worker(7f78e47f0700):seap.c:481:SEAP_reply]
D: oscap:     MSG -> SEXP [oscap(1916230):probe_worker(7f78e47f0700):seap-packet.c:260:SEAP_packet_msg2sexp]
D: oscap: ("seap.msg" ":id" 29 ":reply-id" 29 (2 () ((("linux:rpminfo_item" ":id" "11916230751678" ) ("name" "xorg-x11-server-common" ) ("arch" "x86_64" ) ("epoch" "(none)" ) ("release" "9.el8" ) ("version" "1.20.11" ) [evr_string]("evr" "0:1.20.11-9.el8" ) ("signature_keyid" "199e2f91fd431d51" ) ("extended_name" "xorg-x11-server-common-0:1.20.11-9.el8.x86_64" ) ) ) () ) ) [oscap(1916230):probe_worker(7f78e47f0700):seap-packet.c:261:SEAP_packet_msg2sexp]
D: oscap:     packet size: 2076 [oscap(1916230):probe_worker(7f78e47f0700):seap-packet.c:262:SEAP_packet_msg2sexp]
D: oscap:     name=reply-id, value=0x7f79e8033550 [oscap(1916230):probe_worker(7f78e47f0700):seap-message.c:73:SEAP_msg_free]
D: oscap:     probe_worker_runfn has finished [oscap(1916230):probe_worker(7f78e47f0700):worker.c:179:probe_worker_runfn]
D: oscap:     Received packet [oscap(1916230):oscap(7f7a64d5d0c0):seap-packet.c:794:SEAP_packet_recv]
D: oscap: ("seap.msg" ":id" 29 ":reply-id" 29 (2 () ((("linux:rpminfo_item" ":id" "11916230751678" ) ("name" "xorg-x11-server-common" ) ("arch" "x86_64" ) ("epoch" "(none)" ) ("release" "9.el8" ) ("version" "1.20.11" ) [evr_string]("evr" "0:1.20.11-9.el8" ) ("signature_keyid" "199e2f91fd431d51" ) ("extended_name" "xorg-x11-server-common-0:1.20.11-9.el8.x86_64" ) ) ) () ) ) [oscap(1916230):oscap(7f7a64d5d0c0):seap-packet.c:795:SEAP_packet_recv]
D: oscap:     packet size: 2076 [oscap(1916230):oscap(7f7a64d5d0c0):seap-packet.c:796:SEAP_packet_recv]
D: oscap:     Message received. [oscap(1916230):oscap(7f7a64d5d0c0):oval_probe_ext.c:579:oval_probe_comm]
D: oscap:     name=(null), value=0x7f79e80363c0 [oscap(1916230):oscap(7f7a64d5d0c0):seap-message.c:73:SEAP_msg_free]
D: oscap:     Syschar entry type: 9003 'linux' => decoded OK [oscap(1916230):oscap(7f7a64d5d0c0):oval_sexp.c:982:oval_sexp_to_sysitem]
I: oscap:     Test 'oval:ssg-test_package_xorg-x11-server-common_removed:tst:1' requires that none of the objects defined by 'oval:ssg-obj_test_package_xorg-x11-server-common_removed:obj:1' exist on the system. [oscap(1916230):oscap(7f7a64d5d0c0):oval_resultTest.c:909:_oval_result_test_evaluate_items]
I: oscap:     1 objects defined by 'oval:ssg-obj_test_package_xorg-x11-server-common_removed:obj:1' exist on the system. [oscap(1916230):oscap(7f7a64d5d0c0):oval_resultTest.c:918:_oval_result_test_evaluate_items]
I: oscap:     Test 'oval:ssg-test_package_xorg-x11-server-common_removed:tst:1' does not contain any state to compare object with. [oscap(1916230):oscap(7f7a64d5d0c0):oval_resultTest.c:920:_oval_result_test_evaluate_items]
I: oscap:     All items matching object 'oval:ssg-obj_test_package_xorg-x11-server-common_removed:obj:1' were collected. (flag=complete) [oscap(1916230):oscap(7f7a64d5d0c0):oval_resultTest.c:963:_oval_result_test_evaluate_items]
I: oscap:   Test 'oval:ssg-test_package_xorg-x11-server-common_removed:tst:1' evaluated as false. [oscap(1916230):oscap(7f7a64d5d0c0):oval_resultTest.c:1164:oval_result_test_eval]
I: oscap: Definition 'oval:ssg-package_xorg-x11-server-common_removed:def:1' evaluated as false. [oscap(1916230):oscap(7f7a64d5d0c0):oval_resultDefinition.c:170:oval_result_definition_eval]
Result  fail

D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]
D: oscap: rbt_str_add: non-zero return code [oscap(1916230):oscap(7f7a64d5d0c0):oval_string_map.c:211:oval_string_map_put]

Comment 2 Evgeny Kolesnikov 2023-01-20 05:39:40 UTC

Our preliminary investigation result: the OOM situation could be caused by one of the rules that traverse filesystems looking for invalid rights/ownership.

The suspects are:

- file_permissions_unauthorized_world_writable
- no_files_unowned_by_user
- file_permissions_ungroupowned

In order to confirm if this is the case one can either:

- execute oscap --verbose DEVEL xccdf eval --profile cis_server_l1 --skip-rule file_permissions_unauthorized_world_writable --skip-rule no_files_unowned_by_user --skip-rule file_permissions_ungroupowned --report cis_server_l1.html /usr/share/xml/scap/ssg/content/ssg-rhel8-ds.xml, effectively excluding these rules from the scan
- or temporarily unmount /u01 during the scan

The workaround for the problem is to exclude all offending rules from the scan using a tailoring file.

Comment 6 Evgeny Kolesnikov 2023-02-28 09:10:19 UTC

*** Bug 2147604 has been marked as a duplicate of this bug. ***

Comment 7 Jan Černý 2023-03-27 08:56:34 UTC

This bug boils down to the outstanding memory problems of OpenSCAP which are covered by the KBS https://access.redhat.com/articles/6999111 and that are unlikely to be fixed.