I have tried to narrow the problem and I believe I have reproduced it on limited set. What troublesome OVAL is doing is to check existence of directories provided by variable. When variable contains only one entry, it works correctly. If it contains more than one value, it starts traversing whole subdirectory tree, instead of acknowledging just the existence. I am going to upload reduced OVAL which reproduces this situation, when two users are available on the system:

useradd -M -u 1234 -d /root/ test1                                              
useradd -M -u 1235 -d / test2

See comment on line 50 of the fail_oval.xml

Created attachment 1552369 [details]
minimalized reproducer oval

Created attachment 1552405 [details]
minimalized reproducer oval

Created attachment 1552406 [details]
minimalized reproducer oval

I have investigated this bug more and I have found multiple problems.

I think the OVAL definition oval:mil.disa.stig.rhel7:def:86639 is logically incorrect. One of the tests in this definition, the OVAL test oval:mil.disa.stig.rhel7:tst:8663900 check existence of items described by OVAL object oval:mil.disa.stig.rhel7:obj:8663900. This OVAL object checks the file path specified in variable oval:mil.disa.stig.rhel7:var:8663901. The variable can contain multiple different values, specifying directory paths. The OVAL object sets attributes var_check="all" and operation="equals". Their meaning is that to match the object definition every examined item needs to be equal to all values of the referenced variable at the same time. In this case it isn't possible to satisfy this condition because the referenced variable contains multiple different directory paths - the home directories of users. The check means that there should exist a single directory that has the same path as all the home directories of all the users. That's obviously wrong check. See definition of var_check attribute in https://oval.mitre.org/language/version5.11/ovaldefinition/documentation/oval-definitions-schema.html that contains a nice explanation of this behavior. We need to write a different check that would be logically correct.

I have created a simple demonstration OVAL that demonstrates the behavior of OpenSCAP in this situation. See the attachment demo.zip. In the OVAL definition there is only a single file_object that references a constant_variable using var_check="all" and operation="equals" as in this bugzilla. The constant variable contains 2 different directory paths. When OpenSCAP evaluates the test it first takes the first directory path from the constant_variable values and recursively browses all its subdirectories. Then it takes the scond  directory path from the constant_variable values and recursively browses all its subdirectories. That behavior is wrong, because there is no need for recursive descend because as soon as you verify the directory exists you don't care about its subdirectories. Notice that this happens only if there are 2 or more values in the constant_variable. If one of the values is a big directory tree it causes problems. If one of the values "/" then the whole filesystem is browsed and all the subdirectories at all the levels are collected. This takes very long and the amount of all the data is big and at some point it exceeds the limits and cause the probe to crash.

In my opinion, OpenSCAP doesn't have any special problem with double slash "//" as a file path on inside a file path. It can understand it as a single slash "/" and correctly use it as "/". However, in this case, due to bug described in the previous paragraph, it leads to scanning the whole filesystem and to crash.

Created attachment 1552518 [details]
demo OVAL

This issue was not selected to be included in Red Hat Enterprise Linux 7.8 because it is manifesting in scenario using broken content (see Comment 6 for details). Internal analysis identified fix as moderately risky. The next release will be in Maintenance Support 1 Phase, which means that qualified Critical and Important Security errata advisories (RHSAs) and Urgent Priority Bug Fix errata advisories (RHBAs) may be released as they become available. We are deferring this issue to Red Hat Enterprise Linux 8, but if you believe that it qualifies for the Maintenance Support 1 Phase, please re-open.

Upstream test case: https://github.com/OpenSCAP/openscap/pull/1534

https://github.com/OpenSCAP/openscap/pull/1534

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openscap bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4623