The SOSReport from the customer case reveals that these packages are used:

openscap-1.2.16-8
scap-security-guide-0.1.36-9 (SSG)

The problem which has been logged indicates that there occur:

- a regexp matching that involves strings that are not valid UTF8, or
- a bug in the scanner causes memory corruption, so although no SSG or local content are invalid UTF8, those strings are created in the scanner.

The rule which manifests the problem is xccdf_org.ssgproject.content_rule_audit_rules_privileged_commands.

The customer case mentions that some elements of the datastream have been changed (... I've set all recurse_file_system items in ssg-rhel7-ds-dll.xml to "local" ...), so is it correct to assume that you have used tailoring? If so, could you please share with us what were the changes?

Anyway, the datastream in scap-security-guide-0.1.36-9 contains unanchored filepaths, which may trigger this issue, so it may be that scap-security-guide in 7.6 doesn't suffer from this. You can extract the datastream from the package: http://download.eng.bos.redhat.com/brewroot/packages/scap-security-guide/0.1.40/12.el7/noarch/scap-security-guide-0.1.40-12.el7.noarch.rpm

The nature of this issue is a little bit unclear - we can safely say that the scanner could provide better feedback, so it is easier to pinpoint issues like this one.
However, the trigger for this bug may be a bug in the SSG content.

Dear Matus,
Very thanks for response.

||ssg-rhel7-ds-dll.xml
Where we can find this file. How its generated, and what's its purpose?
since i cannot find it in /usr/share/xml/scap/ssg/content.
Is it same as 'ssg-rhel7-ds.xml'?

||You can extract the datastream from the package: scap-security-guide-0.1.40-12.el7.noarch.rpm
I believe this is the option, when customer does not move to "scap-security-guide-0.1.40-12.el7.noarch.rpm", since this rpm will provide upgraded datastream.
Am I correct?

I have relayed the instruction to customer. Will get back with update :)

Hello, the mentioning of ssg-rhel7-ds-dll.xml is a quote of customer from the customer case that indicates that the customer used tailoring, or otherwise edited the datastream.

Concerning 0.1.40-12.el7.noarch.rpm, you are right - it is a datastream shipped in RHEL 7.6, that contains, among other things, bugfixes for the 7.5 content.

Dear Matus/Matej,
Great Thanks for response.

This is update from Customer:
1. Changed recurse_file_system=local (everywhere) but did not helped.
2.  No tailoring file used.
3. Installed scap-security-guide-0.1.40-12.el7.noarch.rpm(Provided by RHEL-7.6)
 - changed "recurse_file_system" everywhere in /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml into "local" again
 - restarted the scan with:
/usr/bin/oscap xccdf eval \
  --profile xccdf_org.ssgproject.content_profile_stig-rhel7-disa \
  --results-arf arf.xml \
  --verbose WARNING \
  --report report.$$.html \
  /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml >msr.$$.out 2>&1
 - still the same errors (pcre_exec function) and I also see oscap still accessing the remote filesystems (all processes below except the automounter
dll root@zeus ~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv         12G  2.4G  9.7G  20% /
devtmpfs                         1.9G     0  1.9G   0% /dev
/dev/sda1                        509M  115M  394M  23% /boot
/dev/mapper/rootvg-varlv         4.0G  2.4G  1.7G  58% /var
/dev/mapper/datavg02-dbalv       175G   33M  175G   1% /app/dll_dba
/dev/mapper/datavg01-linuxlv      80G   34G   47G  42% /dll_linux
/dev/mapper/rootvg-tmplv         4.0G   52M  4.0G   2% /tmp
/dev/mapper/datavg01-dokuwikilv   20G  308M   20G   2% /app/dokuwiki
/dev/mapper/rootvg-homelv       1014M  381M  634M  38% /home
bess5053:/DLL_unix               180G  155G   26G  86% /nfs/DLL_unix
ehvs6215:/stage                  1.8T  1.7T  103G  95% /nfs/dbastage

dll root@zeus ~# lsof | egrep '/nfs/DLL_unix|/nfs/dbastage'
automount   9891                root   21r      DIR               0,43         0      47205 /nfs/DLL_unix
automount   9891                root   22r      DIR               0,44         0      47206 /nfs/dbastage
automount   9891  9901          root   21r      DIR               0,43         0      47205 /nfs/DLL_unix
automount   9891  9901          root   22r      DIR               0,44         0      47206 /nfs/dbastage
automount   9891  9902          root   21r      DIR               0,43         0      47205 /nfs/DLL_unix
automount   9891  9902          root   22r      DIR               0,44         0      47206 /nfs/dbastage
automount   9891  9983          root   21r      DIR               0,43         0      47205 /nfs/DLL_unix
automount   9891  9983          root   22r      DIR               0,44         0      47206 /nfs/dbastage
automount   9891 10003          root   21r      DIR               0,43         0      47205 /nfs/DLL_unix
automount   9891 10003          root   22r      DIR               0,44         0      47206 /nfs/dbastage
automount   9891 10008          root   21r      DIR               0,43         0      47205 /nfs/DLL_unix
automount   9891 10008          root   22r      DIR               0,44         0      47206 /nfs/dbastage
probe_fil  56405                root    3r      DIR               0,45     40960    2702375 /nfs/dbastage/11.2.0/ctx/data/inxight/lang (ehvs6215:/stage)
icache_wo  56405 56406          root    3r      DIR               0,45      4096    2702373 /nfs/dbastage/11.2.0/ctx (ehvs6215:/stage)
signal_ha  56405 56407          root    3r      DIR               0,45      4096    2702656 /nfs/dbastage/11.2.0/ctx/data (ehvs6215:/stage)
probe_wor  56405 56409          root    3r      DIR               0,45     40960    2702375 /nfs/dbastage/11.2.0/ctx/data/inxight/lang (ehvs6215:/stage)

Dear Matus/Matej,

Supplementary information from Customer.
The only thing I changed was to NOT descend remote filesystems.

Customer is chasing for update, Can you kindly assist.

It looks like that the customer edited the OVAL in the datasteam, setting recurse_file_system="local" in file_object/behaviors. Despite that, the scanner accesses remote filesystems - this has been determined by lsof.

Remote systems access is not directly related to this issue itself, but this setting could be used as a workaround. We have to investigate this and double-check how it could not work.
We can't resolve it now, as majority of the team is on PTO - we will be able look into it after the Christmas holiday.

Dear Matěj Týč ,

My question to customer 
- how "ssg-rhel7-ds-dll.xml" came into the picture, since oscap provides /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml.

Customer's response:
- See my message of Dec 11 2018 at 10:43 AM +01:00 for lsof output.  See the last 4 processes. They are part of oscap.
dll root@zeus ~# lsof | egrep '/nfs/DLL_unix|/nfs/dbastage'
.......
probe_fil  56405                root    3r      DIR               0,45     40960    2702375 /nfs/dbastage/11.2.0/ctx/data/inxight/lang (ehvs6215:/stage)
icache_wo  56405 56406          root    3r      DIR               0,45      4096    2702373 /nfs/dbastage/11.2.0/ctx (ehvs6215:/stage)
signal_ha  56405 56407          root    3r      DIR               0,45      4096    2702656 /nfs/dbastage/11.2.0/ctx/data (ehvs6215:/stage)
probe_wor  56405 56409          root    3r      DIR               0,45     40960    2702375 /nfs/dbastage/11.2.0/ctx/data/inxight/lang (ehvs6215:/stage)

The customer is discussing processes:
probe_fil
icache_wo
signal_ha
probe_wor

But as I go into the source code of openscap scanner: openscap-1.2.17
I cannot find particular processes forked, neither in /usr/libexec/openscap/

Ticket is now, waiting on Engineering.
Thanks

Have you added <behaviors recurse_file_system="local"> as a child element to every file_object element in the ssg-rhel7-ds.xml? Or have you only changed already existing occurrences of behaviors element? In other words, is there any file_object that doesn't contain behaviors child element?

Dear Jan Černý,
We have relayed the query to customer, will revert as he responds.
Thanks

Information from Customer:
I only changed already existing occurrences of behaviors elements.

Can we achieve this?
- Is there any other way to only recurse local filesystems?
- If not: how to change the xml file to prevent recursion of all remote filesystems?

> Is there any other way to only recurse local filesystems?
No, there isn't, because unfortunately the OVAL specification doesn't have anything that would allow to set this option globally.

> If not: how to change the xml file to prevent recursion of all remote filesystems?
Basically you need to make sure no OVAL object describes any entity from the remote filesystems. Add <behaviors recurse_file_system="local"> as a child to every file_object, fileextendedattribute_object, selinuxsecuritycontext_object, filehash_object, filehash58_object, textfilecontent54_object element.

Created attachment 1522150 [details]
Patched openscap library

Created attachment 1522151 [details]
Corresponding openscap scanner

Please try to install the attached packages - they should give you an updated openscap library that prints a more helpful error message that we hope could show us what string exactly is causing the problem.

Those packages can be seen as successors of RHEL7.6 packages - there are no earth-shaking changes.
The openscap RPM package should be enough, but I have also attached the openscap-scanner in case.

Dear Matěj Týč,

Customer installed the rpms ran the test and provided the attached output(mr.38348.out).
Kindly have a look.

Appreciate your help!
Amit

Here are a few notes from our investigation investigation (no solution yet):

The problem manifests itself on the following rules:
* xccdf_org.ssgproject.content_rule_no_user_host_based_files
* xccdf_org.ssgproject.content_rule_no_host_based_files
* xccdf_org.ssgproject.content_rule_audit_rules_privileged_commands
* xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned

Those rules work all the similar way. They use an OVAL that searches the whole filesystem
for files whose name matches a regular expression. In order to find the files matching
the given regular expression, we need to examine the name of every single file.

The problem is that currently we support only UTF-8 strings in OpenSCAP. However, the
file names can have arbitrary encoding.

In order to treat the file names correctly, 

Changing aforementioned rules in a way they check only certain paths doesn't seem to be a proper solution,
because at any path there can exist a file.
However, we can improve the rules so that it is less likely.

Due to this, the solution is most likely changing OpenSCAP library code.

We need to investigate if there is a way how to check if a given file name is a valid UTF-8 string and if
not, we will have to implement a conversion function that will attempt to covert the string to UTF-8
to be sure we run a regex match only on valid UTF-8 string.

Dear Jan Černý,

Many thanks for update on investigation.
We are curious for forthcoming solution.

Thanks

We gave the problem a thought, and I ended up with this conclusion:

- A filename is a sequence of bytes that may be encoded using an encoding. When doing string operations on the filename (s.a. name matching), it needs to be transformed from the byte sequence to a string, i.e. decoded. When one attempts to decode bytes using incorrect decoding algorithm, one may either encounter a failure that indicates that the decoding algorithm is not the right one, but one may also end up with incorrectly decoded string without an indication that something went wrong. In a nutshell, it is impossible to reliably detect the encoding if it is not unknown.
- OpenSCAP assumes that filenames are encoded in a UTF-8 compatible encoding. The log output has revealed several filenames that are encoded differently (our guess is that the ISO-8859-2 encoding has been used), which has caused the error to manifest. The sosreport indicates that the system locale is set to UTF8, so there is no way for the scanner how to determine the right encoding.
- One possible fix for that would involve the ability of the openscap library to accept "hints", i.e. an ordered list of encodings to try if the preferred encoding fails. However, this approach has it's inherent flaws, s.a. if there are multiple encodings in the filesystem, an incorrect one may be picked anyway. So instead, I propose you to go through the offending files and re-encode their filenames, as they may be causing you problems elsewhere. There is a related article about this in the Red Hat knowledge Base: https://access.redhat.com/solutions/23252 In a nutshell, you can use the convmv utility to fix the encoding to the expected value, and you will get rid of OpenSCAP problems you experience right now, and you may avoid future problems coming from naming files in a way that is not used and expected nowadays.

Dear Matěj Týč,

Appreicate your investigation and response.
I have asked customer to go through the offending files and re-encode filenames to UTF-8.
Will revert with Customer's response.

Dear Matěj Týč,

=======Customer have this query=====
1. Is it possible to use filters in ssg-rhel7-ds.xml to prevent descending into remote filesystems?  I know filtering is an option but I don't know how to deploy it.
2. Another option for me is to exclude the 4 rules.
====================================

I have asked him to try option-2, But insights of option-1 would be great!

Customer tried option 2
- Another option for me is to exclude the 4 rules.

This works but the result is an incomplete report.

We have investigated the problem of remote file systems. We think that setting recurse_file_system="local" should work. There might be 2 possibilities why this attribute is ignored. We need to narrow down the problem to one of them.
The possibilities are:
1. There might be a bug in OpenSCAP during processing recurse_file_system which we weren't able to reproduce so far.
2. OpenSCAP doesn't recognize your remote file system as a remote filesystem. A filesystem is considered remote filesystem by OpenSCAP if it starts either with 
// or it contains /: in mnt_fsname returned by getmntent(3).

We have checked /etc/fstab in your sosreport and no filesystem with name is there. Could you tell us what way the remote system is mounted? Should we detect that the filesystem is remote?

Dear Jan Černý,
Thanks for response.

||Should we detect that the filesystem is remote?
Do you mean remote session with Customer?
We use bomgar tool to do it, Will you join the remote, If so I will arrange a remote session. What time/day suits you?
How will you join the call?
I customer is located in Europe/Paris?

Dear Jan Černý,
Response from Customer:
Mounting takes place via the automounter.
So if nothing has been mounted and oscap starts all defined remote mounts will be activated.

In order to skip files or directories, my means of content modification, you have to edit the datastream and find all file_object and textfilecontent54 elements related to affected rules which could possibly match the path to your external filesystems. Once you will have all affected file_object elements, you will need to add a “filter” child element to each of them. For example:
<ns4:filter action="exclude">oval:ssg-state_external_filesystems:ste:1</ns4:filter>
This element is a reference to OVAL state. A single OVAL state should be created for all of them as a child element of ‘states’ element, for example like 
<ns4:file_state id="oval:ssg-state_external_filesystems:ste:1" version="1">
    <ns4:path operation="pattern match">^/mnt/.*</ns4:path>
</ns4:file_state>

Please pay attention to namespaces - using something else than ns4 may be necessary depending on the context.

Dear Matěj Týč,

response from Customer:
//Created By: Koert Gielen  (3/13/2019 4:14 PM)//
Could you please provide me with 1 full example on how to apply filters (for example to exclude directory /dll_linux)?

I am not able (yet) to find the proper documentation on how to apply them. The only thing I found was https://bugzilla.redhat.com/show_bug.cgi?id=1598166 but the contents of this site won't help me any further.

And what do you mean by "please pay attention to namespaces"?
/////////////////////////////////////////////////

Created attachment 1543959 [details]
Full example of usage of filter element in an OVAL definition

Hi,

> Could you please provide me with 1 full example on how to apply filters (for example to exclude directory /dll_linux)?

I have added a full example as attachment 1543959 [details]. In the example, the filter element is used to exclude the /dll_directory from an OVAL object which matches the root filesystem.

> I am not able (yet) to find the proper documentation on how to apply them. 

For documentation about usage of OVAL filters, see the filter section in OVAL language specification: https://oval.mitre.org/language/version5.11/ovaldefinition/documentation/oval-definitions-schema.html#filter

> And what do you mean by "please pay attention to namespaces"?

That means that by the OVAL specification the <filter> element should be in "http://oval.mitre.org/XMLSchema/oval-definitions-5" namespace. Also, the .*_state elements have to be in the same namespace as the corresponding .*_object namespace, for example the <file_state> element has to be in namespace "http://oval.mitre.org/XMLSchema/oval-definitions-5#unix" because <file_object> is in "http://oval.mitre.org/XMLSchema/oval-definitions-5#unix" namespace.
To check if the OVAL is valid, you can run "oscap oval eval <oval_file>".

**Customer's Update Created By: Koert Gielen  (3/20/2019 4:19 PM)**

See the attachment.

And a few example files with wrong filenames:
dll root@ferns openscap# ionice find /dll_linux /DLL_unix -name "Licences_Ismertet*"
/DLL_unix/software/Advicesoft/2015/AS_289_DLL_01_FULL_v2/Advisesoft/Docs/AS_285_ADV/Licences_Ismertet?_285.xls
/DLL_unix/software/Advicesoft/2015/AS_289_DLL_01_FULL_v2/Advisesoft/Docs/AS_286_ADV/Licences_Ismertet?_286.xls
/DLL_unix/software/Advicesoft/2015/AS_289_DLL_01_FULL_v2/Advisesoft/Docs/AS_287_ADV/Licences_Ismertet?_287.xls
/DLL_unix/software/Advicesoft/2015/AS_289_DLL_01_FULL_v2/Advisesoft/Docs/AS_288_ADV/Licences_Ismertet?_288.xls
/DLL_unix/software/Advicesoft/2015/AS_289_DLL_01_FULL_v2/Advisesoft/Docs/AS_289_ADV/Licences_Ismertet?_289.xls
***************************

We have been able to reproduce the issue thanks to the data that we got from the customer.
It seems to manifest on certain type of automount configurations. We are now analysing possible ways of how to fix this issue, and how to test this kind of issues.

Dear Matěj Týč,
Once fix is identified and done, Please do send a oscap test package for customer to test in his environment. (This is requested by Customer)

https://github.com/OpenSCAP/openscap/pull/1315
https://github.com/OpenSCAP/openscap/pull/1329

https://github.com/OpenSCAP/openscap/pull/1315
https://github.com/OpenSCAP/openscap/pull/1329

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2341