Additionally, as a side note: if you look at the report (and also at the report which shows as compliant), you will see lots of "notchecked" tests. Why is that?

I went through our documentation, again.

https://docs.openshift.com/container-platform/4.6/security/compliance_operator/compliance-operator-remediation.html

> "Each ComplianceCheckResult represents a result of one compliance rule check. If the rule can be remediated automatically, a ComplianceRemediation object with the same name, owned by the ComplianceCheckResult is created. Unless requested, the remediations are not applied automatically, which gives an OpenShift Container Platform administrator the opportunity to review what the remediation does and only apply a remediation once it has been verified."

That also means: if a rule cannot be remediated automatically, you will not see a complianceremediation object for this.

The doc further says that some scans cannot be remediated automatically due to permissions issues, among others:

> "The payload can be any Kubernetes object, but because this remediation was produced by a node scan, the remediation payload in the above example is a MachineConfig object. For Platform scans, the remediation payload is often a different kind of an object (for example, a ConfigMap or Secret object), but typically applying that remediation is up to the administrator, because otherwise the Compliance Operator would have required a very broad set of permissions in order to manipulate any generic Kubernetes object. An example of remediating a Platform check is provided later in the text."

You can see check results with:
~~~
oc get ComplianceCheckResult
~~~

And remediations with:
~~~
oc get ComplianceRemediation
~~~

For example, you can list FAILed checks with:
~~~
[akaris@linux ipi-eu-west-1]$ oc get ComplianceCheckResult | grep FAIL
ocp4-cis-api-server-encryption-provider-config                        FAIL     medium
ocp4-cis-audit-log-forwarding-enabled                                 FAIL     medium
ocp4-cis-node-master-kubelet-configure-event-creation                 FAIL     medium
ocp4-cis-node-worker-kubelet-configure-event-creation                 FAIL     medium
~~~

But that does not mean that there is an automatic remediation for each of them:
~~~
[akaris@linux ipi-eu-west-1]$ oc get complianceremediation
NAME                                             STATE
ocp4-cis-api-server-encryption-provider-config   NotApplied
~~~

Rules that come with an automatic remediation will have section `availableFixes`:
~~~
$ oc explain rule.availableFixes
KIND:     Rule
VERSION:  compliance.openshift.io/v1alpha1

DESCRIPTION:
     The Available fixes
~~~

For example:
~~~
$ oc get rules.compliance -o yaml ocp4-api-server-encryption-provider-config
apiVersion: compliance.openshift.io/v1alpha1
availableFixes:
- fixObject:
    apiVersion: config.openshift.io/v1
    kind: APIServer
    metadata:
      name: cluster
    spec:
      encryption:
        type: aescbc
description: '<code>aescbc</code><code>apiserver</code><pre>spec:&#xA;  encryption:&#xA;    type:
  aescbc</pre>object which configures the API server itself.'
id: xccdf_org.ssgproject.content_rule_api_server_encryption_provider_config
kind: Rule
metadata:
  annotations:
    compliance.openshift.io/image-digest: pb-ocp4rcnb2
    compliance.openshift.io/rule: api-server-encryption-provider-config
  creationTimestamp: "2021-01-22T18:17:25Z"
  generation: 1
  labels:
    compliance.openshift.io/profile-bundle: ocp4
  managedFields:
  - apiVersion: compliance.openshift.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:availableFixes: {}
      f:description: {}
      f:id: {}
      f:metadata:
        f:annotations:
          .: {}
          f:compliance.openshift.io/image-digest: {}
          f:compliance.openshift.io/rule: {}
        f:labels:
          .: {}
          f:compliance.openshift.io/profile-bundle: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"22a27fbe-61c0-4795-a338-317b7e87c1ce"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:rationale: {}
      f:severity: {}
      f:title: {}
      f:warning: {}
    manager: compliance-operator
    operation: Update
    time: "2021-01-22T18:17:25Z"
  name: ocp4-api-server-encryption-provider-config
  namespace: openshift-compliance
  ownerReferences:
  - apiVersion: compliance.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ProfileBundle
    name: ocp4
    uid: 22a27fbe-61c0-4795-a338-317b7e87c1ce
  resourceVersion: "155483"
  selfLink: /apis/compliance.openshift.io/v1alpha1/namespaces/openshift-compliance/rules/ocp4-api-server-encryption-provider-config
  uid: bef5b61b-bbee-49e9-9446-62303af07858
rationale: etcd is a highly available key-value store used by OpenShift deployments&#xA;for
  persistent storage of all REST API objects. These objects are&#xA;sensitive in nature
  and should be encrypted at rest to avoid any&#xA;disclosures.
severity: medium
title: Configure the Encryption Provider
warning: <code class="ocp-api-endpoint">/apis/config.openshift.io/v1/apiservers/cluster</code><code
  class="ocp-dump-location"><sub idref="xccdf_org.ssgproject.content_value_ocp_data_root"
  use="legacy" />/apis/config.openshift.io/v1/apiservers/cluster</code>file.
~~~

Rules without automatic remediation do not have that definition and must be fixed manually:
~~~
$ oc get compliancecheckresult ocp4-cis-node-worker-kubelet-configure-event-creation -o yaml | grep -i compliance.openshift.io/rule
    compliance.openshift.io/rule: kubelet-configure-event-creation
          f:compliance.openshift.io/rule: {}
$ oc get rules.compliance -o yaml ocp4-kubelet-configure-event-creation
apiVersion: compliance.openshift.io/v1alpha1
description: '<code>eventRecordQPS</code><code>KubeletConfig</code><pre><sub idref="xccdf_org.ssgproject.content_value_var_event_record_qps"
  use="legacy" /></pre>option along these lines:'
id: xccdf_org.ssgproject.content_rule_kubelet_configure_event_creation
kind: Rule
metadata:
  annotations:
    compliance.openshift.io/image-digest: pb-ocp4rcnb2
    compliance.openshift.io/rule: kubelet-configure-event-creation
  creationTimestamp: "2021-01-22T18:17:22Z"
  generation: 1
  labels:
    compliance.openshift.io/profile-bundle: ocp4
  managedFields:
  - apiVersion: compliance.openshift.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:description: {}
      f:id: {}
      f:metadata:
        f:annotations:
          .: {}
          f:compliance.openshift.io/image-digest: {}
          f:compliance.openshift.io/rule: {}
        f:labels:
          .: {}
          f:compliance.openshift.io/profile-bundle: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"22a27fbe-61c0-4795-a338-317b7e87c1ce"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:rationale: {}
      f:severity: {}
      f:title: {}
      f:warning: {}
    manager: compliance-operator
    operation: Update
    time: "2021-01-22T18:17:22Z"
  name: ocp4-kubelet-configure-event-creation
  namespace: openshift-compliance
  ownerReferences:
  - apiVersion: compliance.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ProfileBundle
    name: ocp4
    uid: 22a27fbe-61c0-4795-a338-317b7e87c1ce
  resourceVersion: "155387"
  selfLink: /apis/compliance.openshift.io/v1alpha1/namespaces/openshift-compliance/rules/ocp4-kubelet-configure-event-creation
  uid: 259b4be8-ac7f-4eb6-8d31-8a8a9f2adbc9
rationale: It is important to capture all events and not restrict event creation.&#xA;Events
  are an important source of security information and analytics that&#xA;ensure that
  your environment is consistently monitored using the event&#xA;data.
severity: medium
title: Kubelet - Ensure Event Creation Is Configured
warning: <code>KubeletConfig</code><code>KubeletConfig</code>object.
~~~

I would hence like to rephrase the request here:

a) How can admins know why some tests show as SKIP?
b) Are there any plans to make the absence of remediations more obvious / to provide more explicit manual remediation 
c) Looking at specific rules, I cannot see the entire instructiosn from the .yaml, for example:

~~~
[akaris@linux ipi-eu-west-1]$ oc get rule -o yaml ocp4-kubelet-configure-event-creation
apiVersion: compliance.openshift.io/v1alpha1
description: '<code>eventRecordQPS</code><code>KubeletConfig</code><pre><sub idref="xccdf_org.ssgproject.content_value_var_event_record_qps"
  use="legacy" /></pre>option along these lines:'
id: xccdf_org.ssgproject.content_rule_kubelet_configure_event_creation
kind: Rule
metadata:
  annotations:
    compliance.openshift.io/image-digest: pb-ocp4rcnb2
    compliance.openshift.io/rule: kubelet-configure-event-creation
  creationTimestamp: "2021-01-22T18:17:22Z"
  generation: 1
  labels:
    compliance.openshift.io/profile-bundle: ocp4
  managedFields:
  - apiVersion: compliance.openshift.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:description: {}
      f:id: {}
      f:metadata:
        f:annotations:
          .: {}
          f:compliance.openshift.io/image-digest: {}
          f:compliance.openshift.io/rule: {}
        f:labels:
          .: {}
          f:compliance.openshift.io/profile-bundle: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"22a27fbe-61c0-4795-a338-317b7e87c1ce"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:rationale: {}
      f:severity: {}
      f:title: {}
      f:warning: {}
    manager: compliance-operator
    operation: Update
    time: "2021-01-22T18:17:22Z"
  name: ocp4-kubelet-configure-event-creation
  namespace: openshift-compliance
  ownerReferences:
  - apiVersion: compliance.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ProfileBundle
    name: ocp4
    uid: 22a27fbe-61c0-4795-a338-317b7e87c1ce
  resourceVersion: "155387"
  selfLink: /apis/compliance.openshift.io/v1alpha1/namespaces/openshift-compliance/rules/ocp4-kubelet-configure-event-creation
  uid: 259b4be8-ac7f-4eb6-8d31-8a8a9f2adbc9
rationale: It is important to capture all events and not restrict event creation.&#xA;Events
  are an important source of security information and analytics that&#xA;ensure that
  your environment is consistently monitored using the event&#xA;data.
severity: medium
title: Kubelet - Ensure Event Creation Is Configured
warning: <code>KubeletConfig</code><code>KubeletConfig</code>object.
~~~

In order to see the full remediation steps, do I have to generate an ARF report and then convert it into HTML? I'm looking for an object  that contains detailed manual remediation steps, in this case the ARF report has:
~~~
Security relevant information should be captured. The eventRecordQPS Kubelet option can be used to limit the rate at which events are gathered. Setting this too low could result in relevant events not being logged, however the unlimited setting of 0 could result in a denial of service on the kubelet. Processing and storage systems should be scaled to handle the expected event load. To set the eventRecordQPS option for the kubelet, create a KubeletConfig option along these lines:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
   name: kubelet-config-$pool
spec:
    machineConfigPoolSelector:
        matchLabels:
            pools.operator.machineconfiguration.openshift.io/$pool_name: ""
    kubeletConfig:
        eventRecordQPS: 5
~~~

And this information should be shown to the administrator via the API. For example, there could be a CRD ManualComplianceRemediation that contains manual instructions for administrators, etc. ... do we already have something like that?

(In reply to Andreas Karis from comment #9)
> I would hence like to rephrase the request here:
> 
> a) How can admins know why some tests show as SKIP?

I'm afraid this is currently not very user-friendly. What you can do it to run the scan with debug: true and then look at the oc logs of the scanner container of the pod running the actual scan. If the rule is not applicable for the current node, you would see something like:

I: oscap: Evaluating definition 'oval:ssg-node_is_ocp4_master_node:def:1': Node is Red Hat OpenShift Container Platform 4 Master Node.
I: oscap:   Evaluating file test 'oval:ssg-test_kube_api_pod_exists:tst:1': Testing if /etc/kubernetes/static-pod-resources/kube-apiserver-certs exists.
I: oscap:     Querying file object 'oval:ssg-object_kube_api_pod_exists:obj:1', flags: 0.
I: oscap:     Creating new syschar for file_object 'oval:ssg-object_kube_api_pod_exists:obj:1'.
I: oscap:     Switching probe to PROBE_OFFLINE_OWN mode.
I: oscap:     I will run file_probe_main:
I: oscap:     Opening file '/host/etc/kubernetes/static-pod-resources/kube-apiserver-certs'.
I: oscap:     Test 'oval:ssg-test_kube_api_pod_exists:tst:1' requires that every object defined by 'oval:ssg-object_kube_api_pod_exists:obj:1' exists on the system.
I: oscap:     0 objects defined by 'oval:ssg-object_kube_api_pod_exists:obj:1' exist on the system.
I: oscap:     Test 'oval:ssg-test_kube_api_pod_exists:tst:1' does not contain any state to compare object with.
I: oscap:     No item matching object 'oval:ssg-object_kube_api_pod_exists:obj:1' was found on the system. (flag=does not exist)
I: oscap:   Test 'oval:ssg-test_kube_api_pod_exists:tst:1' evaluated as false.
I: oscap: Definition 'oval:ssg-node_is_ocp4_master_node:def:1' evaluated as false.
I: oscap: Rule 'xccdf_org.ssgproject.content_rule_etcd_unique_ca' is not applicable.
Result^M        notapplicable

The above says that openscap tried to match the ssg-node_is_ocp4_master_node that didn't match (Definition 'oval:ssg-node_is_ocp4_master_node:def:1' evaluated as false.) and therefore the rule was skipped.

> b) Are there any plans to make the absence of remediations more obvious / to
> provide more explicit manual remediation 

Yes. I hope that after our Jira was opened recently you can access https://issues.redhat.com/browse/CMP-767 At the moment the issue is not assigned to a sprint, but we're going to have a planning/grooming session soon so we'll bring the issue up.
This would provide the manual remediation. To make the presence or absence of a remediation more obvious, would it help to add a label or annotation so that these could be filtered? At the moment an automatic remediation always has the same name as a compliancecheckresult, so you could look for whether a complianceremediation object exists or not for a compliancecheckresult.

> c) Looking at specific rules, I cannot see the entire instructiosn from the
> .yaml, for example:

This would this would be also solved with surfacing the manual checks as said above.

> 
> ~~~
> [akaris@linux ipi-eu-west-1]$ oc get rule -o yaml
> ocp4-kubelet-configure-event-creation
> apiVersion: compliance.openshift.io/v1alpha1
> description: '<code>eventRecordQPS</code><code>KubeletConfig</code><pre><sub
> idref="xccdf_org.ssgproject.content_value_var_event_record_qps"
>   use="legacy" /></pre>option along these lines:'
> id: xccdf_org.ssgproject.content_rule_kubelet_configure_event_creation
> kind: Rule
> metadata:
>   annotations:
>     compliance.openshift.io/image-digest: pb-ocp4rcnb2
>     compliance.openshift.io/rule: kubelet-configure-event-creation
>   creationTimestamp: "2021-01-22T18:17:22Z"
>   generation: 1
>   labels:
>     compliance.openshift.io/profile-bundle: ocp4
>   managedFields:
>   - apiVersion: compliance.openshift.io/v1alpha1
>     fieldsType: FieldsV1
>     fieldsV1:
>       f:description: {}
>       f:id: {}
>       f:metadata:
>         f:annotations:
>           .: {}
>           f:compliance.openshift.io/image-digest: {}
>           f:compliance.openshift.io/rule: {}
>         f:labels:
>           .: {}
>           f:compliance.openshift.io/profile-bundle: {}
>         f:ownerReferences:
>           .: {}
>           k:{"uid":"22a27fbe-61c0-4795-a338-317b7e87c1ce"}:
>             .: {}
>             f:apiVersion: {}
>             f:blockOwnerDeletion: {}
>             f:controller: {}
>             f:kind: {}
>             f:name: {}
>             f:uid: {}
>       f:rationale: {}
>       f:severity: {}
>       f:title: {}
>       f:warning: {}
>     manager: compliance-operator
>     operation: Update
>     time: "2021-01-22T18:17:22Z"
>   name: ocp4-kubelet-configure-event-creation
>   namespace: openshift-compliance
>   ownerReferences:
>   - apiVersion: compliance.openshift.io/v1alpha1
>     blockOwnerDeletion: true
>     controller: true
>     kind: ProfileBundle
>     name: ocp4
>     uid: 22a27fbe-61c0-4795-a338-317b7e87c1ce
>   resourceVersion: "155387"
>   selfLink:
> /apis/compliance.openshift.io/v1alpha1/namespaces/openshift-compliance/rules/
> ocp4-kubelet-configure-event-creation
>   uid: 259b4be8-ac7f-4eb6-8d31-8a8a9f2adbc9
> rationale: It is important to capture all events and not restrict event
> creation.&#xA;Events
>   are an important source of security information and analytics
> that&#xA;ensure that
>   your environment is consistently monitored using the event&#xA;data.
> severity: medium
> title: Kubelet - Ensure Event Creation Is Configured
> warning: <code>KubeletConfig</code><code>KubeletConfig</code>object.
> ~~~
> 
> In order to see the full remediation steps, do I have to generate an ARF
> report and then convert it into HTML? I'm looking for an object  that
> contains detailed manual remediation steps, in this case the ARF report has:
> ~~~
> Security relevant information should be captured. The eventRecordQPS Kubelet
> option can be used to limit the rate at which events are gathered. Setting
> this too low could result in relevant events not being logged, however the
> unlimited setting of 0 could result in a denial of service on the kubelet.
> Processing and storage systems should be scaled to handle the expected event
> load. To set the eventRecordQPS option for the kubelet, create a
> KubeletConfig option along these lines:
> 
> apiVersion: machineconfiguration.openshift.io/v1
> kind: KubeletConfig
> metadata:
>    name: kubelet-config-$pool
> spec:
>     machineConfigPoolSelector:
>         matchLabels:
>             pools.operator.machineconfiguration.openshift.io/$pool_name: ""
>     kubeletConfig:
>         eventRecordQPS: 5
> ~~~
> 
> And this information should be shown to the administrator via the API. For
> example, there could be a CRD ManualComplianceRemediation that contains
> manual instructions for administrators, etc. ... do we already have
> something like that?

(In reply to Jakub Hrozek from comment #10)
> (In reply to Andreas Karis from comment #9)
> > I would hence like to rephrase the request here:
> > 
> > a) How can admins know why some tests show as SKIP?
> 
> I'm afraid this is currently not very user-friendly. What you can do it to
> run the scan with debug: true and then look at the oc logs of the scanner
> container of the pod running the actual scan. If the rule is not applicable
> for the current node, you would see something like:
> 
> I: oscap: Evaluating definition 'oval:ssg-node_is_ocp4_master_node:def:1':
> Node is Red Hat OpenShift Container Platform 4 Master Node.
> I: oscap:   Evaluating file test 'oval:ssg-test_kube_api_pod_exists:tst:1':
> Testing if /etc/kubernetes/static-pod-resources/kube-apiserver-certs exists.
> I: oscap:     Querying file object
> 'oval:ssg-object_kube_api_pod_exists:obj:1', flags: 0.
> I: oscap:     Creating new syschar for file_object
> 'oval:ssg-object_kube_api_pod_exists:obj:1'.
> I: oscap:     Switching probe to PROBE_OFFLINE_OWN mode.
> I: oscap:     I will run file_probe_main:
> I: oscap:     Opening file
> '/host/etc/kubernetes/static-pod-resources/kube-apiserver-certs'.
> I: oscap:     Test 'oval:ssg-test_kube_api_pod_exists:tst:1' requires that
> every object defined by 'oval:ssg-object_kube_api_pod_exists:obj:1' exists
> on the system.
> I: oscap:     0 objects defined by
> 'oval:ssg-object_kube_api_pod_exists:obj:1' exist on the system.
> I: oscap:     Test 'oval:ssg-test_kube_api_pod_exists:tst:1' does not
> contain any state to compare object with.
> I: oscap:     No item matching object
> 'oval:ssg-object_kube_api_pod_exists:obj:1' was found on the system.
> (flag=does not exist)
> I: oscap:   Test 'oval:ssg-test_kube_api_pod_exists:tst:1' evaluated as
> false.
> I: oscap: Definition 'oval:ssg-node_is_ocp4_master_node:def:1' evaluated as
> false.
> I: oscap: Rule 'xccdf_org.ssgproject.content_rule_etcd_unique_ca' is not
> applicable.
> Result^M        notapplicable
> 
> The above says that openscap tried to match the ssg-node_is_ocp4_master_node
> that didn't match (Definition 'oval:ssg-node_is_ocp4_master_node:def:1'
> evaluated as false.) and therefore the rule was skipped.

As you can see from the raw openscap output, openscap returns 'not applicable', but we map this result to 'SKIP'. This is apparently confusing, would it help at least a bit to expose not applicable as well in the user-visible status? If yes, would you mind filing a separate BZ so that we don't conflate the two issues together?

Thanks for the answers, I forwarded them to the customer via the case. Here's the new Bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1920577

Andreas, before we merge more patches to an OCP release, I wanted to check with you if these enhancements would be helpful for what the customer is looking for.

So far we merged upstream:
- a new ComplianceCheck status MANUAL that signifies that this rule cannot be evaluated automatically. For example, CIS contains a handful of rules that say something like "Make sure that your pods don't have more privileges than needed". Previously, these would have had the status of INFO
- a new ComplianceCheck attribute called "instructions" that tells the administrator how to evaluate a rule manually. The contents of the field might look like this:

  id: xccdf_org.ssgproject.content_rule_secrets_no_environment_variables
  instructions: |-
    To find workloads that use environment variables for secrets, run the following:
    $ oc get all -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.namespace} {.metadata.name} {"\n"}{end}' -A
    Review the output and ensure that workloads that can mount secrets as data
    volumes use that instead of environment variables.

Together these should cover the rules where OpenScap/CO can't give you the results on its own, especially together with the fact that the state is a label and can be queried, then you can do something like "give me all the rule names and instructions of rules I need to check manually" with:
$ oc get compliancecheckresult -lcompliance.openshift.io/check-status=MANUAL -nopenshift-compliance -ojson | jq -r '.items[] | {name: .metadata.name, instructions: .instructions}'

Next, we looked at how to present which rules have automated remediations, again with a label. The idea is that you can filter rules that failed and can be remediated automatically:
$ oc get compliancecheckresults -l 'compliance.openshift.io/check-status in (FAIL),compliance.openshift.io/automated-remediation'
and those that failed and need to be looked at manually:
$ oc get compliancecheckresults -l 'compliance.openshift.io/check-status in (FAIL),!compliance.openshift.io/automated-remediation'

Do you think this would help the customer? There is also the issue of the confusing SKIP attribute that had been filed separately.

All three small issues tracked by this BZ were addressed upstream:
 - https://github.com/openshift/compliance-operator/pull/572
 - https://github.com/openshift/compliance-operator/pull/568
 - https://github.com/openshift/compliance-operator/pull/567
so I'm moving the bug to MODIFIED.

For instructions missing issue in https://bugzilla.redhat.com/show_bug.cgi?id=1919367#c17, bug https://bugzilla.redhat.com/show_bug.cgi?id=1936413 created to track.
Due to comment, https://bugzilla.redhat.com/show_bug.cgi?id=1919367#c19, move BZ status to Verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Compliance Operator version 0.1.35 for OpenShift Container Platform 4.6-4.8), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2652