Bug 2075041

Summary: Compliance Check Results FAIL even if the kubelet parameter is correct by default
Product: OpenShift Container Platform Reporter: Pamela Escorza <pescorza>
Component: Compliance OperatorAssignee: Vincent Shen <wenshen>
Status: CLOSED ERRATA QA Contact: xiyuan
Severity: high Docs Contact:
Priority: high    
Version: 4.6CC: jhrozek, lbragsta, mrogers, wenshen, xiyuan
Target Milestone: ---   
Target Release: 4.12.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-02 16:00:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pamela Escorza 2022-04-13 13:34:32 UTC 
Description of problem:

Testing the CIS Benchmarks profiles by the operator-compliance, which is aligned with the "Red Hat OpenShift 4 Hardening Guide v1.1", results in FAIL status with the default configuration which is confusing and might affect service if the remediation is applied as need to update kubelet configuration, for example:

    For rule `ocp4-cis-node-modified-master-kubelet-enable-iptables-util-chains`

As per Kubernetes reference (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) it's by default TRUE
--make-iptables-util-chains     Default: true
    If true, kubelet will ensure iptables utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

 But Compliance result is failing because is not specified in /etc/kubernetes/kubelet.conf.


Version-Release number of selected component (if applicable):
OpenShift all versions
OpenShift-Compliance 1.48

How reproducible:
always

Steps to Reproduce:
1. Create the scansettingbinding with CIS Benchmark profiles to verify
2. Check the results


Actual results:
Rules in FAIL status because configuration is not available at kubelet.conf

Expected results:
Get results PASS as the default values are available at API level
$  oc get --raw /api/v1/nodes/${NODE_NAME}/proxy/configz |   jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"|.evictionHard'

Additional info:
Comment 11 Jakub Hrozek 2022-08-22 15:15:04 UTC 
This PR addresses the operator part:
https://github.com/ComplianceAsCode/compliance-operator/pull/78

not marking this as POST just yet as we're still working on the content part.
Comment 14 xiyuan 2022-09-23 14:11:52 UTC 
Verification pass with 4.12.0-0.nightly-2022-09-22-153054 + compliance-operator.v0.1.55
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-09-22-153054   True        False         119m    Cluster version is 4.12.0-0.nightly-2022-09-22-153054
$ oc get ip
NAME            CSV                           APPROVAL    APPROVED
install-fkwmr   compliance-operator.v0.1.55   Automatic   true

1. ########Create TailoredProfile
$ cat tp.yaml 
apiVersion: compliance.openshift.io/v1alpha1
kind: TailoredProfile
metadata:
  name: testprofile
  namespace: openshift-compliance
spec:
  description: test
  title: test
  enableRules:
    - name: ocp4-kubelet-anonymous-auth
      rationale: test
    - name: ocp4-kubelet-authorization-mode
      rationale: test
    - name: ocp4-kubelet-configure-client-ca
      rationale: test
    - name: ocp4-kubelet-configure-event-creation
      rationale: test
    - name: ocp4-kubelet-configure-tls-cipher-suites
      rationale: test
    - name: ocp4-kubelet-enable-cert-rotation
      rationale: test
    - name: ocp4-kubelet-enable-iptables-util-chains
      rationale: test
    - name: ocp4-kubelet-enable-server-cert-rotation
      rationale: test
    - name: ocp4-kubelet-enable-streaming-connections
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-hard-imagefs-available
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-hard-memory-available
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-hard-nodefs-available
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-soft-imagefs-available
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-soft-memory-available
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-soft-nodefs-available
      rationale: test
    - name: ocp4-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree
      rationale: test
  setValues:
    - name: ocp4-var-streaming-connection-timeouts
      value: "5h0m0s"
      rationale: test
    - name: ocp4-var-kubelet-evictionhard-imagefs-available
      value: "15%"
      rationale: test
    - name: ocp4-var-kubelet-evictionhard-imagefs-inodesfree
      value: "5%"
      rationale: test
    - name: ocp4-var-kubelet-evictionhard-memory-available
      value: "300Mi"
      rationale: test
    - name: ocp4-var-kubelet-evictionhard-nodefs-available
      value: "10%"
      rationale: test
    - name: ocp4-var-kubelet-evictionhard-nodefs-inodesfree
      value: "5%"
      rationale: test
    - name: ocp4-var-kubelet-evictionsoft-imagefs-available
      value: "15%"
      rationale: test
    - name: ocp4-var-kubelet-evictionsoft-imagefs-inodesfree
      value: "11%"
      rationale: test
    - name: ocp4-var-kubelet-evictionsoft-memory-available
      value: "600Mi"
      rationale: test
    - name: ocp4-var-kubelet-evictionsoft-nodefs-available
      value: "10%"
      rationale: test
    - name: ocp4-var-kubelet-evictionsoft-nodefs-inodesfree
      value: "5%"
      rationale: test
      setValues:
    - name: ocp4-var-kubelet-tls-cipher-suites-regex
      rationale: Node
      value: '^(TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384|TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384|TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256)$'
    - name: ocp4-var-kubelet-tls-cipher-suites
      rationale: Platform
      value: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
    - name: ocp4-var-event-record-qps
      rationale: test
      value: "10"
$ oc apply -f tp.yaml 
tailoredprofile.compliance.openshift.io/testprofile configured
#######Create ssb:
$ oc apply -f -<<EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: my-ssb-r
profiles:
  - name: testprofile
    kind: TailoredProfile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default-auto-apply
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1
EOF
scansettingbinding.compliance.openshift.io/my-ssb-r created
$ oc get suite -w
NAME       PHASE     RESULT
my-ssb-r   RUNNING   NOT-AVAILABLE
my-ssb-r   AGGREGATING   NOT-AVAILABLE
my-ssb-r   DONE          NON-COMPLIANT
my-ssb-r   DONE          NON-COMPLIANT
$ oc get ccr
NAME                                                                  STATUS   SEVERITY
testprofile-kubelet-anonymous-auth                                    PASS     medium
testprofile-kubelet-authorization-mode                                PASS     medium
testprofile-kubelet-configure-client-ca                               PASS     medium
testprofile-kubelet-configure-event-creation                          FAIL     medium
testprofile-kubelet-configure-tls-cipher-suites                       FAIL     medium
testprofile-kubelet-enable-cert-rotation                              PASS     medium
testprofile-kubelet-enable-iptables-util-chains                       PASS     medium
testprofile-kubelet-enable-server-cert-rotation                       PASS     medium
testprofile-kubelet-enable-streaming-connections                      FAIL     medium
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available    PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree   FAIL     medium
testprofile-kubelet-eviction-thresholds-set-hard-memory-available     FAIL     medium
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree    PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available    FAIL     medium
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree   FAIL     medium
testprofile-kubelet-eviction-thresholds-set-soft-memory-available     FAIL     medium
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available     FAIL     medium
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree    FAIL     medium
to verifiy the result is correct or not, could use below command:
$  for NODE_NAME in $(oc get nodes -ojsonpath='{.items[0].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"'; done
{
  "enableServer": true,
  "staticPodPath": "/etc/kubernetes/manifests",
  "syncFrequency": "1m0s",
  "fileCheckFrequency": "20s",
  "httpCheckFrequency": "20s",
  "address": "0.0.0.0",
  "port": 10250,
  "tlsCipherSuites": [
    "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
    "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
  ],
  "tlsMinVersion": "VersionTLS12",
  "rotateCertificates": true,
  "serverTLSBootstrap": true,
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/kubelet-ca.crt"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "registryPullQPS": 5,
  "registryBurst": 10,
  "eventRecordQPS": 5,
  "eventBurst": 10,
  "enableDebuggingHandlers": true,
  "healthzPort": 10248,
  "healthzBindAddress": "127.0.0.1",
  "oomScoreAdj": -999,
  "clusterDomain": "cluster.local",
  "clusterDNS": [
    "172.30.0.10"
  ],
  "streamingConnectionIdleTimeout": "4h0m0s",
  "nodeStatusUpdateFrequency": "10s",
  "nodeStatusReportFrequency": "5m0s",
  "nodeLeaseDurationSeconds": 40,
  "imageMinimumGCAge": "2m0s",
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "volumeStatsAggPeriod": "1m0s",
  "systemCgroups": "/system.slice",
  "cgroupRoot": "/",
  "cgroupsPerQOS": true,
  "cgroupDriver": "systemd",
  "cpuManagerPolicy": "none",
  "cpuManagerReconcilePeriod": "10s",
  "memoryManagerPolicy": "None",
  "topologyManagerPolicy": "none",
  "topologyManagerScope": "container",
  "runtimeRequestTimeout": "2m0s",
  "hairpinMode": "promiscuous-bridge",
  "maxPods": 250,
  "podPidsLimit": 4096,
  "resolvConf": "/etc/resolv.conf",
  "cpuCFSQuota": true,
  "cpuCFSQuotaPeriod": "100ms",
  "nodeStatusMaxImages": 50,
  "maxOpenFiles": 1000000,
  "contentType": "application/vnd.kubernetes.protobuf",
  "kubeAPIQPS": 50,
  "kubeAPIBurst": 100,
  "serializeImagePulls": false,
  "evictionHard": {
    "imagefs.available": "15%",
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "evictionPressureTransitionPeriod": "5m0s",
  "enableControllerAttachDetach": true,
  "makeIPTablesUtilChains": true,
  "iptablesMasqueradeBit": 14,
  "iptablesDropBit": 15,
  "featureGates": {
    "APIPriorityAndFairness": true,
    "CSIMigrationAzureFile": false,
    "CSIMigrationvSphere": false,
    "DownwardAPIHugePages": true,
    "RotateKubeletServerCertificate": true
  },
  "failSwapOn": true,
  "memorySwap": {},
  "containerLogMaxSize": "50Mi",
  "containerLogMaxFiles": 5,
  "configMapAndSecretChangeDetectionStrategy": "Watch",
  "systemReserved": {
    "cpu": "500m",
    "memory": "1Gi"
  },
  "enforceNodeAllocatable": [
    "pods"
  ],
  "volumePluginDir": "/etc/kubernetes/kubelet-plugins/volume/exec",
  "logging": {
    "format": "text",
    "flushFrequency": 5000000000,
    "verbosity": 2,
    "options": {
      "json": {
        "infoBufferSize": "0"
      }
    }
  },
  "enableSystemLogHandler": true,
  "shutdownGracePeriod": "0s",
  "shutdownGracePeriodCriticalPods": "0s",
  "enableProfilingHandler": true,
  "enableDebugFlagsHandler": true,
  "seccompDefault": false,
  "memoryThrottlingFactor": 0.8,
  "registerWithTaints": [
    {
      "key": "node-role.kubernetes.io/master",
      "effect": "NoSchedule"
    }
  ],
  "registerNode": true,
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1"
}
$ oc get mcp -w
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
...
master   rendered-master-3d42819a0919615536d6049be17b50a3   True      False      False      3              3                   3                     0                      111m
worker   rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2   True      False      False      3              3                   3                     0                      111m
$ oc get kubeletconfig compliance-operator-kubelet-master -o yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  annotations:
    machineconfiguration.openshift.io/mc-name-suffix: ""
  creationTimestamp: "2022-09-23T13:01:59Z"
  finalizers:
  - 99-master-generated-kubelet
  generation: 16
  labels:
    compliance.openshift.io/scan-name: testprofile
    compliance.openshift.io/suite: my-ssb-r
  name: compliance-operator-kubelet-master
  resourceVersion: "82039"
  uid: 996b1692-fe9f-4cda-9121-2b6362e8f92b
spec:
  kubeletConfig:
    eventRecordQPS: 10
    evictionHard:
      imagefs.inodesFree: 5%
      memory.available: 300Mi
    evictionPressureTransitionPeriod: 0s
    evictionSoft:
      imagefs.available: 15%
      imagefs.inodesFree: 11%
      memory.available: 600Mi
      nodefs.available: 10%
      nodefs.inodesFree: 5%
    evictionSoftGracePeriod:
      imagefs.available: 1m30s
      imagefs.inodesFree: 1m30s
      memory.available: 1m30s
      nodefs.available: 1m30s
      nodefs.inodesFree: 1m30s
    streamingConnectionIdleTimeout: 5h0m0s
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""
status:
  conditions:
  - lastTransitionTime: "2022-09-23T13:16:18Z"
    message: Success
    status: "True"
    type: Success
###############Rescan
$ oc compliance rerun-now scansettingbinding my-ssb-r
Rerunning scans from 'my-ssb-r': testprofile
Re-running scan 'openshift-compliance/testprofile'
[xiyuan@MiWiFi-RA69-srv func]$ oc get ssb
$ oc get suite -w
NAME       PHASE     RESULT
my-ssb-r   RUNNING   NOT-AVAILABLE
my-ssb-r   AGGREGATING   NOT-AVAILABLE
my-ssb-r   DONE          NON-COMPLIANT
^C$ oc get cr
NAME                                                                    STATE
testprofile-kubelet-configure-event-creation                            Applied
testprofile-kubelet-configure-event-creation-1                          Applied
testprofile-kubelet-configure-tls-cipher-suites                         Applied
testprofile-kubelet-configure-tls-cipher-suites-1                       Outdated
testprofile-kubelet-enable-streaming-connections                        Applied
testprofile-kubelet-enable-streaming-connections-1                      Applied
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available      Applied
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available-1    Applied
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available-2    Applied
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available-3    Applied
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree     Outdated
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-1   Applied
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-2   Applied
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-3   Applied
testprofile-kubelet-eviction-thresholds-set-hard-memory-available       Applied
testprofile-kubelet-eviction-thresholds-set-hard-memory-available-1     Outdated
testprofile-kubelet-eviction-thresholds-set-hard-memory-available-2     Applied
testprofile-kubelet-eviction-thresholds-set-hard-memory-available-3     Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available       Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available-1     Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available-2     Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available-3     Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree      Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree-1    Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree-2    Applied
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree-3    Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available      Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-1    Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-2    Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-3    Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-4    Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-5    Outdated
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree     Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-1   Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-2   Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-3   Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-4   Applied
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-5   Outdated
testprofile-kubelet-eviction-thresholds-set-soft-memory-available       Applied
testprofile-kubelet-eviction-thresholds-set-soft-memory-available-1     Applied
testprofile-kubelet-eviction-thresholds-set-soft-memory-available-2     Applied
testprofile-kubelet-eviction-thresholds-set-soft-memory-available-3     Applied
testprofile-kubelet-eviction-thresholds-set-soft-memory-available-4     Applied
testprofile-kubelet-eviction-thresholds-set-soft-memory-available-5     Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available       Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-1     Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-2     Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-3     Outdated
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-4     Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-5     Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree      Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-1    Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-2    Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-3    Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-4    Applied
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-5    Applied
$ oc get ccr
NAME                                                                  STATUS   SEVERITY
testprofile-kubelet-anonymous-auth                                    PASS     medium
testprofile-kubelet-authorization-mode                                PASS     medium
testprofile-kubelet-configure-client-ca                               PASS     medium
testprofile-kubelet-configure-event-creation                          PASS     medium
testprofile-kubelet-configure-tls-cipher-suites                       PASS     medium
testprofile-kubelet-enable-cert-rotation                              PASS     medium
testprofile-kubelet-enable-iptables-util-chains                       PASS     medium
testprofile-kubelet-enable-server-cert-rotation                       PASS     medium
testprofile-kubelet-enable-streaming-connections                      PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available    FAIL     medium
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree   PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-memory-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available     FAIL     medium
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree    FAIL     medium
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available    PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree   PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-memory-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree    PASS     medium
$ oc get mcp -w
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-3d42819a0919615536d6049be17b50a3   False     True       False      3              0                   1                     0                      121m
worker   rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2   False     True       False      3              1                   1                     0                      121m
...
master   rendered-master-3d42819a0919615536d6049be17b50a3   False     True       False      3              1                   1                     0                      128m
worker   rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2   False     True       False      3              2                   2                     0                      128m
master   rendered-master-3d42819a0919615536d6049be17b50a3   False     True       False      3              2                   2                     0                      128m
master   rendered-master-3d42819a0919615536d6049be17b50a3   False     True       False      3              2                   2                     0                      128m
worker   rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2   False     True       False      3              2                   3                     0                      132m
worker   rendered-worker-38af4de9d1e7de225c5a1074b31a5bfd   True      False      False      3              3                   3                     0                      132m
master   rendered-master-6afad49b7227c3da329825d45c1d9ebc   True      False      False      3              3                   3                     0                      133m

#############Rescan
$ oc compliance rerun-now scansettingbinding my-ssb-r
Rerunning scans from 'my-ssb-r': testprofile
Re-running scan 'openshift-compliance/testprofile'
$ oc get suite
NAME       PHASE   RESULT
my-ssb-r   DONE    COMPLIANT
$ oc get ccr
NAME                                                                  STATUS   SEVERITY
testprofile-kubelet-anonymous-auth                                    PASS     medium
testprofile-kubelet-authorization-mode                                PASS     medium
testprofile-kubelet-configure-client-ca                               PASS     medium
testprofile-kubelet-configure-event-creation                          PASS     medium
testprofile-kubelet-configure-tls-cipher-suites                       PASS     medium
testprofile-kubelet-enable-cert-rotation                              PASS     medium
testprofile-kubelet-enable-iptables-util-chains                       PASS     medium
testprofile-kubelet-enable-server-cert-rotation                       PASS     medium
testprofile-kubelet-enable-streaming-connections                      PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available    PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree   PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-memory-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree    PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available    PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree   PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-memory-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available     PASS     medium
testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree    PASS     medium
$ for NODE_NAME in $(oc get nodes -ojsonpath='{.items[0].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"'; done
{
  "enableServer": true,
  "staticPodPath": "/etc/kubernetes/manifests",
  "syncFrequency": "1m0s",
  "fileCheckFrequency": "20s",
  "httpCheckFrequency": "20s",
  "address": "0.0.0.0",
  "port": 10250,
  "tlsCipherSuites": [
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  ],
  "tlsMinVersion": "VersionTLS12",
  "rotateCertificates": true,
  "serverTLSBootstrap": true,
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/kubelet-ca.crt"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "registryPullQPS": 5,
  "registryBurst": 10,
  "eventRecordQPS": 10,
  "eventBurst": 10,
  "enableDebuggingHandlers": true,
  "healthzPort": 10248,
  "healthzBindAddress": "127.0.0.1",
  "oomScoreAdj": -999,
  "clusterDomain": "cluster.local",
  "clusterDNS": [
    "172.30.0.10"
  ],
  "streamingConnectionIdleTimeout": "5h0m0s",
  "nodeStatusUpdateFrequency": "10s",
  "nodeStatusReportFrequency": "5m0s",
  "nodeLeaseDurationSeconds": 40,
  "imageMinimumGCAge": "2m0s",
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "volumeStatsAggPeriod": "1m0s",
  "systemCgroups": "/system.slice",
  "cgroupRoot": "/",
  "cgroupsPerQOS": true,
  "cgroupDriver": "systemd",
  "cpuManagerPolicy": "none",
  "cpuManagerReconcilePeriod": "10s",
  "memoryManagerPolicy": "None",
  "topologyManagerPolicy": "none",
  "topologyManagerScope": "container",
  "runtimeRequestTimeout": "2m0s",
  "hairpinMode": "promiscuous-bridge",
  "maxPods": 250,
  "podPidsLimit": 4096,
  "resolvConf": "/etc/resolv.conf",
  "cpuCFSQuota": true,
  "cpuCFSQuotaPeriod": "100ms",
  "nodeStatusMaxImages": 50,
  "maxOpenFiles": 1000000,
  "contentType": "application/vnd.kubernetes.protobuf",
  "kubeAPIQPS": 50,
  "kubeAPIBurst": 100,
  "serializeImagePulls": false,
  "evictionHard": {
    "imagefs.available": "15%",
    "imagefs.inodesFree": "5%",
    "memory.available": "300Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "evictionSoft": {
    "imagefs.available": "15%",
    "imagefs.inodesFree": "11%",
    "memory.available": "600Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "evictionSoftGracePeriod": {
    "imagefs.available": "1m30s",
    "imagefs.inodesFree": "1m30s",
    "memory.available": "1m30s",
    "nodefs.available": "1m30s",
    "nodefs.inodesFree": "1m30s"
  },
  "evictionPressureTransitionPeriod": "5m0s",
  "enableControllerAttachDetach": true,
  "makeIPTablesUtilChains": true,
  "iptablesMasqueradeBit": 14,
  "iptablesDropBit": 15,
  "featureGates": {
    "APIPriorityAndFairness": true,
    "CSIMigrationAzureFile": false,
    "CSIMigrationvSphere": false,
    "DownwardAPIHugePages": true,
    "RotateKubeletServerCertificate": true
  },
  "failSwapOn": true,
  "memorySwap": {},
  "containerLogMaxSize": "50Mi",
  "containerLogMaxFiles": 5,
  "configMapAndSecretChangeDetectionStrategy": "Watch",
  "systemReserved": {
    "cpu": "500m",
    "memory": "1Gi"
  },
  "enforceNodeAllocatable": [
    "pods"
  ],
  "volumePluginDir": "/etc/kubernetes/kubelet-plugins/volume/exec",
  "logging": {
    "format": "text",
    "flushFrequency": 5000000000,
    "verbosity": 2,
    "options": {
      "json": {
        "infoBufferSize": "0"
      }
    }
  },
  "enableSystemLogHandler": true,
  "shutdownGracePeriod": "0s",
  "shutdownGracePeriodCriticalPods": "0s",
  "enableProfilingHandler": true,
  "enableDebugFlagsHandler": true,
  "seccompDefault": false,
  "memoryThrottlingFactor": 0.8,
  "registerWithTaints": [
    {
      "key": "node-role.kubernetes.io/master",
      "effect": "NoSchedule"
    }
  ],
  "registerNode": true,
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1"
}
Comment 16 errata-xmlrpc 2022-11-02 16:00:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6657
Comment 18 Lance Bragstad 2022-11-23 20:41:30 UTC 
I was able to recreate this on 0.1.57:

$ oc get ccr -n openshift-compliance | grep kubelet-eviction-threshold
ocp4-cis-kubelet-eviction-thresholds-set-hard-imagefs-available       FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree      FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-hard-memory-available        FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-hard-nodefs-available        FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree       FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-soft-imagefs-available       FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree      FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-soft-memory-available        FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-soft-nodefs-available        FAIL           medium
ocp4-cis-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree       FAIL           medium


$ for NODE_NAME in $(oc get nodes -ojsonpath='{.items[*].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1" | .evictionHard."memory.available"'; done
"100Mi"
"100Mi"
"100Mi"
"100Mi"
"100Mi"
"100Mi"
Comment 19 Lance Bragstad 2022-11-23 20:53:07 UTC 
Since we use Jira for issues, I opened a new issue tracking this there: https://issues.redhat.com/browse/OCPBUGS-4062
Comment 20 Vincent Shen 2022-11-28 21:16:41 UTC 
The reason why this failed is that the default variable that we use to check .evictionHard."memory.available" is "200Mi" defined using the variable "ocp4-var-kubelet-evictionhard-memory-available" if this parameter already defaults present in the system, the user can choose to use a tailored profile to set variable "ocp4-var-kubelet-evictionhard-memory-available" to match the default variable if that match desired compliance requirement.