Hide Forgot
Description of problem: Testing the CIS Benchmarks profiles by the operator-compliance, which is aligned with the "Red Hat OpenShift 4 Hardening Guide v1.1", results in FAIL status with the default configuration which is confusing and might affect service if the remediation is applied as need to update kubelet configuration, for example: For rule `ocp4-cis-node-modified-master-kubelet-enable-iptables-util-chains` As per Kubernetes reference (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) it's by default TRUE --make-iptables-util-chains Default: true If true, kubelet will ensure iptables utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) But Compliance result is failing because is not specified in /etc/kubernetes/kubelet.conf. Version-Release number of selected component (if applicable): OpenShift all versions OpenShift-Compliance 1.48 How reproducible: always Steps to Reproduce: 1. Create the scansettingbinding with CIS Benchmark profiles to verify 2. Check the results Actual results: Rules in FAIL status because configuration is not available at kubelet.conf Expected results: Get results PASS as the default values are available at API level $ oc get --raw /api/v1/nodes/${NODE_NAME}/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"|.evictionHard' Additional info:
This PR addresses the operator part: https://github.com/ComplianceAsCode/compliance-operator/pull/78 not marking this as POST just yet as we're still working on the content part.
Verification pass with 4.12.0-0.nightly-2022-09-22-153054 + compliance-operator.v0.1.55 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0-0.nightly-2022-09-22-153054 True False 119m Cluster version is 4.12.0-0.nightly-2022-09-22-153054 $ oc get ip NAME CSV APPROVAL APPROVED install-fkwmr compliance-operator.v0.1.55 Automatic true 1. ########Create TailoredProfile $ cat tp.yaml apiVersion: compliance.openshift.io/v1alpha1 kind: TailoredProfile metadata: name: testprofile namespace: openshift-compliance spec: description: test title: test enableRules: - name: ocp4-kubelet-anonymous-auth rationale: test - name: ocp4-kubelet-authorization-mode rationale: test - name: ocp4-kubelet-configure-client-ca rationale: test - name: ocp4-kubelet-configure-event-creation rationale: test - name: ocp4-kubelet-configure-tls-cipher-suites rationale: test - name: ocp4-kubelet-enable-cert-rotation rationale: test - name: ocp4-kubelet-enable-iptables-util-chains rationale: test - name: ocp4-kubelet-enable-server-cert-rotation rationale: test - name: ocp4-kubelet-enable-streaming-connections rationale: test - name: ocp4-kubelet-eviction-thresholds-set-hard-imagefs-available rationale: test - name: ocp4-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree rationale: test - name: ocp4-kubelet-eviction-thresholds-set-hard-memory-available rationale: test - name: ocp4-kubelet-eviction-thresholds-set-hard-nodefs-available rationale: test - name: ocp4-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree rationale: test - name: ocp4-kubelet-eviction-thresholds-set-soft-imagefs-available rationale: test - name: ocp4-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree rationale: test - name: ocp4-kubelet-eviction-thresholds-set-soft-memory-available rationale: test - name: ocp4-kubelet-eviction-thresholds-set-soft-nodefs-available rationale: test - name: ocp4-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree rationale: test setValues: - name: ocp4-var-streaming-connection-timeouts value: "5h0m0s" rationale: test - name: ocp4-var-kubelet-evictionhard-imagefs-available value: "15%" rationale: test - name: ocp4-var-kubelet-evictionhard-imagefs-inodesfree value: "5%" rationale: test - name: ocp4-var-kubelet-evictionhard-memory-available value: "300Mi" rationale: test - name: ocp4-var-kubelet-evictionhard-nodefs-available value: "10%" rationale: test - name: ocp4-var-kubelet-evictionhard-nodefs-inodesfree value: "5%" rationale: test - name: ocp4-var-kubelet-evictionsoft-imagefs-available value: "15%" rationale: test - name: ocp4-var-kubelet-evictionsoft-imagefs-inodesfree value: "11%" rationale: test - name: ocp4-var-kubelet-evictionsoft-memory-available value: "600Mi" rationale: test - name: ocp4-var-kubelet-evictionsoft-nodefs-available value: "10%" rationale: test - name: ocp4-var-kubelet-evictionsoft-nodefs-inodesfree value: "5%" rationale: test setValues: - name: ocp4-var-kubelet-tls-cipher-suites-regex rationale: Node value: '^(TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384|TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384|TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256)$' - name: ocp4-var-kubelet-tls-cipher-suites rationale: Platform value: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" - name: ocp4-var-event-record-qps rationale: test value: "10" $ oc apply -f tp.yaml tailoredprofile.compliance.openshift.io/testprofile configured #######Create ssb: $ oc apply -f -<<EOF apiVersion: compliance.openshift.io/v1alpha1 kind: ScanSettingBinding metadata: name: my-ssb-r profiles: - name: testprofile kind: TailoredProfile apiGroup: compliance.openshift.io/v1alpha1 settingsRef: name: default-auto-apply kind: ScanSetting apiGroup: compliance.openshift.io/v1alpha1 EOF scansettingbinding.compliance.openshift.io/my-ssb-r created $ oc get suite -w NAME PHASE RESULT my-ssb-r RUNNING NOT-AVAILABLE my-ssb-r AGGREGATING NOT-AVAILABLE my-ssb-r DONE NON-COMPLIANT my-ssb-r DONE NON-COMPLIANT $ oc get ccr NAME STATUS SEVERITY testprofile-kubelet-anonymous-auth PASS medium testprofile-kubelet-authorization-mode PASS medium testprofile-kubelet-configure-client-ca PASS medium testprofile-kubelet-configure-event-creation FAIL medium testprofile-kubelet-configure-tls-cipher-suites FAIL medium testprofile-kubelet-enable-cert-rotation PASS medium testprofile-kubelet-enable-iptables-util-chains PASS medium testprofile-kubelet-enable-server-cert-rotation PASS medium testprofile-kubelet-enable-streaming-connections FAIL medium testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree FAIL medium testprofile-kubelet-eviction-thresholds-set-hard-memory-available FAIL medium testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree PASS medium testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available FAIL medium testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree FAIL medium testprofile-kubelet-eviction-thresholds-set-soft-memory-available FAIL medium testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available FAIL medium testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree FAIL medium to verifiy the result is correct or not, could use below command: $ for NODE_NAME in $(oc get nodes -ojsonpath='{.items[0].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"'; done { "enableServer": true, "staticPodPath": "/etc/kubernetes/manifests", "syncFrequency": "1m0s", "fileCheckFrequency": "20s", "httpCheckFrequency": "20s", "address": "0.0.0.0", "port": 10250, "tlsCipherSuites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256" ], "tlsMinVersion": "VersionTLS12", "rotateCertificates": true, "serverTLSBootstrap": true, "authentication": { "x509": { "clientCAFile": "/etc/kubernetes/kubelet-ca.crt" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "registryPullQPS": 5, "registryBurst": 10, "eventRecordQPS": 5, "eventBurst": 10, "enableDebuggingHandlers": true, "healthzPort": 10248, "healthzBindAddress": "127.0.0.1", "oomScoreAdj": -999, "clusterDomain": "cluster.local", "clusterDNS": [ "172.30.0.10" ], "streamingConnectionIdleTimeout": "4h0m0s", "nodeStatusUpdateFrequency": "10s", "nodeStatusReportFrequency": "5m0s", "nodeLeaseDurationSeconds": 40, "imageMinimumGCAge": "2m0s", "imageGCHighThresholdPercent": 85, "imageGCLowThresholdPercent": 80, "volumeStatsAggPeriod": "1m0s", "systemCgroups": "/system.slice", "cgroupRoot": "/", "cgroupsPerQOS": true, "cgroupDriver": "systemd", "cpuManagerPolicy": "none", "cpuManagerReconcilePeriod": "10s", "memoryManagerPolicy": "None", "topologyManagerPolicy": "none", "topologyManagerScope": "container", "runtimeRequestTimeout": "2m0s", "hairpinMode": "promiscuous-bridge", "maxPods": 250, "podPidsLimit": 4096, "resolvConf": "/etc/resolv.conf", "cpuCFSQuota": true, "cpuCFSQuotaPeriod": "100ms", "nodeStatusMaxImages": 50, "maxOpenFiles": 1000000, "contentType": "application/vnd.kubernetes.protobuf", "kubeAPIQPS": 50, "kubeAPIBurst": 100, "serializeImagePulls": false, "evictionHard": { "imagefs.available": "15%", "memory.available": "100Mi", "nodefs.available": "10%", "nodefs.inodesFree": "5%" }, "evictionPressureTransitionPeriod": "5m0s", "enableControllerAttachDetach": true, "makeIPTablesUtilChains": true, "iptablesMasqueradeBit": 14, "iptablesDropBit": 15, "featureGates": { "APIPriorityAndFairness": true, "CSIMigrationAzureFile": false, "CSIMigrationvSphere": false, "DownwardAPIHugePages": true, "RotateKubeletServerCertificate": true }, "failSwapOn": true, "memorySwap": {}, "containerLogMaxSize": "50Mi", "containerLogMaxFiles": 5, "configMapAndSecretChangeDetectionStrategy": "Watch", "systemReserved": { "cpu": "500m", "memory": "1Gi" }, "enforceNodeAllocatable": [ "pods" ], "volumePluginDir": "/etc/kubernetes/kubelet-plugins/volume/exec", "logging": { "format": "text", "flushFrequency": 5000000000, "verbosity": 2, "options": { "json": { "infoBufferSize": "0" } } }, "enableSystemLogHandler": true, "shutdownGracePeriod": "0s", "shutdownGracePeriodCriticalPods": "0s", "enableProfilingHandler": true, "enableDebugFlagsHandler": true, "seccompDefault": false, "memoryThrottlingFactor": 0.8, "registerWithTaints": [ { "key": "node-role.kubernetes.io/master", "effect": "NoSchedule" } ], "registerNode": true, "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1" } $ oc get mcp -w NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE ... master rendered-master-3d42819a0919615536d6049be17b50a3 True False False 3 3 3 0 111m worker rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2 True False False 3 3 3 0 111m $ oc get kubeletconfig compliance-operator-kubelet-master -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: annotations: machineconfiguration.openshift.io/mc-name-suffix: "" creationTimestamp: "2022-09-23T13:01:59Z" finalizers: - 99-master-generated-kubelet generation: 16 labels: compliance.openshift.io/scan-name: testprofile compliance.openshift.io/suite: my-ssb-r name: compliance-operator-kubelet-master resourceVersion: "82039" uid: 996b1692-fe9f-4cda-9121-2b6362e8f92b spec: kubeletConfig: eventRecordQPS: 10 evictionHard: imagefs.inodesFree: 5% memory.available: 300Mi evictionPressureTransitionPeriod: 0s evictionSoft: imagefs.available: 15% imagefs.inodesFree: 11% memory.available: 600Mi nodefs.available: 10% nodefs.inodesFree: 5% evictionSoftGracePeriod: imagefs.available: 1m30s imagefs.inodesFree: 1m30s memory.available: 1m30s nodefs.available: 1m30s nodefs.inodesFree: 1m30s streamingConnectionIdleTimeout: 5h0m0s tlsCipherSuites: - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 machineConfigPoolSelector: matchLabels: pools.operator.machineconfiguration.openshift.io/master: "" status: conditions: - lastTransitionTime: "2022-09-23T13:16:18Z" message: Success status: "True" type: Success ###############Rescan $ oc compliance rerun-now scansettingbinding my-ssb-r Rerunning scans from 'my-ssb-r': testprofile Re-running scan 'openshift-compliance/testprofile' [xiyuan@MiWiFi-RA69-srv func]$ oc get ssb $ oc get suite -w NAME PHASE RESULT my-ssb-r RUNNING NOT-AVAILABLE my-ssb-r AGGREGATING NOT-AVAILABLE my-ssb-r DONE NON-COMPLIANT ^C$ oc get cr NAME STATE testprofile-kubelet-configure-event-creation Applied testprofile-kubelet-configure-event-creation-1 Applied testprofile-kubelet-configure-tls-cipher-suites Applied testprofile-kubelet-configure-tls-cipher-suites-1 Outdated testprofile-kubelet-enable-streaming-connections Applied testprofile-kubelet-enable-streaming-connections-1 Applied testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available Applied testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available-1 Applied testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available-2 Applied testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available-3 Applied testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree Outdated testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-1 Applied testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-2 Applied testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-3 Applied testprofile-kubelet-eviction-thresholds-set-hard-memory-available Applied testprofile-kubelet-eviction-thresholds-set-hard-memory-available-1 Outdated testprofile-kubelet-eviction-thresholds-set-hard-memory-available-2 Applied testprofile-kubelet-eviction-thresholds-set-hard-memory-available-3 Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available-1 Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available-2 Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available-3 Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree-1 Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree-2 Applied testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree-3 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-1 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-2 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-3 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-4 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available-5 Outdated testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-1 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-2 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-3 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-4 Applied testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-5 Outdated testprofile-kubelet-eviction-thresholds-set-soft-memory-available Applied testprofile-kubelet-eviction-thresholds-set-soft-memory-available-1 Applied testprofile-kubelet-eviction-thresholds-set-soft-memory-available-2 Applied testprofile-kubelet-eviction-thresholds-set-soft-memory-available-3 Applied testprofile-kubelet-eviction-thresholds-set-soft-memory-available-4 Applied testprofile-kubelet-eviction-thresholds-set-soft-memory-available-5 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-1 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-2 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-3 Outdated testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-4 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available-5 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-1 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-2 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-3 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-4 Applied testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-5 Applied $ oc get ccr NAME STATUS SEVERITY testprofile-kubelet-anonymous-auth PASS medium testprofile-kubelet-authorization-mode PASS medium testprofile-kubelet-configure-client-ca PASS medium testprofile-kubelet-configure-event-creation PASS medium testprofile-kubelet-configure-tls-cipher-suites PASS medium testprofile-kubelet-enable-cert-rotation PASS medium testprofile-kubelet-enable-iptables-util-chains PASS medium testprofile-kubelet-enable-server-cert-rotation PASS medium testprofile-kubelet-enable-streaming-connections PASS medium testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available FAIL medium testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree PASS medium testprofile-kubelet-eviction-thresholds-set-hard-memory-available PASS medium testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available FAIL medium testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree FAIL medium testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree PASS medium testprofile-kubelet-eviction-thresholds-set-soft-memory-available PASS medium testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree PASS medium $ oc get mcp -w NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-3d42819a0919615536d6049be17b50a3 False True False 3 0 1 0 121m worker rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2 False True False 3 1 1 0 121m ... master rendered-master-3d42819a0919615536d6049be17b50a3 False True False 3 1 1 0 128m worker rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2 False True False 3 2 2 0 128m master rendered-master-3d42819a0919615536d6049be17b50a3 False True False 3 2 2 0 128m master rendered-master-3d42819a0919615536d6049be17b50a3 False True False 3 2 2 0 128m worker rendered-worker-5f14fd782ad6ad88f31a81be1f6fbbe2 False True False 3 2 3 0 132m worker rendered-worker-38af4de9d1e7de225c5a1074b31a5bfd True False False 3 3 3 0 132m master rendered-master-6afad49b7227c3da329825d45c1d9ebc True False False 3 3 3 0 133m #############Rescan $ oc compliance rerun-now scansettingbinding my-ssb-r Rerunning scans from 'my-ssb-r': testprofile Re-running scan 'openshift-compliance/testprofile' $ oc get suite NAME PHASE RESULT my-ssb-r DONE COMPLIANT $ oc get ccr NAME STATUS SEVERITY testprofile-kubelet-anonymous-auth PASS medium testprofile-kubelet-authorization-mode PASS medium testprofile-kubelet-configure-client-ca PASS medium testprofile-kubelet-configure-event-creation PASS medium testprofile-kubelet-configure-tls-cipher-suites PASS medium testprofile-kubelet-enable-cert-rotation PASS medium testprofile-kubelet-enable-iptables-util-chains PASS medium testprofile-kubelet-enable-server-cert-rotation PASS medium testprofile-kubelet-enable-streaming-connections PASS medium testprofile-kubelet-eviction-thresholds-set-hard-imagefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree PASS medium testprofile-kubelet-eviction-thresholds-set-hard-memory-available PASS medium testprofile-kubelet-eviction-thresholds-set-hard-nodefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree PASS medium testprofile-kubelet-eviction-thresholds-set-soft-imagefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree PASS medium testprofile-kubelet-eviction-thresholds-set-soft-memory-available PASS medium testprofile-kubelet-eviction-thresholds-set-soft-nodefs-available PASS medium testprofile-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree PASS medium $ for NODE_NAME in $(oc get nodes -ojsonpath='{.items[0].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"'; done { "enableServer": true, "staticPodPath": "/etc/kubernetes/manifests", "syncFrequency": "1m0s", "fileCheckFrequency": "20s", "httpCheckFrequency": "20s", "address": "0.0.0.0", "port": 10250, "tlsCipherSuites": [ "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" ], "tlsMinVersion": "VersionTLS12", "rotateCertificates": true, "serverTLSBootstrap": true, "authentication": { "x509": { "clientCAFile": "/etc/kubernetes/kubelet-ca.crt" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "registryPullQPS": 5, "registryBurst": 10, "eventRecordQPS": 10, "eventBurst": 10, "enableDebuggingHandlers": true, "healthzPort": 10248, "healthzBindAddress": "127.0.0.1", "oomScoreAdj": -999, "clusterDomain": "cluster.local", "clusterDNS": [ "172.30.0.10" ], "streamingConnectionIdleTimeout": "5h0m0s", "nodeStatusUpdateFrequency": "10s", "nodeStatusReportFrequency": "5m0s", "nodeLeaseDurationSeconds": 40, "imageMinimumGCAge": "2m0s", "imageGCHighThresholdPercent": 85, "imageGCLowThresholdPercent": 80, "volumeStatsAggPeriod": "1m0s", "systemCgroups": "/system.slice", "cgroupRoot": "/", "cgroupsPerQOS": true, "cgroupDriver": "systemd", "cpuManagerPolicy": "none", "cpuManagerReconcilePeriod": "10s", "memoryManagerPolicy": "None", "topologyManagerPolicy": "none", "topologyManagerScope": "container", "runtimeRequestTimeout": "2m0s", "hairpinMode": "promiscuous-bridge", "maxPods": 250, "podPidsLimit": 4096, "resolvConf": "/etc/resolv.conf", "cpuCFSQuota": true, "cpuCFSQuotaPeriod": "100ms", "nodeStatusMaxImages": 50, "maxOpenFiles": 1000000, "contentType": "application/vnd.kubernetes.protobuf", "kubeAPIQPS": 50, "kubeAPIBurst": 100, "serializeImagePulls": false, "evictionHard": { "imagefs.available": "15%", "imagefs.inodesFree": "5%", "memory.available": "300Mi", "nodefs.available": "10%", "nodefs.inodesFree": "5%" }, "evictionSoft": { "imagefs.available": "15%", "imagefs.inodesFree": "11%", "memory.available": "600Mi", "nodefs.available": "10%", "nodefs.inodesFree": "5%" }, "evictionSoftGracePeriod": { "imagefs.available": "1m30s", "imagefs.inodesFree": "1m30s", "memory.available": "1m30s", "nodefs.available": "1m30s", "nodefs.inodesFree": "1m30s" }, "evictionPressureTransitionPeriod": "5m0s", "enableControllerAttachDetach": true, "makeIPTablesUtilChains": true, "iptablesMasqueradeBit": 14, "iptablesDropBit": 15, "featureGates": { "APIPriorityAndFairness": true, "CSIMigrationAzureFile": false, "CSIMigrationvSphere": false, "DownwardAPIHugePages": true, "RotateKubeletServerCertificate": true }, "failSwapOn": true, "memorySwap": {}, "containerLogMaxSize": "50Mi", "containerLogMaxFiles": 5, "configMapAndSecretChangeDetectionStrategy": "Watch", "systemReserved": { "cpu": "500m", "memory": "1Gi" }, "enforceNodeAllocatable": [ "pods" ], "volumePluginDir": "/etc/kubernetes/kubelet-plugins/volume/exec", "logging": { "format": "text", "flushFrequency": 5000000000, "verbosity": 2, "options": { "json": { "infoBufferSize": "0" } } }, "enableSystemLogHandler": true, "shutdownGracePeriod": "0s", "shutdownGracePeriodCriticalPods": "0s", "enableProfilingHandler": true, "enableDebugFlagsHandler": true, "seccompDefault": false, "memoryThrottlingFactor": 0.8, "registerWithTaints": [ { "key": "node-role.kubernetes.io/master", "effect": "NoSchedule" } ], "registerNode": true, "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1" }
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Compliance Operator bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:6657
I was able to recreate this on 0.1.57: $ oc get ccr -n openshift-compliance | grep kubelet-eviction-threshold ocp4-cis-kubelet-eviction-thresholds-set-hard-imagefs-available FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-hard-memory-available FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-hard-nodefs-available FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-hard-nodefs-inodesfree FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-soft-imagefs-available FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-soft-memory-available FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-soft-nodefs-available FAIL medium ocp4-cis-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree FAIL medium $ for NODE_NAME in $(oc get nodes -ojsonpath='{.items[*].metadata.name}'); do oc get --raw /api/v1/nodes/$NODE_NAME/proxy/configz | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1" | .evictionHard."memory.available"'; done "100Mi" "100Mi" "100Mi" "100Mi" "100Mi" "100Mi"
Since we use Jira for issues, I opened a new issue tracking this there: https://issues.redhat.com/browse/OCPBUGS-4062
The reason why this failed is that the default variable that we use to check .evictionHard."memory.available" is "200Mi" defined using the variable "ocp4-var-kubelet-evictionhard-memory-available" if this parameter already defaults present in the system, the user can choose to use a tailored profile to set variable "ocp4-var-kubelet-evictionhard-memory-available" to match the default variable if that match desired compliance requirement.