Bug 2040401
| Summary: | The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Prashant Dhamdhere <pdhamdhe> |
| Component: | Compliance Operator | Assignee: | Vincent Shen <wenshen> |
| Status: | CLOSED ERRATA | QA Contact: | Prashant Dhamdhere <pdhamdhe> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.10 | CC: | mrogers, stevsmit, wenshen, xiyuan |
| Target Milestone: | --- | ||
| Target Release: | 4.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
* Previously, a manually created `MachineConfig` object for `KubeletConfig` prevented a `KubeletConfig` object from being generated for remediation, leaving the remediation in the `Pending` state. With this release, a `KubeletConfig` object is created by the remediation, regardless if there is a manually created `MachineConfig` object for `KubeletConfig`. As a result, `KubeletConfig` remediations now work as expected. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2040401[*BZ#2040401*])
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-02-07 05:46:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
[Bug_Verification]
Looks good. The KubeletConfig remediation is getting applied successfully on UPI_Vsphere cluster
and the kubeletconfig are getting created for machineConfigPools.
Verified on:
4.10.0-fc.4 + compliance-operator.v0.1.48
# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.10.0-fc.4 True False 91m Cluster version is 4.10.0-fc.4
# oc get csv
NAME DISPLAY VERSION REPLACES PHASE
compliance-operator.v0.1.48 Compliance Operator 0.1.48 Succeeded
elasticsearch-operator.5.3.4-14 OpenShift Elasticsearch Operator 5.3.4-14 Succeeded
# oc get pods
NAME READY STATUS RESTARTS AGE
compliance-operator-5446b844c8-pwzjx 1/1 Running 1 (2m36s ago) 3m15s
ocp4-openshift-compliance-pp-7785bff67c-46jv5 1/1 Running 0 118s
rhcos4-openshift-compliance-pp-c84d79c7-drql5 1/1 Running 0 118s
# oc create -f - << EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ScanSettingBinding
> metadata:
> name: my-cis-ssb
> profiles:
> - name: ocp4-cis
> kind: Profile
> apiGroup: compliance.openshift.io/v1alpha1
> - name: ocp4-cis-node
> kind: Profile
> apiGroup: compliance.openshift.io/v1alpha1
> settingsRef:
> name: default-auto-apply
> kind: ScanSetting
> apiGroup: compliance.openshift.io/v1alpha1
> EOF
scansettingbinding.compliance.openshift.io/my-cis-ssb created
# oc get suite -w
NAME PHASE RESULT
my-cis-ssb LAUNCHING NOT-AVAILABLE
my-cis-ssb LAUNCHING NOT-AVAILABLE
my-cis-ssb RUNNING NOT-AVAILABLE
my-cis-ssb RUNNING NOT-AVAILABLE
my-cis-ssb RUNNING NOT-AVAILABLE
my-cis-ssb AGGREGATING NOT-AVAILABLE
my-cis-ssb AGGREGATING NOT-AVAILABLE
my-cis-ssb AGGREGATING NOT-AVAILABLE
my-cis-ssb DONE NON-COMPLIANT
my-cis-ssb DONE NON-COMPLIANT
# oc get suite
NAME PHASE RESULT
my-cis-ssb DONE NON-COMPLIANT
# oc get scan
NAME PHASE RESULT
ocp4-cis DONE NON-COMPLIANT
ocp4-cis-node-master DONE NON-COMPLIANT
ocp4-cis-node-worker DONE NON-COMPLIANT
# oc get rems |head
NAME STATE
ocp4-cis-api-server-encryption-provider-cipher Applied
ocp4-cis-api-server-encryption-provider-config Applied
ocp4-cis-node-master-kubelet-configure-event-creation Applied
ocp4-cis-node-master-kubelet-configure-tls-cipher-suites Applied
ocp4-cis-node-master-kubelet-enable-iptables-util-chains Applied
ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults MissingDependencies
ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl Applied
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available Applied
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available-1 Applied
# oc get kubeletconfig
NAME AGE
compliance-operator-kubelet-master 55s
compliance-operator-kubelet-worker 62s
# oc get mc |grep kubelet | grep "99-m\|99-w"
99-master-generated-kubelet 3e9f2ca58e00d5dd5a54b18fb5b00c5571b5c8e3 3.2.0 3m47s
99-worker-generated-kubelet 3e9f2ca58e00d5dd5a54b18fb5b00c5571b5c8e3 3.2.0 3m54s
# oc get mc -l compliance.openshift.io/suite=my-cis-ssb
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
75-ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl 3.1.0 100s
75-ocp4-cis-node-worker-kubelet-enable-protect-kernel-sysctl 3.1.0 110s
# oc get mcp -w
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-a6264e2ab5f6847460b3c5e76bde3954 False True False 3 0 0 0 113m
worker rendered-worker-e35f93cd4f8c2d08a204235cf5fc6c60 False True False 2 0 0 0 113m
# oc get ccr |head
NAME STATUS SEVERITY
ocp4-cis-accounts-restrict-service-account-tokens MANUAL medium
ocp4-cis-accounts-unique-service-account MANUAL medium
ocp4-cis-api-server-admission-control-plugin-alwaysadmit PASS medium
ocp4-cis-api-server-admission-control-plugin-alwayspullimages PASS high
ocp4-cis-api-server-admission-control-plugin-namespacelifecycle PASS medium
ocp4-cis-api-server-admission-control-plugin-noderestriction PASS medium
ocp4-cis-api-server-admission-control-plugin-scc PASS medium
ocp4-cis-api-server-admission-control-plugin-securitycontextdeny PASS medium
ocp4-cis-api-server-admission-control-plugin-serviceaccount PASS medium
3.2.0 117m
# oc get kubeletconfig compliance-operator-kubelet-worker -oyaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
annotations:
compliance.openshift.io/remediation: ""
creationTimestamp: "2022-02-02T07:23:50Z"
finalizers:
- 99-worker-generated-kubelet
generation: 19
labels:
compliance.openshift.io/scan-name: ocp4-cis-node-worker
compliance.openshift.io/suite: my-cis-ssb
name: compliance-operator-kubelet-worker
resourceVersion: "73650"
uid: a7d92a19-f5a3-4cc5-b085-80885fd70c0e
spec:
kubeletConfig:
eventRecordQPS: 10
evictionHard:
imagefs.available: 10%
imagefs.inodesFree: 5%
memory.available: 200Mi
nodefs.available: 5%
nodefs.inodesFree: 4%
evictionPressureTransitionPeriod: 0s
evictionSoft:
imagefs.available: 15%
imagefs.inodesFree: 10%
memory.available: 500Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionSoftGracePeriod:
imagefs.available: 1m30s
imagefs.inodesFree: 1m30s
memory.available: 1m30s
nodefs.available: 1m30s
nodefs.inodesFree: 1m30s
makeIPTablesUtilChains: true
tlsCipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
machineConfigPoolSelector:
matchLabels:
pools.operator.machineconfiguration.openshift.io/worker: ""
status:
conditions:
- lastTransitionTime: "2022-02-02T07:23:54Z"
message: Success
status: "True"
type: Success
# oc describe rems ocp4-cis-node-master-kubelet-configure-event-creation
Name: ocp4-cis-node-master-kubelet-configure-event-creation
Namespace: openshift-compliance
Labels: compliance.openshift.io/scan-name=ocp4-cis-node-master
compliance.openshift.io/suite=my-cis-ssb
Annotations: <none>
API Version: compliance.openshift.io/v1alpha1
Kind: ComplianceRemediation
Metadata:
Creation Timestamp: 2022-02-02T07:23:47Z
Generation: 2
Managed Fields:
API Version: compliance.openshift.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:applicationState:
Manager: compliance-operator
Operation: Update
Time: 2022-02-02T07:24:00Z
Owner References:
API Version: compliance.openshift.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: ComplianceCheckResult
Name: ocp4-cis-node-master-kubelet-configure-event-creation
UID: 8ec06b40-32f2-492f-b6e8-af28d1cec85a
Resource Version: 73936
UID: d8ea6198-1534-406f-9fb4-acc4687606b3
Spec:
Apply: true
Current:
Object:
API Version: machineconfiguration.openshift.io/v1
Kind: KubeletConfig
Spec:
Kubelet Config:
Event Record QPS: 10
Outdated:
Type: Configuration
Status:
Application State: Applied
Events: <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0416 |
Description of problem: The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster # oc get suite NAME PHASE RESULT my-cis-ssb DONE NON-COMPLIANT # oc get rems |head NAME STATE ocp4-cis-node-master-kubelet-configure-event-creation Pending ocp4-cis-node-master-kubelet-configure-tls-cipher-suites Pending ocp4-cis-node-master-kubelet-enable-iptables-util-chains Pending ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults MissingDependencies ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl Applied ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available Pending ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available-1 Pending ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree Pending ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-1 Pending # oc get rems |tail ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-2 Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available-1 Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available-2 Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available-1 Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available-2 Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-1 Pending ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-2 Pending # oc get kubeletconfig No resources found # oc get mc -lcompliance.openshift.io/suite=my-cis-ssb NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 75-ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl 3.1.0 2m50s 75-ocp4-cis-node-worker-kubelet-enable-protect-kernel-sysctl 3.1.0 3m35s Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-11-065245 + compliance-operator.v0.1.47 How reproducible: Always Steps to Reproduce: 1. Deploy Disconnected UPI_Vsphere cluster 2. Install compliance operator v0.1.47 3. Create scansettingbinding object with auto-remediation enable $ oc create -f - << EOF apiVersion: compliance.openshift.io/v1alpha1 kind: ScanSettingBinding metadata: name: my-cis-ssb profiles: - name: ocp4-cis kind: Profile apiGroup: compliance.openshift.io/v1alpha1 - name: ocp4-cis-node kind: Profile apiGroup: compliance.openshift.io/v1alpha1 settingsRef: name: default-auto-apply kind: ScanSetting apiGroup: compliance.openshift.io/v1alpha1 EOF 4. Check scan result # oc get scan NAME PHASE RESULT ocp4-cis DONE NON-COMPLIANT ocp4-cis-node-master DONE NON-COMPLIANT ocp4-cis-node-worker DONE NON-COMPLIANT # oc get suite NAME PHASE RESULT my-cis-ssb DONE NON-COMPLIANT 5. Check remediations status # oc get rems |head -n4 NAME STATE ocp4-cis-node-master-kubelet-configure-event-creation Pending ocp4-cis-node-master-kubelet-configure-tls-cipher-suites Pending ocp4-cis-node-master-kubelet-enable-iptables-util-chains Pending Actual results: The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster after scan complete. Expected results: The KubeletConfig remediation should get applied after scan complete and the machinConfig should get generated for KubeletConfig remediation Additional info: Tested on other cluster environment and the KubeletConfig remediation works expected. # oc describe rems ocp4-cis-node-master-kubelet-configure-event-creation Name: ocp4-cis-node-master-kubelet-configure-event-creation Namespace: openshift-compliance Labels: compliance.openshift.io/scan-name=ocp4-cis-node-master compliance.openshift.io/suite=my-cis-ssb Annotations: <none> API Version: compliance.openshift.io/v1alpha1 Kind: ComplianceRemediation Metadata: Creation Timestamp: 2022-01-13T15:33:20Z Generation: 2 Managed Fields: API Version: compliance.openshift.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:spec: f:apply: f:status: .: f:applicationState: Manager: compliance-operator Operation: Update Time: 2022-01-13T15:33:35Z Owner References: API Version: compliance.openshift.io/v1alpha1 Block Owner Deletion: true Controller: true Kind: ComplianceCheckResult Name: ocp4-cis-node-master-kubelet-configure-event-creation UID: 670d7111-154f-43a6-8069-1dd03acad666 Resource Version: 315782 UID: 174b10ed-448e-4eb9-86ad-bc6a09f64641 Spec: Apply: true Current: Object: API Version: machineconfiguration.openshift.io/v1 Kind: KubeletConfig Spec: Kubelet Config: Event Record QPS: 10 Outdated: Type: Configuration Status: Application State: Pending Events: <none>