2040401 – The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster

Bug 2040401 - The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster

Summary: The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Compliance Operator
Sub Component:
Version:	4.10
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.10.0
Assignee:	Vincent Shen
QA Contact:	Prashant Dhamdhere
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-01-13 16:12 UTC by Prashant Dhamdhere
Modified:	2022-02-07 05:46 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	* Previously, a manually created `MachineConfig` object for `KubeletConfig` prevented a `KubeletConfig` object from being generated for remediation, leaving the remediation in the `Pending` state. With this release, a `KubeletConfig` object is created by the remediation, regardless if there is a manually created `MachineConfig` object for `KubeletConfig`. As a result, `KubeletConfig` remediations now work as expected. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2040401[BZ#2040401])
Clone Of:
Environment:
Last Closed:	2022-02-07 05:46:25 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift compliance-operator pull 775	0	None	open	Bug 2040401: fix can't find owner of 98-<pool>-generated-kublet	2022-01-17 20:31:22 UTC
Red Hat Product Errata	RHBA-2022:0416	0	None	None	None	2022-02-07 05:46:29 UTC

Description Prashant Dhamdhere 2022-01-13 16:12:32 UTC

Description of problem:

The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster

# oc get suite
NAME         PHASE   RESULT
my-cis-ssb   DONE    NON-COMPLIANT


# oc get rems |head
NAME                                                                             STATE
ocp4-cis-node-master-kubelet-configure-event-creation                            Pending
ocp4-cis-node-master-kubelet-configure-tls-cipher-suites                         Pending
ocp4-cis-node-master-kubelet-enable-iptables-util-chains                         Pending
ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults                      MissingDependencies
ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl                        Applied
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available      Pending
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available-1    Pending
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree     Pending
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-inodesfree-1   Pending


# oc get rems |tail
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-imagefs-inodesfree-2   Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available       Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available-1     Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-memory-available-2     Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available       Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available-1     Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-available-2     Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree      Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-1    Pending
ocp4-cis-node-worker-kubelet-eviction-thresholds-set-soft-nodefs-inodesfree-2    Pending

# oc get kubeletconfig
No resources found

# oc get mc -lcompliance.openshift.io/suite=my-cis-ssb
NAME                                                           GENERATEDBYCONTROLLER   IGNITIONVERSION   AGE
75-ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl                           3.1.0             2m50s
75-ocp4-cis-node-worker-kubelet-enable-protect-kernel-sysctl                           3.1.0             3m35s


Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-11-065245 + compliance-operator.v0.1.47

How reproducible:
Always

Steps to Reproduce:

1. Deploy Disconnected UPI_Vsphere cluster
2. Install compliance operator v0.1.47
3. Create scansettingbinding object with auto-remediation enable

$ oc create -f - << EOF
apiVersion: compliance.openshift.io/v1alpha1
kind: ScanSettingBinding
metadata:
  name: my-cis-ssb
profiles:
  - name: ocp4-cis
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
  - name: ocp4-cis-node
    kind: Profile
    apiGroup: compliance.openshift.io/v1alpha1
settingsRef:
  name: default-auto-apply
  kind: ScanSetting
  apiGroup: compliance.openshift.io/v1alpha1
EOF

4. Check scan result 

# oc get scan
NAME                   PHASE   RESULT
ocp4-cis               DONE    NON-COMPLIANT
ocp4-cis-node-master   DONE    NON-COMPLIANT
ocp4-cis-node-worker   DONE    NON-COMPLIANT

# oc get suite
NAME         PHASE   RESULT
my-cis-ssb   DONE    NON-COMPLIANT

5. Check remediations status

#  oc get rems |head -n4
NAME                                                                             STATE
ocp4-cis-node-master-kubelet-configure-event-creation                            Pending
ocp4-cis-node-master-kubelet-configure-tls-cipher-suites                         Pending
ocp4-cis-node-master-kubelet-enable-iptables-util-chains                         Pending


Actual results:
The KubeletConfig remediation goes in pending state on UPI_Vsphere cluster after scan complete.

Expected results:
The KubeletConfig remediation should get applied after scan complete and the machinConfig should 
get generated for KubeletConfig remediation


Additional info:

Tested on other cluster environment and the KubeletConfig remediation works expected.

# oc describe rems ocp4-cis-node-master-kubelet-configure-event-creation
Name:         ocp4-cis-node-master-kubelet-configure-event-creation
Namespace:    openshift-compliance
Labels:       compliance.openshift.io/scan-name=ocp4-cis-node-master
              compliance.openshift.io/suite=my-cis-ssb
Annotations:  <none>
API Version:  compliance.openshift.io/v1alpha1
Kind:         ComplianceRemediation
Metadata:
  Creation Timestamp:  2022-01-13T15:33:20Z
  Generation:          2
  Managed Fields:
    API Version:  compliance.openshift.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:apply:
      f:status:
        .:
        f:applicationState:
    Manager:    compliance-operator
    Operation:  Update
    Time:       2022-01-13T15:33:35Z
  Owner References:
    API Version:           compliance.openshift.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  ComplianceCheckResult
    Name:                  ocp4-cis-node-master-kubelet-configure-event-creation
    UID:                   670d7111-154f-43a6-8069-1dd03acad666
  Resource Version:        315782
  UID:                     174b10ed-448e-4eb9-86ad-bc6a09f64641
Spec:
  Apply:  true
  Current:
    Object:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         KubeletConfig
      Spec:
        Kubelet Config:
          Event Record QPS:  10
  Outdated:
  Type:  Configuration
Status:
  Application State:  Pending
Events:               <none>

Comment 1 Vincent Shen 2022-01-19 16:57:24 UTC

Related PR: https://github.com/openshift/compliance-operator/pull/775

Comment 4 Prashant Dhamdhere 2022-02-02 09:37:11 UTC

[Bug_Verification]

Looks good. The KubeletConfig remediation is getting applied successfully on UPI_Vsphere cluster
and the kubeletconfig are getting created for machineConfigPools.

Verified on:

4.10.0-fc.4 + compliance-operator.v0.1.48


# oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-fc.4   True        False         91m     Cluster version is 4.10.0-fc.4

# oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
compliance-operator.v0.1.48       Compliance Operator                0.1.48                Succeeded
elasticsearch-operator.5.3.4-14   OpenShift Elasticsearch Operator   5.3.4-14              Succeeded
 
# oc get pods
NAME                                            READY   STATUS    RESTARTS        AGE
compliance-operator-5446b844c8-pwzjx            1/1     Running   1 (2m36s ago)   3m15s
ocp4-openshift-compliance-pp-7785bff67c-46jv5   1/1     Running   0               118s
rhcos4-openshift-compliance-pp-c84d79c7-drql5   1/1     Running   0               118s


# oc create -f - << EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ScanSettingBinding
> metadata:
>   name: my-cis-ssb
> profiles:
>   - name: ocp4-cis
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
>   - name: ocp4-cis-node
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
> settingsRef:
>   name: default-auto-apply
>   kind: ScanSetting
>   apiGroup: compliance.openshift.io/v1alpha1
> EOF
scansettingbinding.compliance.openshift.io/my-cis-ssb created


# oc get suite -w
NAME         PHASE       RESULT
my-cis-ssb   LAUNCHING   NOT-AVAILABLE
my-cis-ssb   LAUNCHING   NOT-AVAILABLE
my-cis-ssb   RUNNING     NOT-AVAILABLE
my-cis-ssb   RUNNING     NOT-AVAILABLE
my-cis-ssb   RUNNING     NOT-AVAILABLE
my-cis-ssb   AGGREGATING   NOT-AVAILABLE
my-cis-ssb   AGGREGATING   NOT-AVAILABLE
my-cis-ssb   AGGREGATING   NOT-AVAILABLE
my-cis-ssb   DONE          NON-COMPLIANT
my-cis-ssb   DONE          NON-COMPLIANT
 
# oc get suite 
NAME         PHASE   RESULT
my-cis-ssb   DONE    NON-COMPLIANT

# oc get scan
NAME                   PHASE   RESULT
ocp4-cis               DONE    NON-COMPLIANT
ocp4-cis-node-master   DONE    NON-COMPLIANT
ocp4-cis-node-worker   DONE    NON-COMPLIANT


# oc get rems |head 
NAME                                                                             STATE
ocp4-cis-api-server-encryption-provider-cipher                                   Applied
ocp4-cis-api-server-encryption-provider-config                                   Applied
ocp4-cis-node-master-kubelet-configure-event-creation                            Applied
ocp4-cis-node-master-kubelet-configure-tls-cipher-suites                         Applied
ocp4-cis-node-master-kubelet-enable-iptables-util-chains                         Applied
ocp4-cis-node-master-kubelet-enable-protect-kernel-defaults                      MissingDependencies
ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl                        Applied
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available      Applied
ocp4-cis-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available-1    Applied


# oc get kubeletconfig
NAME                                 AGE
compliance-operator-kubelet-master   55s
compliance-operator-kubelet-worker   62s

# oc get mc |grep kubelet | grep "99-m\|99-w"
99-master-generated-kubelet                                    3e9f2ca58e00d5dd5a54b18fb5b00c5571b5c8e3   3.2.0             3m47s
99-worker-generated-kubelet                                    3e9f2ca58e00d5dd5a54b18fb5b00c5571b5c8e3   3.2.0             3m54s

# oc get mc -l compliance.openshift.io/suite=my-cis-ssb
NAME                                                           GENERATEDBYCONTROLLER   IGNITIONVERSION   AGE
75-ocp4-cis-node-master-kubelet-enable-protect-kernel-sysctl                           3.1.0             100s
75-ocp4-cis-node-worker-kubelet-enable-protect-kernel-sysctl                           3.1.0             110s


# oc get mcp -w
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-a6264e2ab5f6847460b3c5e76bde3954   False     True       False      3              0                   0                     0                      113m
worker   rendered-worker-e35f93cd4f8c2d08a204235cf5fc6c60   False     True       False      2              0                   0                     0                      113m


# oc get ccr |head
NAME                                                                           STATUS   SEVERITY
ocp4-cis-accounts-restrict-service-account-tokens                              MANUAL   medium
ocp4-cis-accounts-unique-service-account                                       MANUAL   medium
ocp4-cis-api-server-admission-control-plugin-alwaysadmit                       PASS     medium
ocp4-cis-api-server-admission-control-plugin-alwayspullimages                  PASS     high
ocp4-cis-api-server-admission-control-plugin-namespacelifecycle                PASS     medium
ocp4-cis-api-server-admission-control-plugin-noderestriction                   PASS     medium
ocp4-cis-api-server-admission-control-plugin-scc                               PASS     medium
ocp4-cis-api-server-admission-control-plugin-securitycontextdeny               PASS     medium
ocp4-cis-api-server-admission-control-plugin-serviceaccount                    PASS     medium
                                                                              3.2.0             117m


# oc get kubeletconfig compliance-operator-kubelet-worker -oyaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  annotations:
    compliance.openshift.io/remediation: ""
  creationTimestamp: "2022-02-02T07:23:50Z"
  finalizers:
  - 99-worker-generated-kubelet
  generation: 19
  labels:
    compliance.openshift.io/scan-name: ocp4-cis-node-worker
    compliance.openshift.io/suite: my-cis-ssb
  name: compliance-operator-kubelet-worker
  resourceVersion: "73650"
  uid: a7d92a19-f5a3-4cc5-b085-80885fd70c0e
spec:
  kubeletConfig:
    eventRecordQPS: 10
    evictionHard:
      imagefs.available: 10%
      imagefs.inodesFree: 5%
      memory.available: 200Mi
      nodefs.available: 5%
      nodefs.inodesFree: 4%
    evictionPressureTransitionPeriod: 0s
    evictionSoft:
      imagefs.available: 15%
      imagefs.inodesFree: 10%
      memory.available: 500Mi
      nodefs.available: 10%
      nodefs.inodesFree: 5%
    evictionSoftGracePeriod:
      imagefs.available: 1m30s
      imagefs.inodesFree: 1m30s
      memory.available: 1m30s
      nodefs.available: 1m30s
      nodefs.inodesFree: 1m30s
    makeIPTablesUtilChains: true
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
status:
  conditions:
  - lastTransitionTime: "2022-02-02T07:23:54Z"
    message: Success
    status: "True"
    type: Success

# oc describe rems ocp4-cis-node-master-kubelet-configure-event-creation
Name:         ocp4-cis-node-master-kubelet-configure-event-creation
Namespace:    openshift-compliance
Labels:       compliance.openshift.io/scan-name=ocp4-cis-node-master
              compliance.openshift.io/suite=my-cis-ssb
Annotations:  <none>
API Version:  compliance.openshift.io/v1alpha1
Kind:         ComplianceRemediation
Metadata:
  Creation Timestamp:  2022-02-02T07:23:47Z
  Generation:          2
  Managed Fields:
    API Version:  compliance.openshift.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:applicationState:
    Manager:    compliance-operator
    Operation:  Update
    Time:       2022-02-02T07:24:00Z
  Owner References:
    API Version:           compliance.openshift.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  ComplianceCheckResult
    Name:                  ocp4-cis-node-master-kubelet-configure-event-creation
    UID:                   8ec06b40-32f2-492f-b6e8-af28d1cec85a
  Resource Version:        73936
  UID:                     d8ea6198-1534-406f-9fb4-acc4687606b3
Spec:
  Apply:  true
  Current:
    Object:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         KubeletConfig
      Spec:
        Kubelet Config:
          Event Record QPS:  10
  Outdated:
  Type:  Configuration
Status:
  Application State:  Applied
Events:               <none>

Comment 6 errata-xmlrpc 2022-02-07 05:46:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0416

Note You need to log in before you can comment on or make changes to this bug.