Bug 1907414
| Summary: | [OCP v46] Not all remediations get applied through machineConfig although the status of all rules shows Applied in ComplianceRemediations object | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Prashant Dhamdhere <pdhamdhe> |
| Component: | Compliance Operator | Assignee: | Jakub Hrozek <jhrozek> |
| Status: | CLOSED ERRATA | QA Contact: | Prashant Dhamdhere <pdhamdhe> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.6.z | CC: | jhrozek, josorior, mrogers, nkinder, xiyuan |
| Target Milestone: | --- | Keywords: | UpcomingSprint |
| Target Release: | 4.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1907410 | Environment: | |
| Last Closed: | 2021-01-19 13:53:52 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1907410 | ||
| Bug Blocks: | |||
|
Description
Prashant Dhamdhere
2020-12-14 13:25:22 UTC
This has been fixed upstream for the 4.6 branch as a part of the following PR: https://github.com/openshift/compliance-operator/pull/527 The specific commit for 4.6 is: https://github.com/openshift/compliance-operator/commit/995b63f41a9d67a693139d651e3f419d9a27092f The bug verification failed on the latest version of the compliance operator v0.1.24. The auto-remediation
does not get applied and all remediation rules go in Error state with the below error message
Error Message: not applying remediation that doesn't have a matching MachineconfigPool. Scan: worker-scan
[1] Applied compliancesuite CR through json file
$ oc get csv
NAME DISPLAY VERSION REPLACES PHASE
compliance-operator.v0.1.24 Compliance Operator 0.1.24 compliance-operator.v0.1.17 Succeeded
$ oc get pods
NAME READY STATUS RESTARTS AGE
compliance-operator-67c6f76f54-v7wjx 1/1 Running 0 2m16s
ocp4-openshift-compliance-pp-5bc4b87f99-qch92 1/1 Running 0 76s
rhcos4-openshift-compliance-pp-78d9c5d499-c8c5w 1/1 Running 0 76s
$ cat /tmp/e2e-test-compliance-zsiwoivl-db68w-uqpxgui1isc-config.json
{
"kind": "List",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"apiVersion": "compliance.openshift.io/v1alpha1",
"kind": "ComplianceSuite",
"metadata": {
"name": "worker-compliancesuite",
"namespace": "openshift-compliance"
},
"spec": {
"autoApplyRemediations": true,
"scans": [
{
"content": "ssg-rhcos4-ds.xml",
"contentImage": "quay.io/complianceascode/ocp4:latest",
"debug": true,
"name": "worker-scan",
"noExternalResources": false,
"nodeSelector": {
"node-role.kubernetes.io/wscan": ""
},
"profile": "xccdf_org.ssgproject.content_profile_moderate",
"rawResultStorage": {
"rotation": 0,
"size": ""
},
"rule": "",
"scanType": ""
}
],
"schedule": "0 1 * * *"
}
}
]
}
$ oc create -f /tmp/e2e-test-compliance-zsiwoivl-db68w-uqpxgui1isc-config.json
compliancesuite.compliance.openshift.io/worker-compliancesuite created
$ oc get pods -w
NAME READY STATUS RESTARTS AGE
aggregator-pod-worker-scan 0/1 Completed 0 4m43s
compliance-operator-67c6f76f54-v7wjx 1/1 Running 0 14m
ocp4-openshift-compliance-pp-5bc4b87f99-qch92 1/1 Running 0 13m
rhcos4-openshift-compliance-pp-78d9c5d499-c8c5w 1/1 Running 0 13m
worker-scan-ip-10-0-53-7.us-east-2.compute.internal-pod 0/2 Completed 0 7m53s
worker-scan-ip-10-0-58-26.us-east-2.compute.internal-pod 0/2 Completed 0 7m53s
worker-scan-ip-10-0-76-91.us-east-2.compute.internal-pod 0/2 Completed 0 7m53s
$ oc get compliancesuite
NAME PHASE RESULT
worker-compliancesuite DONE NON-COMPLIANT
$oc get mc
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
00-master eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
00-worker eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
01-master-container-runtime eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
01-master-kubelet eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
01-worker-container-runtime eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
01-worker-kubelet eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
99-master-fips 3.1.0 9h
99-master-generated-registries eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
99-master-ssh 3.1.0 9h
99-worker-fips 3.1.0 9h
99-worker-generated-registries eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
99-worker-ssh 3.1.0 9h
rendered-master-154a7dae66c15fef0545dc9af517c1dd eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
rendered-master-758ebd417045a6b5a48c8a9ce51fcd29 eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
rendered-worker-16ae54de226db0ba7841781a6e574756 eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
rendered-worker-636a9c3209570438fe833281745313af eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
$ oc get complianceremediations |tail
worker-scan-sysctl-net-ipv4-conf-default-send-redirects Error
worker-scan-sysctl-net-ipv4-icmp-echo-ignore-broadcasts Error
worker-scan-sysctl-net-ipv4-icmp-ignore-bogus-error-responses Error
worker-scan-sysctl-net-ipv4-tcp-syncookies Error
worker-scan-sysctl-net-ipv6-conf-all-accept-ra Error
worker-scan-sysctl-net-ipv6-conf-all-accept-redirects Error
worker-scan-sysctl-net-ipv6-conf-all-accept-source-route Error
worker-scan-sysctl-net-ipv6-conf-default-accept-ra Error
worker-scan-sysctl-net-ipv6-conf-default-accept-redirects Error
worker-scan-sysctl-net-ipv6-conf-default-accept-source-route Error
$ oc describe complianceremediations worker-scan-sysctl-net-ipv4-conf-default-send-redirects |tail -5
Outdated:
Status:
Application State: Error
Error Message: not applying remediation that doesn't have a matching MachineconfigPool. Scan: worker-scan
Events: <none>
[2] Applied compliancesuite CR through yaml and noticed the same issue
$ oc create -f - <<EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ComplianceSuite
> metadata:
> name: example-compliancesuite
> spec:
> autoApplyRemediations: true
> schedule: "0 1 * * *"
> scans:
> - name: wscan-scan
> profile: xccdf_org.ssgproject.content_profile_moderate
> content: ssg-rhcos4-ds.xml
> contentImage: quay.io/complianceascode/ocp4:latest
> debug: true
> nodeSelector:
> node-role.kubernetes.io/wscan: ""
> EOF
compliancesuite.compliance.openshift.io/example-compliancesuite created
$ oc get pods
NAME READY STATUS RESTARTS AGE
aggregator-pod-worker-scan 0/1 Completed 0 29m
aggregator-pod-wscan-scan 0/1 Completed 0 7m58s
compliance-operator-67c6f76f54-v7wjx 1/1 Running 0 39m
ocp4-openshift-compliance-pp-5bc4b87f99-qch92 1/1 Running 0 38m
rhcos4-openshift-compliance-pp-78d9c5d499-c8c5w 1/1 Running 0 38m
worker-scan-ip-10-0-53-7.us-east-2.compute.internal-pod 0/2 Completed 0 32m
worker-scan-ip-10-0-58-26.us-east-2.compute.internal-pod 0/2 Completed 0 32m
worker-scan-ip-10-0-76-91.us-east-2.compute.internal-pod 0/2 Completed 0 32m
wscan-scan-ip-10-0-53-7.us-east-2.compute.internal-pod 0/2 Completed 0 10m
wscan-scan-ip-10-0-58-26.us-east-2.compute.internal-pod 0/2 Completed 0 10m
wscan-scan-ip-10-0-76-91.us-east-2.compute.internal-pod 0/2 Completed 0 10m
$ oc get compliancesuite
NAME PHASE RESULT
example-compliancesuite DONE NON-COMPLIANT
worker-compliancesuite DONE NON-COMPLIANT
$ oc get complianceremediations |head
NAME STATE
worker-scan-audit-rules-dac-modification-chmod Error
worker-scan-audit-rules-dac-modification-chown Error
worker-scan-audit-rules-dac-modification-fchmod Error
worker-scan-audit-rules-dac-modification-fchmodat Error
worker-scan-audit-rules-dac-modification-fchown Error
worker-scan-audit-rules-dac-modification-fchownat Error
worker-scan-audit-rules-dac-modification-fremovexattr Error
worker-scan-audit-rules-dac-modification-fsetxattr Error
worker-scan-audit-rules-dac-modification-lchown Error
$ oc get mc
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
00-master eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
00-worker eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
01-master-container-runtime eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
01-master-kubelet eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
01-worker-container-runtime eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
01-worker-kubelet eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
99-master-fips 3.1.0 9h
99-master-generated-registries eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
99-master-ssh 3.1.0 9h
99-worker-fips 3.1.0 9h
99-worker-generated-registries eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
99-worker-ssh 3.1.0 9h
rendered-master-154a7dae66c15fef0545dc9af517c1dd eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
rendered-master-758ebd417045a6b5a48c8a9ce51fcd29 eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
rendered-worker-16ae54de226db0ba7841781a6e574756 eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 8h
rendered-worker-636a9c3209570438fe833281745313af eab9c35dfbeb0d21be6e1db3887acbbb93592d34 3.1.0 9h
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-154a7dae66c15fef0545dc9af517c1dd True False False 3 3 3 0 9h
worker rendered-worker-16ae54de226db0ba7841781a6e574756 True False False 5 5 5 0 9h
The above sounds like a typo to me. You scan targets the "wscan" pool:
"nodeSelector": {
"node-role.kubernetes.io/wscan": ""
},
but according to the oc get mcp output, you only have master and worker scans. This is also reported in the remediation statuses:
Error Message: not applying remediation that doesn't have a matching MachineconfigPool. Scan: worker-scan
Would you mind retesting yet again with a scan that matches the cluster pools?
I labelled all rhcos worker nodes with wscan label before performed scan and the same way, I reproduced this issue while reporting this bug (comment #1) and the remediation got applied. Let me try to retest this with worker nodeSelector, I guess it won't report the remediation error as a worker node MachineconfigPool is available by default on a cluster. LGTM. The remediation gets applied without an error with worker nodeSelector.
The machineConfig gets created for all remediation rules and the status of
these rules show Applied in ComplianceRemediations object.
Verified on :
4.6.0-0.nightly-2021-01-10-033123
compliance-operator.v0.1.24
$ oc get csv
NAME DISPLAY VERSION REPLACES PHASE
compliance-operator.v0.1.24 Compliance Operator 0.1.24 compliance-operator.v0.1.17 Succeeded
$ oc get pods
NAME READY STATUS RESTARTS AGE
compliance-operator-67c6f76f54-94v2h 1/1 Running 0 3m8s
ocp4-openshift-compliance-pp-5bc4b87f99-sggdx 1/1 Running 0 2m8s
rhcos4-openshift-compliance-pp-78d9c5d499-42vj8 1/1 Running 0 2m8s
$ cat /tmp/e2e-test-compliance-zsiwoivl-db68w-uqpxgui1isc-config.json
{
"kind": "List",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"apiVersion": "compliance.openshift.io/v1alpha1",
"kind": "ComplianceSuite",
"metadata": {
"name": "worker-compliancesuite",
"namespace": "openshift-compliance"
},
"spec": {
"autoApplyRemediations": true,
"scans": [
{
"content": "ssg-rhcos4-ds.xml",
"contentImage": "quay.io/complianceascode/ocp4:latest",
"debug": true,
"name": "worker-scan",
"noExternalResources": false,
"nodeSelector": {
"node-role.kubernetes.io/worker": ""
},
"profile": "xccdf_org.ssgproject.content_profile_moderate",
"rawResultStorage": {
"rotation": 0,
"size": ""
},
"rule": "",
"scanType": ""
}
],
"schedule": "0 1 * * *"
}
}
]
}
$ oc create -f /tmp/e2e-test-compliance-zsiwoivl-db68w-uqpxgui1isc-config.json
compliancesuite.compliance.openshift.io/worker-compliancesuite created
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-deca888a5e272c20ad2187d8eb6b35fc True False False 3 3 3 0 129m
worker rendered-worker-9daad341dd3d13a1c6fb4d08c1b13be2 True False False 3 3 3 0 129m
$ oc get pods
NAME READY STATUS RESTARTS AGE
aggregator-pod-worker-scan 0/1 Completed 0 65s
compliance-operator-67c6f76f54-94v2h 1/1 Running 0 8m4s
ocp4-openshift-compliance-pp-5bc4b87f99-sggdx 1/1 Running 0 7m4s
rhcos4-openshift-compliance-pp-78d9c5d499-42vj8 1/1 Running 0 7m4s
worker-scan-ip-10-0-57-140.us-east-2.compute.internal-pod 0/2 Completed 0 4m45s
worker-scan-ip-10-0-61-171.us-east-2.compute.internal-pod 0/2 Completed 0 4m45s
worker-scan-ip-10-0-75-65.us-east-2.compute.internal-pod 0/2 Completed 0 4m45s
$ oc get compliancesuite
NAME PHASE RESULT
worker-compliancesuite DONE NON-COMPLIANT
$ oc get mc -l compliance.openshift.io/scan-name=worker-compliancesuite| head
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
75-worker-scan-audit-rules-dac-modification-chmod 3.1.0 119s
75-worker-scan-audit-rules-dac-modification-chown 3.1.0 107s
75-worker-scan-audit-rules-dac-modification-fchmod 3.1.0 119s
75-worker-scan-audit-rules-dac-modification-fchmodat 3.1.0 109s
75-worker-scan-audit-rules-dac-modification-fchown 3.1.0 110s
75-worker-scan-audit-rules-dac-modification-fchownat 3.1.0 117s
75-worker-scan-audit-rules-dac-modification-fremovexattr 3.1.0 119s
75-worker-scan-audit-rules-dac-modification-fsetxattr 3.1.0 114s
75-worker-scan-audit-rules-dac-modification-lchown 3.1.0 112s
$ oc get mc -l compliance.openshift.io/scan-name=worker-compliancesuite |wc -l
103
$ oc get complianceremediations |head
NAME STATE
worker-scan-audit-rules-dac-modification-chmod Applied
worker-scan-audit-rules-dac-modification-chown Applied
worker-scan-audit-rules-dac-modification-fchmod Applied
worker-scan-audit-rules-dac-modification-fchmodat Applied
worker-scan-audit-rules-dac-modification-fchown Applied
worker-scan-audit-rules-dac-modification-fchownat Applied
worker-scan-audit-rules-dac-modification-fremovexattr Applied
worker-scan-audit-rules-dac-modification-fsetxattr Applied
worker-scan-audit-rules-dac-modification-lchown Applied
$ oc get complianceremediations |wc -l
103
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.6 compliance-operator security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0190 |