2053602 – The rule ocp4-cluster-version-operator-verify-integrity failing while checking CVO history.

Bug 2053602 - The rule ocp4-cluster-version-operator-verify-integrity failing while checking CVO history.

Summary: The rule ocp4-cluster-version-operator-verify-integrity failing while checkin...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Compliance Operator
Sub Component:
Version:	4.9
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Vincent Shen
QA Contact:	Prashant Dhamdhere
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-02-11 15:33 UTC by Mithilesh Kaur Bagga
Modified:	2022-08-02 09:36 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: The ocp4-cluster-version-operator-verify-integrity always checked the first entry in the CVO history Consequence: This would fail in upgrade cases were subsequent versions of OCP would be verified Fix: Use the new content, which is smarter about detecting verified versions Result: The compliance check result for ocp4-cluster-version-operator-verify-integrity should be accurate with the actual CVO history
Clone Of:
Environment:
Last Closed:	2022-04-18 07:54:00 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	6970213	0	None	None	None	2022-08-02 09:36:49 UTC
Red Hat Product Errata	RHBA-2022:1148	0	None	None	None	2022-04-18 07:54:10 UTC

Comment 2 Vincent Shen 2022-02-22 15:50:37 UTC

Hi Hradayesh,

Sorry for the late reply, I think one of temporal workaround, for now, is that Users can use tailored profiles to ignore this rule, and check the integrity manually using the following steps: https://coreos.slack.com/archives/CEGKQ43CP/p1645491504250829?thread_ts=1643984204.117259&cid=CEGKQ43CP

We are working on a solution to refine/fix this rule to do the automated check.

Best,
Vincent

Comment 3 Jakub Hrozek 2022-03-09 13:57:16 UTC

Still in progress

Comment 8 Prashant Dhamdhere 2022-04-04 16:59:20 UTC

[Bug_Verification]

Looks good. The rule is not failing for installed version of CVO history.
Also confirmed, the rule is failing if the verified is false under status 
history for one of the item.


Verified on:
4.8.10 + compliance-operator.v0.1.49 + 4.9.15


$ oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.10    True        False         6m22s   Cluster version is 4.8.10

$ oc project openshift-compliance
Now using project "openshift-compliance" on server "https://api.pdhamdhe0404.qe.devcluster.openshift.com:6443".


$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
compliance-operator.v0.1.49       Compliance Operator                0.1.49                Succeeded
elasticsearch-operator.5.2.9-31   OpenShift Elasticsearch Operator   5.2.9-31              Succeeded


$ oc get pods
NAME                                              READY   STATUS    RESTARTS   AGE
compliance-operator-75c6c56599-sk2cx              1/1     Running   1          3m9s
ocp4-openshift-compliance-pp-56dd949976-9gtq5     1/1     Running   0          97s
rhcos4-openshift-compliance-pp-7595d55cfb-2mwv9   1/1     Running   0          97s


$ oc get rules |grep version-operator
ocp4-cluster-version-operator-exists                                                36m
ocp4-cluster-version-operator-verify-integrity                                      36m



$ oc create -f - << EOF
> apiVersion: compliance.openshift.io/v1alpha1
> kind: ScanSettingBinding
> metadata:
>   name: my-ssb-moderate
> profiles:
>   - name: ocp4-moderate
>     kind: Profile
>     apiGroup: compliance.openshift.io/v1alpha1
> settingsRef:
>   name: default
>   kind: ScanSetting
>   apiGroup: compliance.openshift.io/v1alpha1
> EOF
scansettingbinding.compliance.openshift.io/my-ssb-moderate created


$ oc get suite -w
NAME              PHASE       RESULT
my-ssb-moderate   LAUNCHING   NOT-AVAILABLE
my-ssb-moderate   RUNNING     NOT-AVAILABLE
my-ssb-moderate   AGGREGATING   NOT-AVAILABLE
my-ssb-moderate   DONE          NON-COMPLIANT
my-ssb-moderate   DONE          NON-COMPLIANT
 
$ oc get compliancecheckresult ocp4-moderate-cluster-version-operator-verify-integrity -ojsonpath={.instructions}
Run the following command to retrieve the Cluster Version objects in the system:
$ oc get clusterversion version -o yaml
Make sure verified is true under status history for each item.
 

$ oc get compliancecheckresult ocp4-moderate-cluster-version-operator-verify-integrity
NAME                                                      STATUS   SEVERITY
ocp4-moderate-cluster-version-operator-verify-integrity   PASS     medium
 

$ oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "completionTime": "2022-04-04T09:50:35Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db",
    "startedTime": "2022-04-04T09:17:37Z",
    "state": "Completed",
    "verified": false,
    "version": "4.8.10"
  }
]

$ oc get clusterversion -o json|jq ".items[0].spec"
{
  "channel": "stable-4.8",
  "clusterID": "4193c97b-267f-4272-8f66-e7a260c52df8"
}

$ oc adm upgrade
Cluster version is 4.8.10

Updates:

VERSION IMAGE
4.8.11  quay.io/openshift-release-dev/ocp-release@sha256:26f9da8c2567ddf15f917515008563db8b3c9e43120d3d22f9d00a16b0eb9b97
4.8.12  quay.io/openshift-release-dev/ocp-release@sha256:c3af995af7ee85e88c43c943e0a64c7066d90e77fafdabc7b22a095e4ea3c25a
4.8.13  quay.io/openshift-release-dev/ocp-release@sha256:5d396ad7d5f3cb527580c735e87dfd3b853bbb531e7f03e3a184d0accc223cdf
4.8.14  quay.io/openshift-release-dev/ocp-release@sha256:bf48faa639523b73131ec7c91637d5c94d33a4afe09ac8bdad672862f5e86ccb
4.8.15  quay.io/openshift-release-dev/ocp-release@sha256:92b684258b9f80dadce5b2f4efce0e110fb92b9f08f8837bdcbe7393c57d388f
4.8.17  quay.io/openshift-release-dev/ocp-release@sha256:1935b6c8277e351550bd7bfcc4d5df7c4ba0f7a90165c022e2ffbe789b15574a
4.8.18  quay.io/openshift-release-dev/ocp-release@sha256:321aae3d3748c589bc2011062cee9fd14e106f258807dc2d84ced3f7461160ea
4.8.19  quay.io/openshift-release-dev/ocp-release@sha256:ac19c975be8b8a449dedcdd7520e970b1cc827e24042b8976bc0495da32c6b59
4.8.20  quay.io/openshift-release-dev/ocp-release@sha256:ca7a910891da55bb3b555fab1973878c3918dbf908cfd415ef2941287300e698
4.8.21  quay.io/openshift-release-dev/ocp-release@sha256:f7e664bf56c882f934ed02eb05018e2683ddf42135e33eae1e4192948372d5ae
4.8.22  quay.io/openshift-release-dev/ocp-release@sha256:019e313e9d073c21aeae5c36b6b7e010783ad284c6bc0b0f716bbac501e20d68
4.8.23  quay.io/openshift-release-dev/ocp-release@sha256:3fab205d36c66825423274eac90f4c142a18cdf358b4a666a1783d325afba860
4.8.24  quay.io/openshift-release-dev/ocp-release@sha256:0708475f51e969dd9e6902d958f8ffed668b1b9c8d63b6241e7c9e40d9548eee
4.8.25  quay.io/openshift-release-dev/ocp-release@sha256:b2ff872593d201151f52cf7dd651687ba923c4d6f2e7671bd324020362bd0d44
4.8.26  quay.io/openshift-release-dev/ocp-release@sha256:6814b7970707384c9fd2100e183920feebb335f7af93132ecb18053d120db703
4.8.27  quay.io/openshift-release-dev/ocp-release@sha256:36061ae9ccad77bbae491de8bf50be45eeb3409c5d596f63c445e72db43a872d
4.8.28  quay.io/openshift-release-dev/ocp-release@sha256:ba1299680b542e46744307afc7effc15957a20592d88de4651610b52ed8be9a8
4.8.29  quay.io/openshift-release-dev/ocp-release@sha256:9f9df3f16e7ddd66b95093b40858eb396cc937ec06546f8d70c87560973b9bbf
4.8.31  quay.io/openshift-release-dev/ocp-release@sha256:fbf79da6f2500b1a44a0ac0018d85581632e9e19edacc03ba34d5d2fec455d25
4.8.32  quay.io/openshift-release-dev/ocp-release@sha256:fd4c0a555955f3a3cf490c960e802069dfc9fff10722f2789279c6d8278723a1
4.8.33  quay.io/openshift-release-dev/ocp-release@sha256:352ad2d17474278d17853a804aa68764c719e65367078e5c89ca1cddfbf5cf2d
4.8.34  quay.io/openshift-release-dev/ocp-release@sha256:016a1e055bc839623abb4d4808f4135ee37b97dcf5b5cf4a586519450e6acbc8
4.8.35  quay.io/openshift-release-dev/ocp-release@sha256:f8c5174f61e6a268477a59da75930bd0b4d35c2b90f362007c78b5a2175c04f8


$ oc adm upgrade --to=4.8.35
Updating to 4.8.35
 
 
$ oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.10    True        True          15s     Working towards 4.8.35: downloading update
version   4.8.10    True        True          16s     Working towards 4.8.35: 9 of 681 done (1% complete)
version   4.8.10    True        True          39s     Working towards 4.8.35
version   4.8.10    True        True          39s     Working towards 4.8.35: downloading update
version   4.8.10    True        True          39s     Working towards 4.8.35: downloading update
version   4.8.10    True        True          39s     Working towards 4.8.35
version   4.8.10    True        True          39s     Working towards 4.8.35: 2 of 681 done (0% complete)
version   4.8.10    True        True          39s     Working towards 4.8.35: 3 of 681 done (0% complete)
version   4.8.10    True        True          39s     Working towards 4.8.35: 4 of 681 done (0% complete)
version   4.8.10    True        True          39s     Working towards 4.8.35: 6 of 681 done (0% complete)
version   4.8.10    True        True          39s     Working towards 4.8.35: 9 of 681 done (1% complete)
version   4.8.10    True        True          54s     Working towards 4.8.35: 71 of 681 done (10% complete)
version   4.8.10    True        True          5m24s   Working towards 4.8.35: 72 of 681 done (10% complete)
version   4.8.10    True        True          5m39s   Working towards 4.8.35: 95 of 681 done (13% complete)
version   4.8.10    True        True          6m24s   Working towards 4.8.35: 95 of 681 done (13% complete), waiting on kube-apiserver
version   4.8.10    True        True          6m54s   Working towards 4.8.35: 95 of 681 done (13% complete)
version   4.8.10    True        True          12m     Working towards 4.8.35: 95 of 681 done (13% complete), waiting on kube-apiserver
version   4.8.10    True        True          13m     Working towards 4.8.35: 95 of 681 done (13% complete)
version   4.8.10    True        True          18m     Working towards 4.8.35: 95 of 681 done (13% complete), waiting on kube-apiserver
version   4.8.10    True        True          20m     Working towards 4.8.35: 95 of 681 done (13% complete)
...
....
version   4.8.10    True        True          52m     Working towards 4.8.35: 555 of 681 done (81% complete), waiting on dns
version   4.8.10    True        True          55m     Working towards 4.8.35: 573 of 681 done (84% complete)
version   4.8.10    True        True          61m     Working towards 4.8.35: 573 of 681 done (84% complete), waiting on machine-config
version   4.8.10    True        True          64m     Working towards 4.8.35: 573 of 681 done (84% complete), waiting on machine-config
version   4.8.10    True        True          64m     Working towards 4.8.35: 573 of 681 done (84% complete)


$ oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.35    True        False         113m    Cluster version is 4.8.35


$ oc get suite
NAME              PHASE   RESULT
my-ssb-moderate   DONE    NON-COMPLIANT


$ oc-compliance rerun-now compliancesuite/my-ssb-moderate
Rerunning scans from 'my-ssb-moderate': ocp4-moderate
Re-running scan 'openshift-compliance/ocp4-moderate'


$ oc get suite -w
NAME              PHASE       RESULT
my-ssb-moderate   LAUNCHING   NOT-AVAILABLE
my-ssb-moderate   RUNNING     NOT-AVAILABLE
my-ssb-moderate   AGGREGATING   NOT-AVAILABLE
my-ssb-moderate   DONE          NON-COMPLIANT
my-ssb-moderate   DONE          NON-COMPLIANT


$ oc get compliancecheckresult ocp4-moderate-cluster-version-operator-verify-integrity
NAME                                                      STATUS   SEVERITY
ocp4-moderate-cluster-version-operator-verify-integrity   PASS     medium


$ oc get clusterversion -o json|jq ".items[0].spec"
{
  "channel": "stable-4.8",
  "clusterID": "4193c97b-267f-4272-8f66-e7a260c52df8",
  "desiredUpdate": {
    "force": false,
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:f8c5174f61e6a268477a59da75930bd0b4d35c2b90f362007c78b5a2175c04f8",
    "version": "4.8.35"
  }
}

$ oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "completionTime": "2022-04-04T13:02:49Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:f8c5174f61e6a268477a59da75930bd0b4d35c2b90f362007c78b5a2175c04f8",
    "startedTime": "2022-04-04T11:32:46Z",
    "state": "Completed",
    "verified": true,
    "version": "4.8.35"
  },
  {
    "completionTime": "2022-04-04T09:50:35Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db",
    "startedTime": "2022-04-04T09:17:37Z",
    "state": "Completed",
    "verified": false,
    "version": "4.8.10"
  }
]


$ oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release:4.9.15-x86_64 --allow-explicit-upgrade=true --force
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image quay.io/openshift-release-dev/ocp-release:4.9.15-x86_64


$ oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.35    True        True          7s      Working towards quay.io/openshift-release-dev/ocp-release:4.9.15-x86_64: downloading update
version   4.8.35    True        True          16s     Working towards 4.9.15: 9 of 737 done (1% complete)
...
....

version   4.8.35    True        True          87m     Working towards 4.9.15: 702 of 737 done (95% complete)
version   4.8.35    True        True          87m     Working towards 4.9.15: 704 of 737 done (95% complete)
version   4.9.15    True        False         0s      Cluster version is 4.9.15
version   4.9.15    True        False         3m4s    Cluster version is 4.9.15


$ oc get clusterversion 
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.15    True        False         10m     Cluster version is 4.9.15


$ oc get pods
NAME                                              READY   STATUS    RESTARTS   AGE
compliance-operator-75c6c56599-zlrk6              1/1     Running   0          29m
ocp4-openshift-compliance-pp-56dd949976-s6f22     1/1     Running   0          20m
rhcos4-openshift-compliance-pp-7595d55cfb-xb5bf   1/1     Running   0          20m


$ oc get suite
NAME              PHASE   RESULT
my-ssb-moderate   DONE    NON-COMPLIANT


$ oc-compliance rerun-now compliancesuite/my-ssb-moderate
Rerunning scans from 'my-ssb-moderate': ocp4-moderate
Re-running scan 'openshift-compliance/ocp4-moderate'


$ oc get suite -w
NAME              PHASE       RESULT
my-ssb-moderate   LAUNCHING   NOT-AVAILABLE
my-ssb-moderate   RUNNING     NOT-AVAILABLE
my-ssb-moderate   AGGREGATING   NOT-AVAILABLE
my-ssb-moderate   DONE          NON-COMPLIANT
my-ssb-moderate   DONE          NON-COMPLIANT


$ oc get compliancecheckresult ocp4-moderate-cluster-version-operator-verify-integrity
NAME                                                      STATUS   SEVERITY
ocp4-moderate-cluster-version-operator-verify-integrity   FAIL     medium



$ oc get clusterversion -o json|jq ".items[0].status.history"
[
  {
    "completionTime": "2022-04-04T16:28:25Z",
    "image": "quay.io/openshift-release-dev/ocp-release:4.9.15-x86_64",
    "startedTime": "2022-04-04T15:00:50Z",
    "state": "Completed",
    "verified": false,
    "version": "4.9.15"
  },
  {
    "completionTime": "2022-04-04T13:02:49Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:f8c5174f61e6a268477a59da75930bd0b4d35c2b90f362007c78b5a2175c04f8",
    "startedTime": "2022-04-04T11:32:46Z",
    "state": "Completed",
    "verified": true,
    "version": "4.8.35"
  },
  {
    "completionTime": "2022-04-04T09:50:35Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:53576e4df71a5f00f77718f25aec6ac7946eaaab998d99d3e3f03fcb403364db",
    "startedTime": "2022-04-04T09:17:37Z",
    "state": "Completed",
    "verified": false,
    "version": "4.8.10"
  }
]

Comment 10 errata-xmlrpc 2022-04-18 07:54:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Compliance Operator bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1148

Note You need to log in before you can comment on or make changes to this bug.