2070887 – Cv condition ImplicitlyEnabledCapabilities doesn’t complain about the disabled capabilities which is previously enabled

Bug 2070887 - Cv condition ImplicitlyEnabledCapabilities doesn’t complain about the disabled capabilities which is previously enabled

Summary: Cv condition ImplicitlyEnabledCapabilities doesn’t complain about the disable...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Cluster Version Operator
Sub Component:
Version:	4.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	medium
Target Milestone:	---
Target Release:	4.11.0
Assignee:	Jack Ottofaro
QA Contact:	Yang Yang
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	2070891 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-04-01 10:03 UTC by Yang Yang
Modified:	2022-08-10 11:03 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-08-10 11:03:06 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
CVO log file (2.34 MB, text/plain) 2022-04-01 10:03 UTC, Yang Yang	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-version-operator pull 761	0	None	open	Bug 2070887: pkg/cvo/sync_worker.go: set implicitly enabled caps earlier	2022-04-04 19:15:40 UTC
Red Hat Product Errata	RHSA-2022:5069	0	None	None	None	2022-08-10 11:03:20 UTC

Description Yang Yang 2022-04-01 10:03:31 UTC

Created attachment 1869925 [details]
CVO log file

Description of problem:

Disabling previously enabled cap by removing spec.capabilities.additionalEnabledCapabilities, cv condition ImplicitlyEnabledCapabilities doesn’t complain about the change.

$ oc get clusterversion -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2022-04-01T08:50:01Z"
    generation: 3
    name: version
    resourceVersion: "31704"
    uid: d64ba7f4-2e1e-45d3-bfa0-1ba28c4a726d
  spec:
    capabilities:
      baselineCapabilitySet: None
    channel: stable-4.11
    clusterID: 7ed605ca-954b-4a10-a209-2ed602e7e19e
  status:
    availableUpdates: null
    capabilities:
      enabledCapabilities:
      - marketplace
      knownCapabilities:
      - baremetal
      - marketplace
      - openshift-samples
    conditions:
    - lastTransitionTime: "2022-04-01T08:50:04Z"
      message: 'Unable to retrieve available updates: currently reconciling cluster
        version 4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest not found
        in the "stable-4.11" channel'
      reason: VersionNotFound
      status: "False"
      type: RetrievedUpdates
    - lastTransitionTime: "2022-04-01T08:50:04Z"
      message: Capabilities match configured spec
      reason: AsExpected
      status: "False"
      type: ImplicitlyEnabledCapabilities
    - lastTransitionTime: "2022-04-01T08:50:04Z"
      message: Payload loaded version="4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest"
        image="registry.build01.ci.openshift.org/ci-ln-2ydg80t/release@sha256:77bf803cfc45670f8bf7a5c08fce7de247c7c40f10c43f2c1ae51458d294bfc4"
      reason: PayloadLoaded
      status: "True"
      type: ReleaseAccepted
    - lastTransitionTime: "2022-04-01T09:15:42Z"
      message: Done applying 4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest
      status: "True"
      type: Available
    - lastTransitionTime: "2022-04-01T09:14:27Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2022-04-01T09:15:42Z"
      message: Cluster version is 4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest
      status: "False"
      type: Progressing
    desired:
      image: registry.build01.ci.openshift.org/ci-ln-2ydg80t/release@sha256:77bf803cfc45670f8bf7a5c08fce7de247c7c40f10c43f2c1ae51458d294bfc4
      version: 4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest
    history:
    - completionTime: "2022-04-01T09:15:42Z"
      image: registry.build01.ci.openshift.org/ci-ln-2ydg80t/release@sha256:77bf803cfc45670f8bf7a5c08fce7de247c7c40f10c43f2c1ae51458d294bfc4
      startedTime: "2022-04-01T08:50:04Z"
      state: Completed
      verified: false
      version: 4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest
    observedGeneration: 2
    versionHash: Q_qcbW4ZuAs=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The ImplicitlyEnabledCaps is always nil.

$ grep ImplicitlyEnabledCaps cvo.log
I0401 09:27:12.640861       1 status.go:171] Synchronizing status errs=field.ErrorList(nil) status=&cvo.SyncWorkerStatus{Generation:3, Failure:error(nil), Done:570, Total:784, Completed:0, Reconciling:true, Initial:false, VersionHash:"Q_qcbW4ZuAs=", LastProgress:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Actual:v1.Release{Version:"4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest", Image:"registry.build01.ci.openshift.org/ci-ln-2ydg80t/release@sha256:77bf803cfc45670f8bf7a5c08fce7de247c7c40f10c43f2c1ae51458d294bfc4", URL:"", Channels:[]string(nil)}, Verified:false, loadPayloadStatus:cvo.LoadPayloadStatus{Step:"PayloadLoaded", Message:"Payload loaded version=\"4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest\" image=\"registry.build01.ci.openshift.org/ci-ln-2ydg80t/release@sha256:77bf803cfc45670f8bf7a5c08fce7de247c7c40f10c43f2c1ae51458d294bfc4\"", Failure:error(nil), Release:v1.Release{Version:"4.11.0-0.ci.test-2022-04-01-025354-ci-ln-2ydg80t-latest", Image:"registry.build01.ci.openshift.org/ci-ln-2ydg80t/release@sha256:77bf803cfc45670f8bf7a5c08fce7de247c7c40f10c43f2c1ae51458d294bfc4", URL:"", Channels:[]string(nil)}, Verified:false, LastTransitionTime:time.Time{wall:0xc089ce7ce6114037, ext:976922616390, loc:(*time.Location)(0x2c58160)}}, CapabilitiesStatus:cvo.CapabilityStatus{Status:v1.ClusterVersionCapabilitiesStatus{EnabledCapabilities:[]v1.ClusterVersionCapability{"marketplace"}, KnownCapabilities:[]v1.ClusterVersionCapability{"baremetal", "marketplace", "openshift-samples"}}, ImplicitlyEnabledCaps:[]v1.ClusterVersionCapability(nil)}}


Version-Release number of selected component (if applicable):
The payload was built from cvo#754 by using cluster-bot
build openshift/cluster-version-operator#754

How reproducible:
1/1

Steps to Reproduce:
1.Install a cluster with the cap set:
spec:
    capabilities:
      additionalEnabledCapabilities:
      - marketplace
      baselineCapabilitySet: None

2.Remove additionalEnabledCapabilities:
 spec:
    capabilities:
      baselineCapabilitySet: None


Actual results:
CV condition ImplicitlyEnabledCapabilities doesn’t complain about the change

Expected results:
CV condition ImplicitlyEnabledCapabilities complains about the change


Additional info:

Comment 1 Lalatendu Mohanty 2022-04-01 13:31:21 UTC

*** Bug 2070891 has been marked as a duplicate of this bug. ***

Comment 2 Jack Ottofaro 2022-04-01 14:23:04 UTC

"baselineCapabilitySet: None" overrides additionalEnabledCapabilities, that is the latter is ignored. So in your case you started out with no capabilities enabled in step 1 and therefore step 2 was a no-op.

To test implicitly enabled capabilities due to post-install admin changes (not updates as this is not supported until OTA-574 is Done), initially set "baselineCapabilitySet: v4.11" and then change it to "None".

Comment 3 Jack Ottofaro 2022-04-04 19:06:14 UTC

(In reply to Jack Ottofaro from comment #2)
> "baselineCapabilitySet: None" overrides additionalEnabledCapabilities, that
> is the latter is ignored. So in your case you started out with no
> capabilities enabled in step 1 and therefore step 2 was a no-op.
> 
> To test implicitly enabled capabilities due to post-install admin changes
> (not updates as this is not supported until OTA-574 is Done), initially set
> "baselineCapabilitySet: v4.11" and then change it to "None".

Disregard the above comment. "None" should work as you describe. It's if you had not set "baselineCapabilitySet" to anything that "additionalEnabledCapabilities" are ignored.

A latter change to how we set the implicitly enabled capabilities broke the setting of this in status. This line [1] needs to be moved up since implicitly enabled capability changes do not trigger !capabilitiesEqual so we return before setting the status.

[1] https://github.com/jottofar/cluster-version-operator/blob/16916ca3863d2303aa266d1b7cf004c92b7b320d/pkg/cvo/sync_worker.go#L411

Comment 4 Yang Yang 2022-04-06 13:37:26 UTC

Retried nightly build 4.11.0-0.nightly-2022-04-06-000911 and faced it as well. 

Starting a cluster with baselineCapabilitySet: None and additionalEnabledCapabilities: marketplace and then  remove additionalEnabledCapabilities, then we have ImplicitlyEnabledCapabilities=false.

# oc get clusterversion -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2022-04-06T03:10:43Z"
    generation: 3
    name: version
    resourceVersion: "148562"
    uid: ae0ab57e-235f-4d4b-835e-897679d56795
  spec:
    capabilities:
      baselineCapabilitySet: None
    channel: stable-4.11
    clusterID: 9c704201-9265-488a-9f08-b51eac802a3c
  status:
    availableUpdates: null
    capabilities:
      enabledCapabilities:
      - marketplace
      knownCapabilities:
      - baremetal
      - marketplace
      - openshift-samples
    conditions:
    - lastTransitionTime: "2022-04-06T03:10:46Z"
      message: 'Unable to retrieve available updates: currently reconciling cluster
        version 4.11.0-0.nightly-2022-04-06-000911 not found in the "stable-4.11"
        channel'
      reason: VersionNotFound
      status: "False"
      type: RetrievedUpdates
    - lastTransitionTime: "2022-04-06T03:10:46Z"
      message: Capabilities match configured spec
      reason: AsExpected
      status: "False"
      type: ImplicitlyEnabledCapabilities
    - lastTransitionTime: "2022-04-06T03:10:46Z"
      message: Payload loaded version="4.11.0-0.nightly-2022-04-06-000911" image="registry.ci.openshift.org/ocp/release@sha256:885b2a993c72a438183a8a7f04f6adb405ef6167c8724fb00682093941de2472"
      reason: PayloadLoaded
      status: "True"
      type: ReleaseAccepted
    - lastTransitionTime: "2022-04-06T03:31:19Z"
      message: Done applying 4.11.0-0.nightly-2022-04-06-000911
      status: "True"
      type: Available
    - lastTransitionTime: "2022-04-06T03:21:01Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2022-04-06T03:31:19Z"
      message: Cluster version is 4.11.0-0.nightly-2022-04-06-000911
      status: "False"
      type: Progressing
    desired:
      image: registry.ci.openshift.org/ocp/release@sha256:885b2a993c72a438183a8a7f04f6adb405ef6167c8724fb00682093941de2472
      version: 4.11.0-0.nightly-2022-04-06-000911
    history:
    - completionTime: "2022-04-06T03:31:19Z"
      image: registry.ci.openshift.org/ocp/release@sha256:885b2a993c72a438183a8a7f04f6adb405ef6167c8724fb00682093941de2472
      startedTime: "2022-04-06T03:10:46Z"
      state: Completed
      verified: false
      version: 4.11.0-0.nightly-2022-04-06-000911
    observedGeneration: 3
    versionHash: KK4BRYwcro0=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 5 Jack Ottofaro 2022-04-06 18:03:42 UTC

(In reply to Yang Yang from comment #4)
> Retried nightly build 4.11.0-0.nightly-2022-04-06-000911 and faced it as
> well. 
> 

Try it again once https://github.com/openshift/cluster-version-operator/pull/761 merges

Comment 6 W. Trevor King 2022-04-06 19:37:20 UTC

Fix is just about to land, but this is peripheral enough that if for some reason we can't fix this by 4.11 GA, I expect we'd still ship.  Setting blocker-

Comment 8 Yang Yang 2022-04-07 07:52:07 UTC

Verifying with 4.11.0-0.nightly-2022-04-07-053433

Steps to verify:
1) Install a cluster with below cap set
 spec:
    capabilities:
      additionalEnabledCapabilities:
      - marketplace
      - baremetal
      - openshift-samples
      baselineCapabilitySet: None

2) Check the marketplace, baremetal and openshift-samples are installed
# oc get co | grep 'marketplace\|samples\|baremetal'
baremetal                                  4.11.0-0.nightly-2022-04-07-053433   True        False         False      84m     
marketplace                                4.11.0-0.nightly-2022-04-07-053433   True        False         False      84m     
openshift-samples                          4.11.0-0.nightly-2022-04-07-053433   True        False         False      81m     

3) Remove openshift-samples
# oc edit clusterversion
clusterversion.config.openshift.io/version edited

4) Check ImplicitlyEnabledCapabilities=true and tells us openshift-samples cannot be disabled
# oc get clusterversion -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2022-04-07T06:14:54Z"
    generation: 3
    name: version
    resourceVersion: "52260"
    uid: 90484f76-4337-4ff6-ba43-18a3d508420b
  spec:
    capabilities:
      additionalEnabledCapabilities:
      - marketplace
      - baremetal
      baselineCapabilitySet: None
    channel: stable-4.11
    clusterID: 9f6ac256-dbac-4caf-ad23-1db1e121a74d
  status:
    availableUpdates: null
    capabilities:
      enabledCapabilities:
      - baremetal
      - marketplace
      - openshift-samples
      knownCapabilities:
      - baremetal
      - marketplace
      - openshift-samples
    conditions:
    - lastTransitionTime: "2022-04-07T06:14:56Z"
      message: 'Unable to retrieve available updates: currently reconciling cluster
        version 4.11.0-0.nightly-2022-04-07-053433 not found in the "stable-4.11"
        channel'
      reason: VersionNotFound
      status: "False"
      type: RetrievedUpdates
    - lastTransitionTime: "2022-04-07T07:46:23Z"
      message: 'The following capabilities could not be disabled: openshift-samples'
      reason: CapabilitiesImplicitlyEnabled
      status: "True"
      type: ImplicitlyEnabledCapabilities
    - lastTransitionTime: "2022-04-07T06:14:56Z"
      message: Payload loaded version="4.11.0-0.nightly-2022-04-07-053433" image="registry.ci.openshift.org/ocp/release@sha256:8f4ac72e32e701f6609fe00c4f021670cab8466eb38b98c141764ceb6c3d8ab5"
      reason: PayloadLoaded
      status: "True"
      type: ReleaseAccepted
    - lastTransitionTime: "2022-04-07T06:35:55Z"
      message: Done applying 4.11.0-0.nightly-2022-04-07-053433
      status: "True"
      type: Available
    - lastTransitionTime: "2022-04-07T06:35:40Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2022-04-07T06:35:55Z"
      message: Cluster version is 4.11.0-0.nightly-2022-04-07-053433
      status: "False"
      type: Progressing
    desired:
      image: registry.ci.openshift.org/ocp/release@sha256:8f4ac72e32e701f6609fe00c4f021670cab8466eb38b98c141764ceb6c3d8ab5
      version: 4.11.0-0.nightly-2022-04-07-053433
    history:
    - completionTime: "2022-04-07T06:35:55Z"
      image: registry.ci.openshift.org/ocp/release@sha256:8f4ac72e32e701f6609fe00c4f021670cab8466eb38b98c141764ceb6c3d8ab5
      startedTime: "2022-04-07T06:14:56Z"
      state: Completed
      verified: false
      version: 4.11.0-0.nightly-2022-04-07-053433
    observedGeneration: 2
    versionHash: T1J7iQHnTDU=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

# oc get co | grep 'marketplace\|samples\|baremetal'
baremetal                                  4.11.0-0.nightly-2022-04-07-053433   True        False         False      85m     
marketplace                                4.11.0-0.nightly-2022-04-07-053433   True        False         False      85m     
openshift-samples                          4.11.0-0.nightly-2022-04-07-053433   True        False         False      82m     

5) Remove baremetal
# oc edit clusterversion
clusterversion.config.openshift.io/version edited

6) Check ImplicitlyEnabledCapabilities=true and tells us baremetal cannot be disabled
# oc get clusterversion -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2022-04-07T06:14:54Z"
    generation: 4
    name: version
    resourceVersion: "52543"
    uid: 90484f76-4337-4ff6-ba43-18a3d508420b
  spec:
    capabilities:
      additionalEnabledCapabilities:
      - marketplace
      baselineCapabilitySet: None
    channel: stable-4.11
    clusterID: 9f6ac256-dbac-4caf-ad23-1db1e121a74d
  status:
    availableUpdates: null
    capabilities:
      enabledCapabilities:
      - baremetal
      - marketplace
      - openshift-samples
      knownCapabilities:
      - baremetal
      - marketplace
      - openshift-samples
    conditions:
    - lastTransitionTime: "2022-04-07T06:14:56Z"
      message: 'Unable to retrieve available updates: currently reconciling cluster
        version 4.11.0-0.nightly-2022-04-07-053433 not found in the "stable-4.11"
        channel'
      reason: VersionNotFound
      status: "False"
      type: RetrievedUpdates
    - lastTransitionTime: "2022-04-07T07:46:23Z"
      message: 'The following capabilities could not be disabled: baremetal, openshift-samples'
      reason: CapabilitiesImplicitlyEnabled
      status: "True"
      type: ImplicitlyEnabledCapabilities
    - lastTransitionTime: "2022-04-07T06:14:56Z"
      message: Payload loaded version="4.11.0-0.nightly-2022-04-07-053433" image="registry.ci.openshift.org/ocp/release@sha256:8f4ac72e32e701f6609fe00c4f021670cab8466eb38b98c141764ceb6c3d8ab5"
      reason: PayloadLoaded
      status: "True"
      type: ReleaseAccepted
    - lastTransitionTime: "2022-04-07T06:35:55Z"
      message: Done applying 4.11.0-0.nightly-2022-04-07-053433
      status: "True"
      type: Available
    - lastTransitionTime: "2022-04-07T06:35:40Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2022-04-07T06:35:55Z"
      message: Cluster version is 4.11.0-0.nightly-2022-04-07-053433
      status: "False"
      type: Progressing
    desired:
      image: registry.ci.openshift.org/ocp/release@sha256:8f4ac72e32e701f6609fe00c4f021670cab8466eb38b98c141764ceb6c3d8ab5
      version: 4.11.0-0.nightly-2022-04-07-053433
    history:
    - completionTime: "2022-04-07T06:35:55Z"
      image: registry.ci.openshift.org/ocp/release@sha256:8f4ac72e32e701f6609fe00c4f021670cab8466eb38b98c141764ceb6c3d8ab5
      startedTime: "2022-04-07T06:14:56Z"
      state: Completed
      verified: false
      version: 4.11.0-0.nightly-2022-04-07-053433
    observedGeneration: 2
    versionHash: T1J7iQHnTDU=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

# oc get co | grep 'marketplace\|samples\|baremetal'
baremetal                                  4.11.0-0.nightly-2022-04-07-053433   True        False         False      86m     
marketplace                                4.11.0-0.nightly-2022-04-07-053433   True        False         False      86m     
openshift-samples                          4.11.0-0.nightly-2022-04-07-053433   True        False         False      83m 

Looks good to me. Moving to verified state.

Comment 11 errata-xmlrpc 2022-08-10 11:03:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069

Note You need to log in before you can comment on or make changes to this bug.