Bug 1879976 - No other channel visible from console
Summary: No other channel visible from console
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.7.0
Assignee: W. Trevor King
QA Contact: Yang Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-17 13:37 UTC by Rajnikant
Modified: 2021-02-24 15:19 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The cluster-version operator compared available-update and current-target pullspecs exactly when merging Cincinnati metadata like channel membership and errata URI. Consequence: Users installing from or updating to mirrored release images who used the alternative pullspec would not receive Cincinnati metadata. Fix: The CVO now compares releases by digest, regardless of which registry that digested image is hosted in. Result: The CVO now correctly associates Cincinnati metadata such as channel membership, regardless of which repository a digested image is hosted in.
Clone Of:
Environment:
Last Closed: 2021-02-24 15:18:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 490 0 None closed Bug 1879976: pkg/cvo: Compare Cincinnati data by digest when merging metadata 2021-02-18 20:21:49 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:19:03 UTC

Comment 3 Robb Hamilton 2020-09-21 18:57:33 UTC
I am unable to reproduce, but what I suspect was happening is starting in 4.6, the console checks for the channels in `cv.status.desired.channels` (but otherwise falls back to hard-coded values--see https://github.com/openshift/console/blob/master/frontend/public/module/k8s/cluster-settings.ts#L45), and I suspect `cv.status.desired.channels` in 4.6.0-fc.5 did not include `fast` and `stable`, so they did not appear in the dropdown picker (but `candidate` did).  In trying to reproduce, I am not seeing `cv.spec.desired.channels` in 4.6.0-fc.4 or 4.6.0-fc.5, so the console is falling back to the hard-coded values (see https://github.com/openshift/console/blob/master/frontend/public/module/k8s/cluster-settings.ts#L45).  

@Trevor, does this sound plausible to you?

Comment 4 W. Trevor King 2020-09-21 19:36:47 UTC
> ...I suspect `cv.status.desired.channels` in 4.6.0-fc.5 did not include `fast` and `stable`...

Correct.  Feature candidates (fc) and release candidates (rc) are never going to be supported releases, and only supported releases will be promoted into fast/stable channels.

> In trying to reproduce, I am not seeing `cv.spec.desired.channels` in 4.6.0-fc.4 or 4.6.0-fc.5...

Are you installing via their canonical pullspecs?

  $ curl -s https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp-dev-preview/4.6.0-fc.5/release.txt | grep Pull
  Pull From: quay.io/openshift-release-dev/ocp-release@sha256:5883d0db15939484bd477147e6949c53fbc6f551ec20a0f1106b8a3acfb86ef8

I realized last week that if you install from an alternative pullspec, e.g. registry.svc.ci.openshift.org/ocp/release@sha256:5883d0db15939484bd477147e6949c53fbc6f551ec20a0f1106b8a3acfb86ef8 , that the cluster-version operator will not claim to find a matching version from which to extract the current channel list.  Going forward, we will likely pivot to comparing only the digest, for digest-based pullspecs, instead of the full pullspec with irrelevant canonical repository ;).

Comment 5 W. Trevor King 2020-09-21 19:47:21 UTC
'oc get -o yaml clusterversion version' should be sufficient to distinguish between cluster-version side stuff and the consuming console code, if you can attach that.

Comment 6 Robb Hamilton 2020-09-21 19:52:45 UTC
Reassigning to Trevor as this is an issue with the CVO.  He'll either fix this bug or file a new one to cover the issue noted in https://bugzilla.redhat.com/show_bug.cgi?id=1879976#c4.

Comment 7 W. Trevor King 2020-09-21 20:00:07 UTC
I'll wait for ClusterVersion YAML confirming that the issue here is the image comparison discussed in comment 4.  Rajni, can you provide that YAML, per comment 5?

Comment 8 W. Trevor King 2020-10-04 02:22:20 UTC
Still waiting on ClusterVersion YAML requested in comment 7.

Comment 9 W. Trevor King 2020-10-25 15:59:35 UTC
Still waiting on ClusterVersion YAML requested in comment 7.  I'm going to assume we'll never get it, and we'll just move forward with the fix for the comment 4 image comparison issue next sprint.

Comment 10 W. Trevor King 2020-12-04 22:35:51 UTC
I thought I understood what was going on here with comment 4, but turns out we match ourselves by version number in the Cincinnati graph [1], so I'm back to having no idea what was going on here.  Closing INSUFFICIENT_DATA, but please re-open with ClusterVersion YAML if you can reproduce.

[1]: https://github.com/openshift/cluster-version-operator/blob/dc6f4f07d8986272b538b5d6bed9e8b6412b3ce6/pkg/cincinnati/cincinnati.go#L119

Comment 11 W. Trevor King 2020-12-15 00:41:39 UTC
I should have linked code in comment 4, back when I seem to have understood where the pullspec comparison was happening which should have been a digest comparison or version-string comparison ;).  Reproduced today by:

1. Install a 4.6.6 cluster with an alternative pullspec, e.g. via cluster-bot 'launch 4.6.6', which gives a cluster with:

   status:
     desired:
       image: registry.svc.ci.openshift.org/ocp/release@sha256:c7e8f18e8116356701bd23ae3a23fb9892dd5ea66c8300662ef30563d7104f39
       version: 4.6.6

2. Set the channel (this is the usual default, but my cluster-bot-launched cluster clears the default, so I had to manually restore it):

     $ oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "stable-4.6"}]'

3. Waited 30 minutes.  Still getting:

     $ oc get -o json clusterversion version | jq .status.desired.channels
     null

4. Patch to use the canonical 4.6.6 pullspec:

     $ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:c7e8f18e8116356701bd23ae3a23fb9892dd5ea66c8300662ef30563d7104f39

5. Moments later:

     $ oc get -o json clusterversion version | jq .status.desired.channels
     [
       "candidate-4.6",
       "eus-4.6",
       "fast-4.6",
       "stable-4.6"
     ]

I'm hunting for the code I missed in comment 10.

Comment 16 Yang Yang 2020-12-21 13:36:34 UTC
Reproduced with 4.6.0-0.nightly-2020-12-16-201440 by:
1. Install a 4.6 cluster with registry.svc.ci.openshift.org/ocp/release@sha256:faf7c37ca625da7ddf71be3d492a4a245536d600ff99595e6893b1dab026fea1

2. Patched to use below graph
{
  "nodes": [
    {
      "version": "4.6.0-0.nightly-2020-12-16-201440",
      "payload": "quay.io/openshift-release-dev/ocp-release@sha256:faf7c37ca625da7ddf71be3d492a4a245536d600ff99595e6893b1dab026fea1",
      "metadata": {
	"io.openshift.upgrades.graph.release.channels": "channel-a,channel-b"
      }
    },
    {
      "version": "4.7.0-fc.0",
      "payload": "quay.io/openshift-release-dev/ocp-release@sha256:2419f9cd3ea9bd114764855653012e305ade2527210d332bfdd6dbdae538bd66",
      "metadata": {
	"io.openshift.upgrades.graph.release.channels": "channel-a,channel-b"
      }
    }
  ],
  "edges": [
    [0,1]
  ]
}

3. #  oc get -o json clusterversion version | jq .status.desired.channels
      null


Verified with 4.7.0-0.nightly-2020-12-19-192007 by:
1. Install a 4.7 cluster with registry.svc.ci.openshift.org/ocp/release@sha256:3b16e75f9f7b38cd7e02d304ea4982de2c03061b2705851edccfad36b22d4cb4

2. Patched to use below graph 
{
  "nodes": [
    {
      "version": "4.7.0-0.nightly-2020-12-19-192007",
      "payload": "example.com@sha256:3b16e75f9f7b38cd7e02d304ea4982de2c03061b2705851edccfad36b22d4cb4",
      "metadata": {
	"io.openshift.upgrades.graph.release.channels": "channel-a,channel-b"
      }
    },
    {
      "version": "4.7.0-0.nightly-2020-12-20-031435",
      "payload": "registry.svc.ci.openshift.org/ocp/release@sha256:5fbd961fe2fcafd95dbccaa731975ed65b237b7bc461070c6b1f193cd607d37b",
      "metadata": {
	"io.openshift.upgrades.graph.release.channels": "channel-a,channel-b"
      }
    }
  ],
  "edges": [
    [0,1]
  ]
}

3. # oc get -o json clusterversion version | jq .status.desired.channels
[
  "channel-a",
  "channel-b"
]

With the fix, status.desired.channels is able to be populated if digest matches.

Comment 18 errata-xmlrpc 2021-02-24 15:18:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.