Bug 1683491

Stale?  I see no dups (or any nightlies at all?) now, even when I supply a nightly current version:

$ curl -sH 'Accept: application/json' 'https://api.openshift.com/api/upgrades_info/v1/graph?channel=does-not-exist&version=4.0.0-0.nightly-2019-02-20-194410' | jq -r '.nodes[].version'
4.0.0-5
4.0.0-4
4.0.0-6
4.0.0-7
4.0.0-8
4.0.0-9
4.0.0-0.1
4.0.0-0.okd-0
4.0.0-0.2
4.0.0-0.3
4.0.0-0.4
4.0.0-0.5
4.0.0-0.6
4.0.0-0.7

(In reply to W. Trevor King from comment #1)
> Stale?  I see no dups (or any nightlies at all?) now, even when I supply a
> nightly current version:
> 
> $ curl -sH 'Accept: application/json'
> 'https://api.openshift.com/api/upgrades_info/v1/graph?channel=does-not-
> exist&version=4.0.0-0.nightly-2019-02-20-194410' | jq -r '.nodes[].version'
> 4.0.0-5
> 4.0.0-4
> 4.0.0-6
> 4.0.0-7
> 4.0.0-8
> 4.0.0-9
> 4.0.0-0.1
> 4.0.0-0.okd-0
> 4.0.0-0.2
> 4.0.0-0.3
> 4.0.0-0.4
> 4.0.0-0.5
> 4.0.0-0.6
> 4.0.0-0.7

Maybe you should use https://openshift-release.svc.ci.openshift.org/graph address for nightly build, and https://api.openshift.com/api/upgrades_info/v1/graph address for release build.

And there are different response between my try against 4.0.0-0.nightly-2019-02-27-154505 on Monday and my try against 4.0.0-0.nightly-2019-03-04-180718 yestoday .

# curl --silent --header 'Accept:application/json' https://openshift-release.svc.ci.openshift.org/graph | jq '. as $graph | $graph.nodes | map(.version == "4.0.0-0.nightly-2019-02-27-154505") | index(true) as $orig | $graph.edges | map(select(.[0] == $orig)[1]) | map($graph.nodes[.])'
[
  {
    "version": "4.0.0-0.nightly-2019-02-27-172930",
    "payload": "registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-02-27-172930"
  },
  {
    "version": "4.0.0-0.nightly-2019-02-27-190649",
    "payload": "registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-02-27-190649"
  },
  {
    "version": "4.0.0-0.nightly-2019-02-27-172930",
    "payload": "registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-02-27-172930"
  },
  {
    "version": "4.0.0-0.nightly-2019-02-27-190649",
    "payload": "registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-02-27-190649"
  }
]

# curl --silent --header 'Accept:application/json' https://openshift-release.svc.ci.openshift.org/graph | jq '. as $graph | $graph.nodes | map(.version == "4.0.0-0.nightly-2019-03-04-180718") | index(true) as $orig | $graph.edges | map(select(.[0] == $orig)[1]) | map($graph.nodes[.])'
[
  {
    "version": "4.0.0-0.nightly-2019-03-04-234414",
    "payload": "registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-03-04-234414"
  },
  {
    "version": "4.0.0-0.nightly-2019-03-05-034154",
    "payload": "registry.svc.ci.openshift.org/ocp/release:4.0.0-0.nightly-2019-03-05-034154"
  }
]

I wonder if the policy on server end changed this.

https://openshift-release.svc.ci.openshift.org/graph is not a Cincinnati stack. It's just a hack to easily get the nightly images into a graph. Are the duplicate entries problematic?

> Are the duplicate entries problematic?
No big problem. Just no need to give a duplicate available updates since there are not two identical nodes in nightly build list. And it's very confused. So removing the dup one makes it better.

My understanding is that this graph is handed out by the release controller [1] and I don't know what the general direction is with this endpoint. We could either run a Cincinnati instance for this endpoint or incorporate the nightly releases into the instance running at https://api.{,staging.}openshift.com:443/api/upgrades_info/v1/graph. That would effectively result in having two disjoint graphs in the result, which should just work.

[1]: https://github.com/openshift/release-controller/blob/ecc8b22e46952c1585502c28b992559ade39233e/cmd/release-controller/graph.go#L23-L78

Hi Stefan

I tried again but still get duplicate output from the

Hey Liujia, 

no action has been taken to prevent the duplicate from showing up. AFAIU we don't consider this a problem on the release controller's graph endpoint since it's meant for nightlies only. These versions aren't supported in any way.
In Cincinnati this will not happen since there are safety checks [1] which guard against duplicate versions when adding releases to the graph.

I think we can close this ticket since it's not an issue in practice. Any objections to that?

[1]: https://github.com/openshift/cincinnati/blob/f595336f5570fa07609036b4b97342260353a42a/cincinnati/src/lib.rs#L105-L109

That's ok for me. Since https://openshift-release.svc.ci.openshift.org/graph is not considered as a supported graph endpoint, so close the bug.

Do test based rc.7 from quay. Still hit the issue even with a supported graph endpoint(default one), so re-open this bug to keep tracking the issue.

# ./oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-rc.7   True        False         43s     Cluster version is 4.1.0-rc.7

# ./oc get clusterversion -o json|jq ".items[0].spec"
{
  "channel": "prerelease-4.1",
  "clusterID": "bff6de88-7500-4925-b85d-013fcf4649e5",
  "upstream": "https://api.openshift.com/api/upgrades_info/v1/graph"
}

# ./oc adm upgrade
Cluster version is 4.1.0-rc.7

Updates:

VERSION    IMAGE
4.1.0-rc.8 quay.io/openshift-release-dev/ocp-release@sha256:8250bbf79d4f567e24a7b85795aba97f9e75a9df18738891a0cb6ba42e422584
4.1.0-rc.8 quay.io/openshift-release-dev/ocp-release@sha256:8250bbf79d4f567e24a7b85795aba97f9e75a9df18738891a0cb6ba42e422584
4.1.0-rc.9 quay.io/openshift-release-dev/ocp-release@sha256:49c4b6bf70061e522e3525aed534d087c9abfba7c39cbcbdd1bd770ab096bf9e

Thanks for reporting this new occurrence of the double-edge. It has been discussed Slack and the reason is known. The edge has been specified in the release-metadata of the payload as well as on quay via an image tag label.
There wasn't clear consensus if we want the edge removed by Cincinnati, since it would be easier to remove it on quay where it was falsely specified as a duplicate.

> There wasn't clear consensus if we want the edge removed by Cincinnati, since it would be easier to remove it on quay where it was falsely specified as a duplicate.

People tagging in Quay can make mistakes.  I think we want to dedup in Cincinnati and alert the admins maintaining the datasource (Quay labels), to avoid confusing downstream users.  I'm also fine moving the datasource away from Quay labels and guarding against admin mistakes with some sort of preflight-testing of admin changes to the datasource.  But I think we want to move on at least one of those.

> I think we want to dedup in Cincinnati and alert the admins maintaining the datasource (Quay labels), to avoid confusing downstream users.

Ironically the hard part is the alarming here ;-) Who are the admins and how can Cincinnati alert them?

Liujia, we have merged a fix to production yesterday. Can you please verify that the bug is gone?

[root@preserve-jliu-worker 20190711_14830]# ./oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-rc.7   True        False         5m25s   Cluster version is 4.1.0-rc.7

[root@preserve-jliu-worker 20190711_14830]# ./oc get clusterversion -o json|jq ".items[0].spec"
{
  "channel": "prerelease-4.1",
  "clusterID": "fe69dbb3-cc92-4cdc-82c0-439543fff7da",
  "upstream": "https://api.openshift.com/api/upgrades_info/v1/graph"
}

[root@preserve-jliu-worker 20190711_14830]# ./oc adm upgrade
Cluster version is 4.1.0-rc.7

Updates:

VERSION    IMAGE
4.1.0-rc.9 quay.io/openshift-release-dev/ocp-release@sha256:49c4b6bf70061e522e3525aed534d087c9abfba7c39cbcbdd1bd770ab096bf9e
4.1.0-rc.8 quay.io/openshift-release-dev/ocp-release@sha256:8250bbf79d4f567e24a7b85795aba97f9e75a9df18738891a0cb6ba42e422584

Moreover, checked all upgrade paths for both prerelease-4.1 and stable-4.1 channel from Cincinnati stack, no dup path found. Verified the bug.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1809