Description of problem: During upgrading, found the progress reported in clusterversion is not increasing. [root@preserved-bind-and-bastion ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-19-220030 True True 8m51s Working towards 4.0.0-0.nightly-2019-03-19-220030: 33% complete [root@preserved-bind-and-bastion ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-19-220030 True True 12m Working towards 4.0.0-0.nightly-2019-03-19-220030: 46% complete [root@preserved-bind-and-bastion ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-19-220030 True True 15m Working towards 4.0.0-0.nightly-2019-03-19-220030: 2% complete [root@preserved-bind-and-bastion ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-19-220030 True True 16m Working towards 4.0.0-0.nightly-2019-03-19-220030: 2% complete [root@preserved-bind-and-bastion ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-19-220030 True True 16m Working towards 4.0.0-0.nightly-2019-03-19-220030: 2% complete Version-Release number of the following components: Upgrade from 4.0.0-0.nightly-2019-03-19-004004 to 4.0.0-0.nightly-2019-03-19-220030 How reproducible: Always Steps to Reproduce: 1. Install version 4.0.0-0.nightly-2019-03-19-004004 2. Upgrade it to 4.0.0-0.nightly-2019-03-19-220030 3. Check progress percentage during upgrading Actual results: progress percentage is not increasing Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
Recheck this and this means there are 2 round upgrade happen here? so the percentage is re-calculated? version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Working towards 4.0.0-0.nightly-2019-03-22-002648: 33% complete NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 17m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 18m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 19m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 20m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 21m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 21m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 21m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out The connection to the server api.wjiang-ocp.qe.devcluster.openshift.com:6443 was refused - did you specify the right host or port? NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 21m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out Error from server (NotFound): Unable to list {"config.openshift.io" "v1" "clusterversions"}: the server could not find the requested resource (get clusterversions.config.openshift.io) NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 21m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 21m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 22m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 23m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 24m Unable to apply 4.0.0-0.nightly-2019-03-22-002648: the cluster operator machine-config has not yet successfully rolled out NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 24m Working towards 4.0.0-0.nightly-2019-03-22-002648: 2% complete NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-03-22-002648 True True 24m Working towards 4.0.0-0.nightly-2019-03-22-002648: 2% complete
This is a cluster-version-operator issue, not an installer issue, so I'm reassigning to the Upgrade component.
Colin points out that OS upgrades [1] can lead to node restarts, which could force cluster-version operator moves (as the node it was running on is drained for the upgrade), which will result in a new sync cycle (as the new CVO pod comes up somewhere else). So if we want something closer to monotonic progress reports, we'd need each CVO sync cycle to take a look at the existing object state and silently fast-forward through tasks which were already in sync. [1]: https://github.com/openshift/machine-config-operator/blob/2b28eb287e5bc7b654680a6e85f767ff05604371/docs/OSUpgrades.md
*** Bug 1720735 has been marked as a duplicate of this bug. ***
*** Bug 1723540 has been marked as a duplicate of this bug. ***
Bug 1723540 also report two meaningless errors during upgrading: [root@dhcp-41-193 ~]# oc get clusterversion Error from server (NotFound): Unable to list "config.openshift.io/v1, Resource=clusterversions": the server could not find the requested resource (get clusterversions.config.openshift.io) [root@dhcp-41-193 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.2 True True 7m27s Working towards 4.1.3: 67% complete [root@dhcp-41-193 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.2 True True 8m11s Unable to apply 4.1.3: an unknown error has occurred
(In reply to W. Trevor King from comment #3) > Colin points out that OS upgrades [1] can lead to node restarts, which could > force cluster-version operator moves (as the node it was running on is > drained for the upgrade), which will result in a new sync cycle (as the new > CVO pod comes up somewhere else). So if we want something closer to > monotonic progress reports, we'd need each CVO sync cycle to take a look at > the existing object state and silently fast-forward through tasks which were > already in sync. > > [1]: > https://github.com/openshift/machine-config-operator/blob/ > 2b28eb287e5bc7b654680a6e85f767ff05604371/docs/OSUpgrades.md CVO progress upgrade progress percent is informational and it is eventually increasing. And we have no plans to provide the never resetting gurantees.
I'm hoping to address the % resets as part of a general reroll of % handling in bug 1768255.