Bug 1692353 - CVO does not report all components failing to upgrade
Summary: CVO does not report all components failing to upgrade
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Abhinav Dahiya
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-25 12:41 UTC by Ben Parees
Modified: 2019-03-27 18:24 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-27 18:24:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ben Parees 2019-03-25 12:41:42 UTC
Description of problem:
The CVO seems to only report the most recent operator it saw failing to upgrade, not all of them.  This is misleading for admins about why the upgrade is blocked.


See:
https://openshift-gce-devel.appspot.com/build/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/259/


https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/259/artifacts/e2e-aws-upgrade/clusteroperators.json

Shows a number of operators that are still reporting the previous version. (    "version": "4.0.0-0.ci-2019-03-18-152932" is the new version, and  "version": "4.0.0-0.ci-2019-03-18-124953" is the old version). 

But:
https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.0/259/artifacts/e2e-aws-upgrade/clusterversion.json

only indicates the openshift controller manager operator is upgrading:

                    {
                        "lastTransitionTime": "2019-03-18T17:32:56Z",
                        "message": "Cluster operator openshift-controller-manager is still updating",
                        "reason": "ClusterOperatorNotAvailable",
                        "status": "True",
                        "type": "Failing"
                    },
                    {
                        "lastTransitionTime": "2019-03-18T16:43:50Z",
                        "message": "Unable to apply 4.0.0-0.ci-2019-03-18-152932: the cluster operator openshift-controller-manager has not yet successfully rolled out",
                        "reason": "ClusterOperatorNotAvailable",
                        "status": "True",
                        "type": "Progressing"
                    },

Comment 1 Abhinav Dahiya 2019-03-27 18:24:00 UTC
CVO performs upgrades in stages to make sure operators like control plane are upgraded correctly before upgrading any other operator. So it reports the the one it is currently upgrading. If the operators are failing after reporting Available for a version, during an upgrade please open bugs for specific operator.


Note You need to log in before you can comment on or make changes to this bug.