Bug 1798049
| Summary: | CVO got panic when downgrading to 4.2.10 | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Scott Dodson <sdodson> | |
| Component: | Cluster Version Operator | Assignee: | W. Trevor King <wking> | |
| Status: | CLOSED ERRATA | QA Contact: | Gaoyun Pei <gpei> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 4.3.0 | CC: | aos-bugs, ccoleman, gpei, jokerman, padillon, wking | |
| Target Milestone: | --- | |||
| Target Release: | 4.3.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1783221 | |||
| : | 1800346 (view as bug list) | Environment: | ||
| Last Closed: | 2020-02-25 06:18:00 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1783221 | |||
| Bug Blocks: | 1800346 | |||
|
Comment 3
Gaoyun Pei
2020-02-15 06:29:55 UTC
> Verify this bug using payload 4.3.0-0.nightly-2020-02-14-234906, downgrade to 4.2.19 still failed. On upgrades and downgrades, the important CVO is that for the target release. So for 4.3.0-0.nightly-2020-02-14-234906 -> 4.2.19, you will still hit this failure mode because 4.2.19 does not contain the patch. If you want to independently verify this fix for this 4.3 Bugzilla, you will need to use a whatever -> 4.3-nightly upgrade in which there is a manifest change that removes either a container or a service port that was not the final entry in its list (e.g. see the unit test removing the test-A container [1]). I'm not sure an appropriate source release image exists off the shelf. You could create one by adding additional ports to a service in your target 4.3 nightly. Or you could just verify this bug by saying "we don't see any regressions" and then test the 4.3->4.2 downgrade as part of verifying the 4.2.z bug 1800346. [1]: https://github.com/openshift/cluster-version-operator/pull/282/files#diff-415c13f11ffc32696c5d69b900b3fe58R251-R268 Digging into the manifest change that triggered the initial issue. We don't have 4.3.0-0.nightly-2019-12-12-155629 around anymore, but we do have the temporally close 4.3.0-0.nightly-2019-12-13-072740. Comparing between the 4.3 nightly and 4.2.10:
$ oc adm release extract --to 4.2.10 quay.io/openshift-release-dev/ocp-release:4.2.10
$ oc adm release extract --to 4.3.0-0.nightly-2019-12-13-072740 quay.io/openshift-release-dev/ocp-release-nightly:4.3.0-0.nightly-2019-12-13-072740
$ diff -U3 4.2.10/0000_50_cluster-machine-approver_02-deployment.yaml 4.3.0-0.nightly-2019-12-13-072740/0000_50_cluster-machine-approver_04-deployment.yaml
--- 4.2.10/0000_50_cluster-machine-approver_02-deployment.yaml 2019-12-02 22:52:11.000000000 -0800
+++ 4.3.0-0.nightly-2019-12-13-072740/0000_50_cluster-machine-approver_04-deployment.yaml 2019-12-06 16:35:48.000000000 -0800
@@ -21,8 +23,31 @@
hostNetwork: true
serviceAccountName: machine-approver-sa
containers:
+ - args:
+ ...
+ name: kube-rbac-proxy
+ ...
+ name: machine-approver-tls
- name: machine-approver-controller
...
so the issue is that the kube-rbac-proxy container spec (the first entry in that array) is being removed, and subsequent iteration into the machine-approver-controller container spec hits the panic. Unless 4.4 -> 4.3 downgrades were hitting a similar panic already, you'd need to synthesize another change like this (or by adding a Service port) in order to verify this 4.3.z bug in a whatever -> 4.3-nightly upgrade/downgrade.
Thanks very much for the detailed explanation, I must be confused the CVO version when running downgrade. Actually I should already made the 4.4 -> 4.3 downgrade test in https://bugzilla.redhat.com/show_bug.cgi?id=1783221#c7. Since there's no issue of 4.3 CVO, I'll prefer to move this bug to verified and will test the initial problem in BZ#1800346. Thanks. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0528 |