+++ This bug was initially created as a clone of Bug #1939060 +++
Description of problem:
While upgrading ovn-kubernetes via CNO, I can see workers and masters both getting upgraded simultaneously. The method I used is:
- op: add
- op: add
# sets network operator deployment unmanaged from clusterversion
oc patch --type=json -p "$(cat ~/override-cno-control-patch.yaml)" clusterversion version
- name: network-operator
- name: OVN_IMAGE
# build from images/Dockerfile.bf and https://github.com/ovn-org/ovn-kubernetes/pull/2005
# overrides ovn-kubernetes image
oc patch -p "$(cat ~/override-ovn-kubernetes-image-patch.yaml)" deploy network-operator -n openshift-network-operator
--- Additional comment from trozet on 2021-03-15 14:17:17 UTC ---
Note I saw this behavior on 4.8, but I assume the behavior also exists in 4.7 so targeting that version.
--- Additional comment from trozet on 2021-03-15 14:18:27 UTC ---
Created attachment 1763387 [details]
--- Additional comment from fpaoline on 2021-03-17 17:12:28 UTC ---
Update: I suspect this is because this is not a real version upgrade, but you changed the image only. The code path is here
upgrade from 4.6.23 to 4.7.0-0.nightly-2021-03-27-082615 successfully with OVN plugin.
Move this to 'verified'
We're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the UpgradeBlocker flag has been added to this bug. It will be removed if the assessment indicates that this should not block upgrade edges. The expectation is that the assignee answers these questions.
Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking?
example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet
example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time
What is the impact? Is it serious enough to warrant blocking edges?
example: Up to 2 minute disruption in edge routing
example: Up to 90seconds of API downtime
example: etcd loses quorum and you have to restore from backup
How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)?
example: Issue resolves itself after five minutes
example: Admin uses oc to fix things
example: Admin must SSH to hosts, restore from backups, or other non standard admin activities
Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)?
example: No, it’s always been like this we just never noticed
example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1
(In reply to Lalatendu Mohanty from comment #4)
> We're asking the following questions to evaluate whether or not this bug
> warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The
> ultimate goal is to avoid delivering an update which introduces new risk or
> reduces cluster functionality in any way. Sample answers are provided to
> give more context and the UpgradeBlocker flag has been added to this bug. It
> will be removed if the assessment indicates that this should not block
> upgrade edges. The expectation is that the assignee answers these questions.
> Who is impacted? If we have to block upgrade edges based on this issue,
> which edges would need blocking?
This fixes an issue upgrading from 4.6 -> 4.7 for customers using the ovn-kubernetes network plugin (which is not the default plugin yet).
4.7 -> 4.7 is *NOT* problematic.
> What is the impact? Is it serious enough to warrant blocking edges?
Nodes whose ovnkube-node pod has not yet been upgraded after the ovnkube-master pod *has* been upgraded may experience connection failures to Services, especially those that have HostNetwork endpoints.
> How involved is remediation (even moderately serious impacts might be
> acceptable if they are easy to mitigate)?
The problem will resolve itself after all ovnkube-node pods are upgraded; but the interruption during upgrade may cause disruption the cluster, or the might cause the upgrade to fail if non-host-network pods are scheduled on non-upgraded nodes.
> standard admin activities
> Is this a regression (if all previous versions were also vulnerable,
> updating to the new, vulnerable version does not increase exposure)?
4.6 -> 4.7 is affected. 4.7 -> 4.7 is *not* affected. Given that we have always upgraded masters before nodes in ovnkube, we just got lucky before. So I guess technically a regression from the user point of view.
Based on comment 6, 4.7 -> 4.7 is not impacted. And 4.6 -> 4.7 is blocked already on bug 1935539 and the vSphere HW thing. This bugfix is beating the fix for the bug 1935539 series into 4.7.z. So we don't have to decide if we consider this an update-blocker, because the edges it impacts are already blocked. Dropping UpgradeBlocker.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.7.5 security and bug fix update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.