Bug 1998423
Summary: | upgrade from 4.8.6 to 4.9.0-0.nightly-2021-08-26-164418, blocked by dns upgrade due to FailedCreatePodSandBox for pods | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | |
Component: | Networking | Assignee: | Alexander Constantinescu <aconstan> | |
Networking sub component: | ovn-kubernetes | QA Contact: | huirwang | |
Status: | CLOSED ERRATA | Docs Contact: | ||
Severity: | high | |||
Priority: | high | CC: | aconstan, anbhat, astoycos, danw, dhellmann, lmohanty, mkennell, scuppett, wking, zzhao | |
Version: | 4.9 | Keywords: | Upgrades | |
Target Milestone: | --- | |||
Target Release: | 4.9.0 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | If docs needed, set a value | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1999894 (view as bug list) | Environment: | ||
Last Closed: | 2021-10-18 17:49:29 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1999895 |
Description
Junqi Zhao
2021-08-27 06:50:01 UTC
omg logs -n openshift-ovn-kubernetes ovnkube-master-h8x2k -c ovnkube-master: > namespace.go:585] Failed to get join switch port IP address for node ip-10-0-164-193.us-east-2.compute.internal: provided IP is already allocated In ovnkube-master, while a nodes cache is syncing. It fails when an attempt is made to reserve logical router port IPs that already exists. This node is the node where pod k8s_dns-default-j7wdt fails to get its IP from its annotation during CNI invocation because OVN is unhealthy for this particular node. This is being worked on upstream to not fail if the IPs that exists is what is expected: See here: https://github.com/ovn-org/ovn-kubernetes/pull/2456 I have yet to determine blocker status - consulting Dan W. Possible fix merged in upstream ovn and downstream cherry pick opened. Waiting on CI to confirm fix. *** Bug 1999894 has been marked as a duplicate of this bug. *** We're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the UpgradeBlocker flag has been added to this bug. It will be removed if the assessment indicates that this should not block upgrade edges. The expectation is that the assignee answers these questions. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time What is the impact? Is it serious enough to warrant blocking edges? example: Up to 2 minute disruption in edge routing example: Up to 90seconds of API downtime example: etcd loses quorum and you have to restore from backup How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? example: Issue resolves itself after five minutes example: Admin uses oc to fix things example: Admin must SSH to hosts, restore from backups, or other non standard admin activities Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? example: No, it’s always been like this we just never noticed example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1 Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? This happens independently of upgrades or regular clusters, but only happens in a very specific scenario that most customers will very unlikely hit. I can't estimate the percentage though, but must be less 10 %. What is the impact? Is it serious enough to warrant blocking edges? One or several nodes can be impacted. The result is, the node is completely hosed (no pod can start, and all existing will lose networking) How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? Restarting the existing ovnkube-master leader instance __should__ fix it. Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? Yes. Only customers upgrading or using this z-stream version with this fix are impacted. The next will include the fix. Expanding on my previous comment:
> Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)?
Only 4.8.10 and 4.7.29 are impacted by the problem. So any customer upgrading to those version, can be affected.
As this issue only occurs when there is change in node count, this is not a typical scenario during upgrades. In general users do not attempt to increase the node count during upgrade. Hence we are removing the upgrade blocker keyword from the bug. *** Bug 1999894 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |