Bug 1889460 - [4.6] ovnkube-master may segfault when trying to add IPs to a nil address set
Summary: [4.6] ovnkube-master may segfault when trying to add IPs to a nil address set
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.z
Assignee: Tim Rozet
QA Contact: Ross Brattain
URL:
Whiteboard:
Depends On: 1888827
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-19 17:52 UTC by Tim Rozet
Modified: 2021-02-22 13:54 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1888827
Environment:
Last Closed: 2021-02-22 13:54:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 353 0 None closed Bug 1889460: Guard against nil address set usage 2021-02-12 03:09:54 UTC
Red Hat Product Errata RHBA-2021:0510 0 None None None 2021-02-22 13:54:49 UTC

Description Tim Rozet 2020-10-19 17:52:10 UTC
+++ This bug was initially created as a clone of Bug #1888827 +++

Description of problem:
In 1887585 there is a trace back for
2020-10-14T16:28:07.226416950Z panic: runtime error: invalid memory address or nil pointer dereference
2020-10-14T16:28:07.226416950Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x156dc41]
2020-10-14T16:28:07.226416950Z 
2020-10-14T16:28:07.226416950Z goroutine 262020-10-14T16:28:07.226442933Z  [running]:
2020-10-14T16:28:07.226442933Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*ovnAddressSets).AddIPs(0x0, 0xc0643075a0, 0x1, 0x1, 0x0, 0x0)

This can happen if we were unable to find/create the address set in OVN because we return nil and assign that to the nsinfo struct as the address set. This is likely to happen when OVN NB is under extreme load and unresponsive:

'ErrorAddingLogicalPort' failed to add IP "10.188.0.106" to address set "4cb995b8-07d8-406c-a563-a9766e0f2745/cluster-density-242_v4/a1144416766333236008", stderr: "2020-10-14T16:22:38Z|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)\n" (OVN command '/usr/bin/ovn-nbctl --timeout=15 add address_set 4cb995b8-07d8-406c-a563-a9766e0f2745 addresses "10.188.0.106"' failed: signal: alarm clock)

This will cause ovn-kubemaster to eventually crash and restart. But reconciliation logic should handle ensuring the address set is properly created on restart.

Comment 2 Ross Brattain 2021-02-12 03:11:18 UTC
Unable to reproduce panic on 4.6.0-0.nightly-2021-02-11-155630

Verified

Comment 5 errata-xmlrpc 2021-02-22 13:54:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.18 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0510


Note You need to log in before you can comment on or make changes to this bug.