Bug 1888827

Summary: ovnkube-master may segfault when trying to add IPs to a nil address set
Product: OpenShift Container Platform Reporter: Tim Rozet <trozet>
Component: NetworkingAssignee: Tim Rozet <trozet>
Networking sub component: ovn-kubernetes QA Contact: Ross Brattain <rbrattai>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bbennett, rbrattai
Version: 4.6   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1889460 (view as bug list) Environment:
Last Closed: 2021-02-24 15:26:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1899746    
Bug Blocks: 1889460    

Description Tim Rozet 2020-10-15 20:35:24 UTC 
Description of problem:
In 1887585 there is a trace back for
2020-10-14T16:28:07.226416950Z panic: runtime error: invalid memory address or nil pointer dereference
2020-10-14T16:28:07.226416950Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x156dc41]
2020-10-14T16:28:07.226416950Z 
2020-10-14T16:28:07.226416950Z goroutine 262020-10-14T16:28:07.226442933Z  [running]:
2020-10-14T16:28:07.226442933Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*ovnAddressSets).AddIPs(0x0, 0xc0643075a0, 0x1, 0x1, 0x0, 0x0)

This can happen if we were unable to find/create the address set in OVN because we return nil and assign that to the nsinfo struct as the address set. This is likely to happen when OVN NB is under extreme load and unresponsive:

'ErrorAddingLogicalPort' failed to add IP "10.188.0.106" to address set "4cb995b8-07d8-406c-a563-a9766e0f2745/cluster-density-242_v4/a1144416766333236008", stderr: "2020-10-14T16:22:38Z|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)\n" (OVN command '/usr/bin/ovn-nbctl --timeout=15 add address_set 4cb995b8-07d8-406c-a563-a9766e0f2745 addresses "10.188.0.106"' failed: signal: alarm clock)

This will cause ovn-kubemaster to eventually crash and restart. But reconciliation logic should handle ensuring the address set is properly created on restart.
Comment 2 Anurag saxena 2020-11-17 15:18:05 UTC 
@rbrattai Can you help verifying it?
Comment 4 Ross Brattain 2020-11-24 03:28:31 UTC 
Unable to reproduce segfault on 4.7.0-0.nightly-2020-11-22-204912

Verified
Comment 7 errata-xmlrpc 2021-02-24 15:26:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633