Bug 1868315 - CoreDNS pods observing panic without any provbe failures
Summary: CoreDNS pods observing panic without any provbe failures
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.6.0
Assignee: Stephen Greene
QA Contact: Hongan Li
Depends On:
Blocks: 1869309
TreeView+ depends on / blocked
Reported: 2020-08-12 09:50 UTC by Apoorva Jagtap
Modified: 2023-01-17 17:02 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Running CoreDNS 1.6.6 Consequence: Intermittent `invalid memory address or nil pointer dereference` followed by timeouts in kube API access Fix: Backport CoreDNS#4022 to OCP, which fixes handling errors with Endpoint Tombstones. Result: CoreDNS behaves as intended without intermittent panics.
Clone Of:
Last Closed: 2020-10-27 16:28:06 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift coredns pull 37 0 None closed Bug 1868315: Back port Handle endpoint tombstones (upstream #3887) 2021-02-17 06:24:23 UTC
Red Hat Knowledge Base (Solution) 5175271 0 None None None 2020-08-31 09:24:25 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:28:21 UTC

Description Apoorva Jagtap 2020-08-12 09:50:12 UTC
Description of problem:
Encountering 'Observed a panic' error messages in the logs of CoreDns pods.

Version-Release number of selected component (if applicable):
- RHOCP v4.4.9
- CoreDNS version 1.6.6

How reproducible:
- The scenario is observed in CU's environment momentarily. 
- Not sure whether it could be reproduced or not.

Actual results:
Encountering 'Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)' messages in the logs of some CoreDNS pods, without any liveness/readiness failure, leading to DNS failures in the cluster.

Expected results:
- This bug has been reported in the upstream: https://github.com/coredns/coredns/issues/4022
- And the fix is available for CoreDNS v1.7.0: https://github.com/coredns/coredns/pull/3887

- Would it possible to backport it to CoreDNS v1.6.6?

Comment 7 Hongan Li 2020-08-19 07:42:32 UTC
verified with 4.6.0-0.nightly-2020-08-17-150352	and no issue/panic during the regression test.

Comment 9 errata-xmlrpc 2020-10-27 16:28:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.