Bug 1447928 - Observed a panic: Pop() of key not in store:
Summary: Observed a panic: Pop() of key not in store:
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 3.4.1
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Phil Cameron
QA Contact: zhaozhanqi
: 1437441 1458587 (view as bug list)
Depends On:
Blocks: 1464563 1464567 1477358
TreeView+ depends on / blocked
Reported: 2017-05-04 09:12 UTC by Vladislav Walek
Modified: 2020-12-14 08:36 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: coding error Consequence: Pop() panics and stops router Fix: origin PR 14232 Result:
Clone Of:
: 1464563 1464567 1477358 (view as bug list)
Last Closed: 2017-08-10 05:21:25 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Origin (Github) 14232 0 None None None 2017-05-30 19:06:58 UTC
Red Hat Product Errata RHEA-2017:1716 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 09:02:50 UTC

Description Vladislav Walek 2017-05-04 09:12:28 UTC
Description of problem:

Hello, after errata https://access.redhat.com/errata/RHBA-2017:1129 customer upgraded to router image. However, he is facing similar issue as https://bugzilla.redhat.com/show_bug.cgi?id=1429823

E0504 07:53:54.020124       1 runtime.go:64] Observed a panic: "Pop() of key not in store: namespace/pod_name/d
2399987-309d-11e7-829b-0211d686705d" (Pop() of key not in store: namespace/pod_name/d2399987-309d-11e7-829b-021

I will attach full log.
Workaround is to delete the pod and automatic recreation of the pod.

Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.4.1
Router image

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 2 Ben Bennett 2017-05-10 19:41:03 UTC
More detail on the duplicate bug: https://bugzilla.redhat.com/show_bug.cgi?id=1437441

Comment 3 Ben Bennett 2017-05-10 19:41:45 UTC
*** Bug 1437441 has been marked as a duplicate of this bug. ***

Comment 6 Phil Cameron 2017-05-17 20:44:15 UTC
for proposed fix.

Comment 7 Vladislav Walek 2017-05-23 08:09:03 UTC
Hello Phil,
thanks you for reply. In which version it will be merged for OSCP ? Thank you

Comment 8 Phil Cameron 2017-05-23 12:49:19 UTC
No decision has been made yet. I expect it to ultimately be in 3.5, 3.4, 3.3. There is a new implementation that doesn't use the fixed code base for 3.6 and beyond so it won't be there.

Comment 10 zhaozhanqi 2017-06-05 07:39:21 UTC
verified this bug on openshift v3.6.94

no find the panic logs when running the following script:


function _simulate_eq_panic() {
    sleep 0.0$((RANDOM%3))
    echo "  - worker name: $1 ... "

    case "$((RANDOM%3))" in
      0)  oc create  -f "$2"  ;;
      1)  oc replace -f "$2"  ;;
      2)  oc delete  -f "$2"  ;;
      *)  oc replace -f "$2"  ;;

}  #  End of function  _simulate_eq_panic.

#  main():

for i in `seq ${ntimes}`; do
  _simulate_eq_panic "worker_${i}" "${routefile}" &

_simulate_eq_panic "main" "${routefile}"

Comment 11 Ben Bennett 2017-06-22 15:10:30 UTC
*** Bug 1458587 has been marked as a duplicate of this bug. ***

Comment 13 errata-xmlrpc 2017-08-10 05:21:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.