Red Hat Bugzilla – Bug 1447928
Observed a panic: Pop() of key not in store:
Last modified: 2017-08-16 15:51 EDT
Description of problem: Hello, after errata https://access.redhat.com/errata/RHBA-2017:1129 customer upgraded to 3.4.1.18 router image. However, he is facing similar issue as https://bugzilla.redhat.com/show_bug.cgi?id=1429823 E0504 07:53:54.020124 1 runtime.go:64] Observed a panic: "Pop() of key not in store: namespace/pod_name/d 2399987-309d-11e7-829b-0211d686705d" (Pop() of key not in store: namespace/pod_name/d2399987-309d-11e7-829b-021 1d686705d) /builddir/build/BUILD/atomic-openshift-git-0.0f9d380/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kube rnetes/pkg/util/runtime/runtime.go:70 /builddir/build/BUILD/atomic-openshift-git-0.0f9d380/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kube rnetes/pkg/util/runtime/runtime.go:63 /builddir/build/BUILD/atomic-openshift-git-0.0f9d380/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kube rnetes/pkg/util/runtime/runtime.go:49 /usr/lib/golang/src/runtime/asm_amd64.s:479 /usr/lib/golang/src/runtime/panic.go:458 I will attach full log. Workaround is to delete the pod and automatic recreation of the pod. Version-Release number of selected component (if applicable): OpenShift Container Platform 3.4.1 Router image 3.4.1.18 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
More detail on the duplicate bug: https://bugzilla.redhat.com/show_bug.cgi?id=1437441
*** Bug 1437441 has been marked as a duplicate of this bug. ***
See https://github.com/openshift/origin/pull/14232 for proposed fix.
Hello Phil, thanks you for reply. In which version it will be merged for OSCP ? Thank you
No decision has been made yet. I expect it to ultimately be in 3.5, 3.4, 3.3. There is a new implementation that doesn't use the fixed code base for 3.6 and beyond so it won't be there.
verified this bug on openshift v3.6.94 no find the panic logs when running the following script: #!/bin/bash function _simulate_eq_panic() { sleep 0.0$((RANDOM%3)) echo " - worker name: $1 ... " case "$((RANDOM%3))" in 0) oc create -f "$2" ;; 1) oc replace -f "$2" ;; 2) oc delete -f "$2" ;; *) oc replace -f "$2" ;; esac } # End of function _simulate_eq_panic. # # main(): # routefile=${1:-"https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/unsecure/route_unsecure.json"} ntimes=${2:-50} for i in `seq ${ntimes}`; do _simulate_eq_panic "worker_${i}" "${routefile}" & done _simulate_eq_panic "main" "${routefile}"
*** Bug 1458587 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716