Created attachment 1378189 [details] oc logs -p router-1142-d5s4f registry.reg-aws.openshift.com:443/openshift3/ose-haproxy-router:v3.7.9-1 Found this in us-east-1: $ oc get pod -n default -l router=router NAME READY STATUS RESTARTS AGE router-1142-5zmgr 1/1 Running 34 23d router-1142-8zkgv 1/1 Running 20 23d router-1142-d5s4f 1/1 Running 9 23d router-1142-vkqq7 1/1 Running 8 23d router-1142-xpjhw 1/1 Running 37 23d Looking at all 5 routers `oc get logs -p` I see this at the end of all of them: panic: runtime error: index out of range goroutine 174195 [running]: github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*ptNode).match(0xc420c75dd0, 0xc420c84658, 0x0, 0x8, 0x1, 0x0) /builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/patricia.go:148 +0x197 github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*patriciaTree).matchPrefix(0xc420c7fa00, 0xf287600, 0xc44ea815b8, 0xf2a0700) /builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/patricia.go:38 +0x90 github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*patriciaTree).(github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.matchPrefix)-fm(0xf287600, 0xc44ea815b8, 0xc431896ac8) /builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/matchers.go:23 +0x3e github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*cMux).serve(0xc420c654c0, 0xf2fa960, 0xc431896ac8, 0xc420c82360, 0xc420ff9190) /builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/cmux.go:129 +0x265 created by github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*cMux).Serve /builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/cmux.go:119 +0x16c
This was introduced by: https://github.com/openshift/origin/pull/16975
The bug fix is in the vendor tree. We need to update cockroachdb/cmux to the following commit id: b64f5908f4945f4b11ed4a0a9d3cc1e23350866d (at least) The fix is in patricia tree overflowing on a boundary condition on http 1.1 Fast request match. Either we switch to a slower but more accurate match i.e. not use FastMatch, or we update the repo to include this fix. Glide update PR coming up.
hi,@Eric @Ben @Rajat I'm wondering I still cannot understand how to reproduce this issue. since we did some round of testing in 3.9. but not found this kind of issue. Could you give some clue or steps to reproduce this in order to avoid happen same issue in future. thanks.
I honestly have no idea how to reproduce other than run it in online. I'm ok with QA just verifying the code has changed and we'll see if they continue online. Rajat, what version has the fix?
The master branch has the fix. Anyway the fix was in an upstream package. Its a time-sensitive bug that is difficult to reproduce, so I support comment#5. Closing this bug as 'fixed' upstream.
*** Bug 1552742 has been marked as a duplicate of this bug. ***
*** Bug 1582818 has been marked as a duplicate of this bug. ***