Bug 1582818

Summary: HAProxy router causes panic due to index out of range in cmux library
Product: OpenShift Container Platform Reporter: Kenjiro Nakayama <knakayam>
Component: NetworkingAssignee: Ben Bennett <bbennett>
Status: CLOSED DUPLICATE QA Contact: Meng Bo <bmeng>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.7.0CC: aos-bugs, bperkins, hongli
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-29 14:14:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
oc get dc router -o yaml none

Description Kenjiro Nakayama 2018-05-27 04:48:34 UTC
Description of problem:

- After installed OCP and deployed haproxy router v3.7, HAProxy router causes panic and restarting.

Version-Release number of selected component (if applicable):
- atomic-openshift-3.7.44-2.git.0.6b061d4.el7.x86_64
- w/ HAProxy image v3.7.44 and v3.7.46 also happened.

  image: "openshift3/ose-haproxy-router:v3.7.44",
  image: openshift3/ose-haproxy-router:v3.7.46

How reproducible:
- 100% on only Customer's env

Steps to Reproduce:
1. Install OpenShift v3.7
2. Wait for some hours or days

Actual results:
- Start panicking with following errors:

  # oc logs -p router-1-5k2s6
  I0522 23:52:45.621933       1 template.go:254] Starting template router (v3.7.46)
  I0522 23:52:45.734512       1 metrics.go:148] Router health and metrics port listening at 0.0.0.0:1936
  I0522 23:52:47.311747       1 router.go:441] Router reloaded:
   - Checking http://localhost:80 ...
   - Health check ok : 0 retry attempt(s).
  I0522 23:52:47.311877       1 router.go:228] Router is including routes in all namespaces
  I0522 23:52:47.679454       1 router.go:441] Router reloaded:
   - Checking http://localhost:80 ...
   - Health check ok : 0 retry attempt(s).
  I0522 23:52:52.692311       1 router.go:441] Router reloaded:
   - Checking http://localhost:80 ...
   - Health check ok : 0 retry attempt(s).
  W0523 00:00:49.389745       1 reflector.go:343] github.com/openshift/origin/pkg/router/controller/factory/factory.go:108: watch of *route.Route ended with: etcdserver: mvcc: required revision has been compacted
  panic: runtime error: index out of range
  goroutine 1633 [running]:
  github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*ptNode).match(0xc4211c5c20, 0xc4211bbd98, 0x0, 0x8, 0x1, 0x0)
          /builddir/build/BUILD/atomic-openshift-git-0.e81594b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/patricia.go:148 +0x197
  github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*patriciaTree).matchPrefix(0xc4211cb9c0, 0xf2f2600, 0xc420e7e118, 0xf30b700)
          /builddir/build/BUILD/atomic-openshift-git-0.e81594b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/patricia.go:38 +0x90
  github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*patriciaTree).(github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.matchPrefix)-fm(0xf2f2600, 0xc420e7e118, 0xc42095e398)
          /builddir/build/BUILD/atomic-openshift-git-0.e81594b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/matchers.go:23 +0x3e
  github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*cMux).serve(0xc4211a9ec0, 0xf365a80, 0xc42095e398, 0xc42119dce0, 0xc42103baa0)
          /builddir/build/BUILD/atomic-openshift-git-0.e81594b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/cmux.go:129 +0x265
  created by github.com/openshift/origin/vendor/github.com/cockroachdb/cmux.(*cMux).Serve
          /builddir/build/BUILD/atomic-openshift-git-0.e81594b/_output/local/go/src/github.com/openshift/origin/vendor/github.com/cockroachdb/cmux/cmux.go:119 +0x16c

Expected results:
- No panic

Additional info:
- Attached installtion steps and DC of router

Comment 2 Kenjiro Nakayama 2018-05-27 04:50:29 UTC
Created attachment 1442188 [details]
oc get dc router  -o yaml

Comment 4 Ben Bennett 2018-05-29 14:14:45 UTC

*** This bug has been marked as a duplicate of bug 1532060 ***