Bug 1371826 - panic error found in haproxy router log
Summary: panic error found in haproxy router log
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Ben Bennett
QA Contact: zhaozhanqi
URL:
Whiteboard:
: 1387717 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-31 08:17 UTC by Meng Bo
Modified: 2017-03-08 18:26 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Editing a route then deleting it and re-creating it led to a panic. The deletion code led to a different, unexepected state, with an empty array after an edit was made. Consequence: The router panics and crashed. Fix: Harden the code to a) not result in that state and b) tolerate the state should it accidentally occur. Result: A more robust router.
Clone Of:
Environment:
Last Closed: 2016-09-27 09:47:02 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC
Origin (Github) 10747 None None None 2016-08-31 19:58:55 UTC

Description Meng Bo 2016-08-31 08:17:53 UTC
Description of problem:
There are some panic error in haproxy router log after add path to the existing route and delete the route then recreate it.


Version-Release number of selected component (if applicable):
openshift v3.3.0.27
ose-haproxy-router:v3.3.0.27  16a98b46907b

How reproducible:
always

Steps to Reproduce:
1. Create router in the cluster and set the loglevel to 4
    spec:
      containers:
      - command:
        - /usr/bin/openshift-router
        - --loglevel=4

2. Create pod/svc/route with user
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/unsecure/list_for_unsecure.json
3. Edit the route created above and add a path field to it
$ oc edit route
...
spec:
  host: unsecure.example.com
  path: /test
  to:
    kind: Service
    name: service-unsecure
    weight: 100
4. Delete the route after edit
5. Recreate the same route
6. Check the router pod log 
7. Check the route in the haproxy.config

Actual results:
6. Some panic error found in the log:
I0831 03:40:16.742650       1 controller.go:97] Processing Route: route -> service-unsecure
I0831 03:40:16.742674       1 controller.go:98]            Alias: unsecure.example.com
I0831 03:40:16.742679       1 controller.go:99]            Event: ADDED
E0831 03:40:16.742799       1 runtime.go:52] Recovered from panic: "index out of range" (runtime error: index out of range)
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:58
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:51
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:41
/usr/lib/golang/src/runtime/asm_amd64.s:472
/usr/lib/golang/src/runtime/panic.go:443
/usr/lib/golang/src/runtime/panic.go:27
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/pkg/router/controller/unique_host.go:109
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/pkg/router/controller/controller.go:101
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/pkg/router/controller/controller.go:52
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:88
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:89
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:49
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:41
/usr/lib/golang/src/runtime/asm_amd64.s:1998

7. The route was not added to the haproxy router config

Expected results:
Should not have such panic error.

Additional info:

Comment 1 Troy Dawson 2016-09-02 19:36:55 UTC
This has been merged into ose and is in OSE v3.3.0.29 or newer.

Comment 3 Meng Bo 2016-09-05 09:49:20 UTC
Checked on OCP build v3.3.0.29 with router image v3.3.0.29.

Issue has been fixed.

Comment 5 errata-xmlrpc 2016-09-27 09:47:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933

Comment 6 Ben Bennett 2016-10-28 17:44:02 UTC
*** Bug 1387717 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.