Bug 1371826

Summary: panic error found in haproxy router log
Product: OpenShift Container Platform Reporter: Meng Bo <bmeng>
Component: NetworkingAssignee: Ben Bennett <bbennett>
Networking sub component: router QA Contact: zhaozhanqi <zzhao>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, bbennett, bperkins, clichybi, stwalter, tdawson
Version: 3.3.0   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Editing a route then deleting it and re-creating it led to a panic. The deletion code led to a different, unexepected state, with an empty array after an edit was made. Consequence: The router panics and crashed. Fix: Harden the code to a) not result in that state and b) tolerate the state should it accidentally occur. Result: A more robust router.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-27 09:47:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meng Bo 2016-08-31 08:17:53 UTC
Description of problem:
There are some panic error in haproxy router log after add path to the existing route and delete the route then recreate it.


Version-Release number of selected component (if applicable):
openshift v3.3.0.27
ose-haproxy-router:v3.3.0.27  16a98b46907b

How reproducible:
always

Steps to Reproduce:
1. Create router in the cluster and set the loglevel to 4
    spec:
      containers:
      - command:
        - /usr/bin/openshift-router
        - --loglevel=4

2. Create pod/svc/route with user
$ oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/unsecure/list_for_unsecure.json
3. Edit the route created above and add a path field to it
$ oc edit route
...
spec:
  host: unsecure.example.com
  path: /test
  to:
    kind: Service
    name: service-unsecure
    weight: 100
4. Delete the route after edit
5. Recreate the same route
6. Check the router pod log 
7. Check the route in the haproxy.config

Actual results:
6. Some panic error found in the log:
I0831 03:40:16.742650       1 controller.go:97] Processing Route: route -> service-unsecure
I0831 03:40:16.742674       1 controller.go:98]            Alias: unsecure.example.com
I0831 03:40:16.742679       1 controller.go:99]            Event: ADDED
E0831 03:40:16.742799       1 runtime.go:52] Recovered from panic: "index out of range" (runtime error: index out of range)
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:58
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:51
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:41
/usr/lib/golang/src/runtime/asm_amd64.s:472
/usr/lib/golang/src/runtime/panic.go:443
/usr/lib/golang/src/runtime/panic.go:27
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/pkg/router/controller/unique_host.go:109
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/pkg/router/controller/controller.go:101
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/pkg/router/controller/controller.go:52
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:88
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:89
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:49
/builddir/build/BUILD/atomic-openshift-git-0.f1694b3/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:41
/usr/lib/golang/src/runtime/asm_amd64.s:1998

7. The route was not added to the haproxy router config

Expected results:
Should not have such panic error.

Additional info:

Comment 1 Troy Dawson 2016-09-02 19:36:55 UTC
This has been merged into ose and is in OSE v3.3.0.29 or newer.

Comment 3 Meng Bo 2016-09-05 09:49:20 UTC
Checked on OCP build v3.3.0.29 with router image v3.3.0.29.

Issue has been fixed.

Comment 5 errata-xmlrpc 2016-09-27 09:47:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933

Comment 6 Ben Bennett 2016-10-28 17:44:02 UTC
*** Bug 1387717 has been marked as a duplicate of this bug. ***