Bug 1402488

Summary: After namespace router shard implementation router intermittently does not get the route
Product: OpenShift Container Platform Reporter: Steven Walter <stwalter>
Component: NetworkingAssignee: Ram Ranganathan <ramr>
Networking sub component: router QA Contact: zhaozhanqi <zzhao>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs, bbennett
Version: 3.3.0   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-27 16:25:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Steven Walter 2016-12-07 16:22:57 UTC
Description of problem:
We have implemented router shard namespaces in our automation lab. During testing of this change we found that intermittently that the router does not pick up the route. In some cases it is enough to re-create the route and in the worse case I need to restart the router pod for the route to be added to `/var/lib/haproxy/router/routes.json`.

Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
Unconfirmed

Steps to Reproduce:
Putting in next comment



Actual results:
$ oc exec ${INTERNAL_ROUTER} -n default -- bash -c 'cat /var/lib/haproxy/router/routes.json' | grep internal-router-test
$ oc exec ${DMZ_ROUTER} -n default -- bash -c 'cat /var/lib/haproxy/router/routes.json' | grep internal-router-test
$

Expected results:
$ INTERNAL_ROUTER=$(oc get pods -l router=router --no-headers=true -n default | awk '{print $1}')
$ DMZ_ROUTER=$(oc get pods -l router=router-dmz --no-headers=true -n default | awk '{print $1}')
$ oc exec ${INTERNAL_ROUTER} -n default -- bash -c 'cat /var/lib/haproxy/router/routes.json' | grep internal-router-test
    "internal-router-test_php": {
      "Namespace": "internal-router-test",
      "Host": "php-internal-router-test.paas-a.am.lilly.com",
        "internal-router-test/php": 100
$ oc exec ${DMZ_ROUTER} -n default -- bash -c 'cat /var/lib/haproxy/router/routes.json' | grep internal-router-test
$

Comment 2 Steven Walter 2016-12-07 16:24:33 UTC
After waiting the 10 minute refresh interval, the routes still did not show up.

Router logs just show:

    I1205 12:00:14.036891       1 router.go:404] Router reloaded:
     - Checking HAProxy /healthz on port 1936 ...
     - HAProxy port 1936 health check ok : 0 retry attempt(s).
    I1205 12:00:21.950161       1 router.go:404] Router reloaded:
     - Checking HAProxy /healthz on port 1936 ...
     - HAProxy port 1936 health check ok : 0 retry attempt(s).
    I1205 12:00:32.653189       1 router.go:404] Router reloaded:
     - Checking HAProxy /healthz on port 1936 ...
     - HAProxy port 1936 health check ok : 0 retry attempt(s).
    I1205 12:00:33.593595       1 router.go:404] Router reloaded:
     - Checking HAProxy /healthz on port 1936 ...
     - HAProxy port 1936 health check ok : 0 retry attempt(s).
    I1205 12:04:04.020082       1 router.go:404] Router reloaded:
     - Checking HAProxy /healthz on port 1936 ...
     - HAProxy port 1936 health check ok : 0 retry attempt(s).

etc

Comment 5 zhaozhanqi 2016-12-08 07:12:32 UTC
hi, Steven

From the comment 1

<--snip-->

 oc exec ${INTERNAL_ROUTER} -n default -- bash -c 'cat /var/lib/haproxy/router/routes.json' | grep internal-router-test
    "internal-router-test_php": {
      "Namespace": "internal-router-test",
      "Host": "php-internal-router-test.paas-a.am.lilly.com",
        "internal-router-test/php": 100
  [14:14:39] u3x1609:~ $ oc exec ${DMZ_ROUTER} -n default -- bash -c 'cat /var/lib/haproxy/router/routes.json' | grep internal-router-test

<--snip-->

I think this is correct result since ${DMZ_ROUTER} router only match label dmz-router=yes"

Comment 6 Steven Walter 2016-12-08 14:49:09 UTC
Hi,

Yes, that is what they are hoping for. That output is correct. However they only see it after manually restarting the router. It is supposed to show up in the ${INTERNAL_ROUTER}, but sometimes when they create the route it does not show in the ${INTERNAL_ROUTER}, and they have to delete the pod (creates a new pod) to have it show up inside the router.

Comment 8 Ben Bennett 2017-01-27 16:25:03 UTC

*** This bug has been marked as a duplicate of bug 1383663 ***