Bug 1734806

Summary: Metrics is not accessible via route
Product: OpenShift Container Platform Reporter: jooho lee <jlee>
Component: RoutingAssignee: Dan Mace <dmace>
Status: CLOSED NOTABUG QA Contact: Hongan Li <hongli>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, dmace, mmasters, nagrawal
Target Milestone: ---Flags: jlee: needinfo? (dmace)
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-16 19:18:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Metric error messages none

Description jooho lee 2019-07-31 13:44:10 UTC 
Description of problem:

Hawkular pod is running well and it is accessible from router pod.

~~~
sh-4.2$ oc rsh router-13-ljsxs
sh-4.2$ curl -vvvk https://10.129.0.24:8443  #(pod ip: 10.129.0.24)
...
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta http-equiv="refresh" content="0; url=hawkular/metrics">
  <title>Hawkular Metrics</title>
</head>

<body>
  <h1>Hawkular Metrics</h1>
  <h3>A time series metrics engine based on Cassandra</h3>
</body>
</html>
* Connection #0 to host 10.129.0.24 left intact

~~~


However, when a client try to access metrics via route, it can not match backend.

~~~
sh-4.2$ oc rsh router-13-ljsxs
sh-4.2$ curl -vvvk -H "Host: metrics.dev.ocp.thops.cloud" https://localhost
  </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
~~~

There are 4 routers and all routers behavior is the same.


Version-Release number of selected component (if applicable):


How reproducible:
I can not reproduce on my end. I tried to redeploy routers but the result was the same

Steps to Reproduce:
1.
2.
3.

Actual results:
Success to access metrics


Expected results:
Can not find metrics backend.


Additional info:
Comment 2 jooho lee 2019-07-31 13:46:40 UTC 
Created attachment 1595089 [details]
Metric error messages
Comment 3 jooho lee 2019-07-31 13:48:20 UTC 
Hi,

Only Metrics/console has this issue.

For other applications, haproxy work fine.
~~~
curl -vvvk -H "Host: redisdemo-ext-redisdemo.dev.ocp.thops.cloud" http://localhost  
* About to connect() to localhost port 80 (#0)
*   Trying ::1...
* Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: redisdemo-ext-redisdemo.dev.ocp.thops.cloud
> 
< HTTP/1.1 302 
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Set-Cookie: redisdemo-dev=NTQwZWNiOGMtNGI4OC00N2M1LTkxNTAtYmQzMTY3ZTNiMDI5; Path=/; HttpOnly
< Location: http://redisdemo-ext-redisdemo.dev.ocp.thops.cloud/session
< Content-Language: en
< Content-Length: 0
< Date: Tue, 30 Jul 2019 12:31:18 GMT
< Set-Cookie: 64382ecd689f405b024b5a364cf46420=997e1751eb80679b6ca9691aaf5d8a00; path=/; HttpOnly
< 
* Connection #0 to host localhost left intact

~~~
Comment 4 Miciah Dashiel Butler Masters 2019-07-31 20:21:44 UTC 
Are re-encrypt routes in general not working?

Have you verified that the certificates are valid?  The curl commands you are using should have printed information about the server certificates; did you delete that information?  If so, please include it so we can verify that the certificates match the host names ("metrics.dev.ocp.thops.cloud" in the case of the metrics route and "console.openshift-console.svc" in the case of the console route).

Can you provide the HAProxy logs, the definitions of the affected routes, and the corresponding services' definitions as well?
Comment 11 jooho lee 2019-08-16 19:18:05 UTC 
After reinstallation of Metric, the issue is gone. I believe recreating route can fix the issue.

So I will close this ticket.