Bug 1734806 - Metrics is not accessible via route [NEEDINFO]
Summary: Metrics is not accessible via route
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Dan Mace
QA Contact: Hongan Li
Depends On:
TreeView+ depends on / blocked
Reported: 2019-07-31 13:44 UTC by jooho lee
Modified: 2019-08-16 19:18 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-08-16 19:18:05 UTC
Target Upstream Version:
jlee: needinfo? (dmace)

Attachments (Terms of Use)
Metric error messages (3.19 KB, text/plain)
2019-07-31 13:46 UTC, jooho lee
no flags Details

Description jooho lee 2019-07-31 13:44:10 UTC
Description of problem:

Hawkular pod is running well and it is accessible from router pod.

sh-4.2$ oc rsh router-13-ljsxs
sh-4.2$ curl -vvvk  #(pod ip:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
  <meta http-equiv="refresh" content="0; url=hawkular/metrics">
  <title>Hawkular Metrics</title>

  <h1>Hawkular Metrics</h1>
  <h3>A time series metrics engine based on Cassandra</h3>
* Connection #0 to host left intact


However, when a client try to access metrics via route, it can not match backend.

sh-4.2$ oc rsh router-13-ljsxs
sh-4.2$ curl -vvvk -H "Host: metrics.dev.ocp.thops.cloud" https://localhost
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.

There are 4 routers and all routers behavior is the same.

Version-Release number of selected component (if applicable):

How reproducible:
I can not reproduce on my end. I tried to redeploy routers but the result was the same

Steps to Reproduce:

Actual results:
Success to access metrics

Expected results:
Can not find metrics backend.

Additional info:

Comment 2 jooho lee 2019-07-31 13:46:40 UTC
Created attachment 1595089 [details]
Metric error messages

Comment 3 jooho lee 2019-07-31 13:48:20 UTC

Only Metrics/console has this issue.

For other applications, haproxy work fine.
curl -vvvk -H "Host: redisdemo-ext-redisdemo.dev.ocp.thops.cloud" http://localhost  
* About to connect() to localhost port 80 (#0)
*   Trying ::1...
* Connection refused
*   Trying
* Connected to localhost ( port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: redisdemo-ext-redisdemo.dev.ocp.thops.cloud
< HTTP/1.1 302 
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Set-Cookie: redisdemo-dev=NTQwZWNiOGMtNGI4OC00N2M1LTkxNTAtYmQzMTY3ZTNiMDI5; Path=/; HttpOnly
< Location: http://redisdemo-ext-redisdemo.dev.ocp.thops.cloud/session
< Content-Language: en
< Content-Length: 0
< Date: Tue, 30 Jul 2019 12:31:18 GMT
< Set-Cookie: 64382ecd689f405b024b5a364cf46420=997e1751eb80679b6ca9691aaf5d8a00; path=/; HttpOnly
* Connection #0 to host localhost left intact


Comment 4 Miciah Dashiel Butler Masters 2019-07-31 20:21:44 UTC
Are re-encrypt routes in general not working?

Have you verified that the certificates are valid?  The curl commands you are using should have printed information about the server certificates; did you delete that information?  If so, please include it so we can verify that the certificates match the host names ("metrics.dev.ocp.thops.cloud" in the case of the metrics route and "console.openshift-console.svc" in the case of the console route).

Can you provide the HAProxy logs, the definitions of the affected routes, and the corresponding services' definitions as well?

Comment 11 jooho lee 2019-08-16 19:18:05 UTC
After reinstallation of Metric, the issue is gone. I believe recreating route can fix the issue.

So I will close this ticket.

Note You need to log in before you can comment on or make changes to this bug.