Bug 1477738 - Stable metrics pods failing to provide metrics in webUI
Stable metrics pods failing to provide metrics in webUI
Status: CLOSED NOTABUG
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
unspecified
Unspecified Unspecified
high Severity high
: ---
: 3.2.1
Assigned To: Scott Dodson
Johnny Liu
: UpcomingRelease
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-02 14:50 EDT by Eric Jones
Modified: 2017-09-20 11:44 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-20 11:44:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eric Jones 2017-08-02 14:50:37 EDT
Description of problem:
Customer redeployed certificates in cluster and began to experience issues with metrics. After some troubleshooting, we deleted metrics and redeployed. After redeploying, the pods are completely stable but webUI metrics tab shows error message.

Looking at this page in the Inspect console for the browser, and specifically navigating to the Network tab shows 403 error messages for some metrics calls but the hawkular-metrics url is fully routeable.

Version-Release number of selected component (if applicable):
3.2.0

Additional info:
Attaching pod logs, pod yaml, rc yaml, service yaml, route yaml, and several screenshots shortly.
Comment 2 Matt Wringe 2017-08-03 11:59:59 EDT
Can we please get his updated to 3.2.1?
Comment 7 Matt Wringe 2017-08-08 14:53:43 EDT
The error message in the Hawkular Metrics log is indicating the that CA certificate used to sign the Master API is not valid:

"unable to find valid certification path to requested target"

Can you run the following commands:

Create and enter a debug pod:

'oc debug ${HAWKULAR_METRICS_POD_NAME}'

Run the following commands once within the pod:

'curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
$MASTER_URL'

'cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt $MASTER_URL'

That should help us get a handle on what is happening here and if the certificates for the master are signed properly or not.
Comment 14 Eric Jones 2017-09-20 11:44:12 EDT
Closing this bug as not a bug.

Closing as we actually identified the issue and are working towards a resolution in https://bugzilla.redhat.com/show_bug.cgi?id=1479930

Note You need to log in before you can comment on or make changes to this bug.