Bug 1469295 - After upgrading to 3.5 hawkular-metrics pod cannot start with error against permissions
After upgrading to 3.5 hawkular-metrics pod cannot start with error against p...
Product: OpenShift Container Platform
Classification: Red Hat
Component: Metrics (Show other bugs)
Unspecified Unspecified
high Severity high
: ---
: 3.7.0
Assigned To: Matt Wringe
Junqi Zhao
Depends On:
  Show dependency treegraph
Reported: 2017-07-10 17:44 EDT by Eric Jones
Modified: 2017-08-03 12:09 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-08-03 12:09:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Eric Jones 2017-07-10 17:44:06 EDT
Description of problem:
Customer upgraded OpenShift cluster from 3.3 to 3.4 and then to 3.5 and tried to deploy metrics.

It deployed but eh hawkular-metrics pod fails with:
Starting Hawkular Metrics Error: the service account for Hawkular Metrics does not have permission to view resources in this namespace. View permissions are required for Hawkular Metrics to function properly. Usually this can be resolved by running: oc adm policy add-role-to-user view system:serviceaccount:openshift-infra:hawkular -n openshift-infra 

Version-Release number of selected component (if applicable):
metrics-hawkular-metrics   v3.5                4ede8a0257c8 = 3.5.0-22

metrics-heapster           v3.5                56f0e1727405 = 3.5.0-16

metrics-cassandra          v3.5                46585da34fbe = 3.5.0-19

Additional info:
Found BZ 1448462 and tested the commands there (cacert_output) but it had vastly different output than what that bug help so I opened this new one.

I will be attaching logs and that acert_output file shortly.
Comment 2 Matt Wringe 2017-07-11 09:47:41 EDT
From the attached cacert_output the error is that the OpenShift Master endpoint is not accepting a connection and is closing it (Connection reset by peer)

This can mean a few things.

The master API is not available. It could be behind a firewall or not exposed in a way that the Hawkular Metrics pod can access it.

The master API is not available at the expected hostname. By default this is https://kubernetes.default.svc:443 but the system may be setup to use a different internal hostname for it (you can configure a different hostname by specifying the openshift_metrics_master_url property in your inventory file).
Comment 7 Matt Wringe 2017-08-03 12:09:53 EDT
I am going to mark this as 'noabug' as it looks like they incorrectly set their metrics URL in their inventory file.

Note You need to log in before you can comment on or make changes to this bug.