Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1591077

Summary: Heapster can not collect data
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: HawkularAssignee: Ruben Vargas Palma <rvargasp>
Status: CLOSED NOTABUG QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: high    
Version: 3.10.0CC: aos-bugs, jsanda
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-21 03:30:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
metrics logs
none
Metrics diagram can not be shown in web UI none

Description Junqi Zhao 2018-06-14 02:29:40 UTC 
Created attachment 1451082 [details]
metrics logs

Description of problem:
Metrics default namespace is openshift-metrics(see https://bugzilla.redhat.com/show_bug.cgi?id=1570583), after deploying metrics 3.10, heaster can not collect data
There is repeatly error in heapster pod log, heaster can not collect data
metrics:heapster" cannot list nodes at the cluster scope: User "system:serviceaccount:openshift-metrics:heapster" cannot list all nodes in the cluster
E0614 02:02:04.041642       1 reflector.go:190] k8s.io/heapster/metrics/heapster.go:322: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:openshift-metrics:heapster" cannot list pods at the cluster scope: User "system:serviceaccount:openshift-metrics:heapster" cannot list all pods in the cluster
E0614 02:02:04.041921       1 reflector.go:190] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:84: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:openshift-metrics:heapster" cannot list namespaces at the cluster scope: User "system:serviceaccount:openshift-metrics:heapster" cannot list all namespaces in the cluster


# oc get sa heapster -n openshift-metrics -o yaml
apiVersion: v1
imagePullSecrets:
- name: heapster-dockercfg-g8trq
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"labels":{"metrics-infra":"support"},"name":"heapster","namespace":"openshift-metrics"},"secrets":[{"name":"heapster-secrets"},{"name":"hawkular-metrics-certs"},{"name":"hawkular-metrics-account"}]}
  creationTimestamp: 2018-06-14T01:58:04Z
  labels:
    metrics-infra: support
  name: heapster
  namespace: openshift-metrics
  resourceVersion: "13827"
  selfLink: /api/v1/namespaces/openshift-metrics/serviceaccounts/heapster
  uid: 603afb4d-6f76-11e8-8765-42010af00021
secrets:
- name: heapster-secrets
- name: hawkular-metrics-certs
- name: hawkular-metrics-account
- name: heapster-dockercfg-g8trq
- name: heapster-token-4mr62


Version-Release number of selected component (if applicable):
metrics-cassandra-v3.10.0-0.68.0.0
metrics-hawkular-metrics-v3.10.0-0.68.0.0
metrics-heapster-v3.10.0-0.68.0.0


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.10, inventory file see [Additional info] part
2.
3.

Actual results:
Heaster can not collect data

Expected results:
Heaster can collect data

Additional info:
openshift_metrics_install_metrics=true
openshift_metrics_image_prefix=registry.reg-aws.openshift.com:443/openshift3/
openshift_metrics_image_version=v3.10
openshift_metrics_cassandra_storage_type=dynamic
Comment 1 Junqi Zhao 2018-06-14 02:36:14 UTC 
Blocks metrics testing
Comment 2 Junqi Zhao 2018-06-14 02:50:47 UTC 
Created attachment 1451083 [details]
Metrics diagram can not be shown in web UI
Comment 3 John Sanda 2018-06-14 16:04:51 UTC 
The metrics playbook now includes a script, clean_old_namespace.yaml, that is run after everything is installed. It deletes resources from the old namespace, i.e., openshift-infra. It also deletes the cluster role bindings. The heapster service account is bound to the role. In case of an upgrade, we delete the old SA and create a new one in the new namespace. We need to make sure that the role is added to the new SA.
Comment 4 John Sanda 2018-06-14 16:11:35 UTC 
Ruben,

The problem also exists with the hawkular role bindings. They are gone as well. We need to make sure the new hawkular-metrics SAs are getting bound as well.
Comment 6 Junqi Zhao 2018-06-21 03:30:42 UTC 
Change is reverted, metrics is now installed under openshift-infra project and heapster can collect data


# rpm -qa | grep openshift-ansible
openshift-ansible-roles-3.10.2-1.git.190.5abfddb.el7.noarch
openshift-ansible-playbooks-3.10.2-1.git.190.5abfddb.el7.noarch
openshift-ansible-docs-3.10.2-1.git.190.5abfddb.el7.noarch
openshift-ansible-3.10.2-1.git.190.5abfddb.el7.noarch

# openshift version
openshift v3.10.2

metrics version: v3.10.2-1