Bug 1548622
| Summary: | Heaspter pod can not be started when Hawkular Metrics is already in the STARTED state | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||
| Component: | Hawkular | Assignee: | John Sanda <jsanda> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Junqi Zhao <juzhao> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 3.9.0 | CC: | aos-bugs, juzhao, pruan, smunilla, sross | ||||
| Target Milestone: | --- | Keywords: | Regression, TestBlocker | ||||
| Target Release: | 3.9.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | 3.9.0.53 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-06-18 17:28:23 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
# oc get ep heapster -o yaml
apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: 2018-02-24T02:03:05Z
labels:
metrics-infra: heapster
name: heapster
name: heapster
namespace: openshift-infra
resourceVersion: "141159"
selfLink: /api/v1/namespaces/openshift-infra/endpoints/heapster
uid: da5ea064-1906-11e8-8eb0-fa163ef863c8
subsets:
- notReadyAddresses:
- ip: 10.129.0.26
nodeName: 172.16.120.7
targetRef:
kind: Pod
name: heapster-7p4lw
namespace: openshift-infra
resourceVersion: "140515"
uid: ea5c3e49-1906-11e8-8eb0-fa163ef863c8
ports:
- port: 8082
protocol: TCP
Blocks all metrics testing Also failed on 3.9.0.51 but same test passed on 3.9.0.47 Can you get any heapster logs? It appears to be a cert issue. I manually ran curl from the heapster pod and get this error: sh-4.2$ curl --insecure -v https://hawkular-metrics:443/hawkular/metrics/status * About to connect() to hawkular-metrics port 443 (#0) * Trying 172.30.253.64... * Connected to hawkular-metrics (172.30.253.64) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * NSS error -8054 (SEC_ERROR_REUSED_ISSUER_AND_SERIAL) * You are attempting to import a cert with the same issuer/serial as an existing cert, but that is not the same cert. * Closing connection 0 curl: (35) You are attempting to import a cert with the same issuer/serial as an existing cert, but that is not the same cert. I think 3.9.0.51 may have hit https://github.com/openshift/origin/pull/18713 which has since been merged and available in later builds. Moving to ON_QA as per request from Peter Ruan. metrics installation works now using 3.9.0.53 which has https://github.com/openshift/origin/pull/18713 merged in. (In reply to John Sanda from comment #4) > Can you get any heapster logs? logs is like the following, and issue is fixed on OCP v3.9.0-0.53.0: Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 35. Status Code 000 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 35]. Retrying. Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 35. Status Code 000 'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 35]. Retrying. Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 35. Status Code 000 ******************************** |
Created attachment 1400067 [details] ansible log Description of problem: Heaspter pod can not be started, describe heapster-pod, info is "Readiness probe failed: The heapster process is not yet started, it is waiting for the Hawkular Metrics to start.", but hawkular-metrics-pod is already in the STARTED state and is available. The heapster endpoint is not generated. # oc describe po ${heapster-pod} Name: heapster-7p4lw Normal SuccessfulMountVolume 17m kubelet, 172.16.120.7 MountVolume.SetUp succeeded for volume "heapster-secrets" Normal SuccessfulMountVolume 17m kubelet, 172.16.120.7 MountVolume.SetUp succeeded for volume "hawkular-metrics-account" Normal Pulled 17m kubelet, 172.16.120.7 Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/metrics-heapster:v3.9" already present on machine Normal Created 17m kubelet, 172.16.120.7 Created container Normal Started 17m kubelet, 172.16.120.7 Started container Warning Unhealthy 2m (x91 over 17m) kubelet, 172.16.120.7 Readiness probe failed: The heapster process is not yet started, it is waiting for the Hawkular Metrics to start. # oc rsh ${hawkular-metrics-pod} sh-4.2$ python /opt/hawkular/scripts/hawkular-metrics-liveness.py The MetricsService is in the STARTED state and is available. sh-4.2$ python /opt/hawkular/scripts/hawkular-metrics-readiness.py The MetricService is in the STARTED state. The service is now ready. # oc get po NAME READY STATUS RESTARTS AGE hawkular-cassandra-1-6x98x 1/1 Running 0 31m hawkular-metrics-jmpm7 1/1 Running 0 31m heapster-7p4lw 0/1 Running 3 31m # oc get ep NAME ENDPOINTS AGE hawkular-cassandra 10.128.0.23:7001,10.128.0.23:9042,10.128.0.23:7000 22m hawkular-cassandra-nodes 10.128.0.23:7001,10.128.0.23:9042,10.128.0.23:7000 22m hawkular-metrics 10.128.0.24:8443 22m heapster 21m Version-Release number of selected component (if applicable): metrics-cassandra-v3.9.0-0.52.0.0 metrics-hawkular-metrics-v3.9.0-0.51.0.0 metrics-heapster-v3.9.0-0.51.0.0 # rpm -qa | grep openshift-ansible openshift-ansible-roles-3.9.0-0.48.0.git.0.2fb33db.el7.noarch openshift-ansible-playbooks-3.9.0-0.48.0.git.0.2fb33db.el7.noarch openshift-ansible-3.9.0-0.48.0.git.0.2fb33db.el7.noarch openshift-ansible-docs-3.9.0-0.48.0.git.0.2fb33db.el7.noarch How reproducible: Always Steps to Reproduce: 1. Deploy metrics 3.9 2. 3. Actual results: Heaspter pod can not be started Additional info: # metrics parameters openshift_metrics_install_metrics=true openshift_metrics_image_prefix=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ openshift_metrics_image_version=v3.9