Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1623987

Summary: After running redeploy-certificates.yml playbook in OCP 3.9 prometheus stop working
Product: OpenShift Container Platform Reporter: oarribas <oarribas>
Component: MonitoringAssignee: Frederic Branczyk <fbranczy>
Status: CLOSED DEFERRED QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: anpicker, aos-bugs, aship, clasohm, cvogel, dmoessne, farandac, gsapienz, jialiu, jokerman, jrosenta, mifiedle, mmccomas, nate.childers, oarribas, openshift-bugs-escalate, vrutkovs, yanpzhan, yapei
Target Milestone: ---Keywords: NeedsTestCase
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1596557 Environment:
Last Closed: 2019-11-20 19:04:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1592303, 1596233, 1596557, 1667981    
Bug Blocks:    
Attachments:
Description Flags
503 error for prometheus route
none
pods logs none

Comment 6 Junqi Zhao 2019-03-01 07:54:25 UTC 
After running redeploy-certificates.yml playbook in OCP 3.9, prometheus works well except all routes meet 503 (Service Unavailable) error

# oc -n openshift-metrics get pod
NAME                             READY     STATUS    RESTARTS   AGE
prometheus-0                     6/6       Running   6          2h
prometheus-node-exporter-pff22   1/1       Running   1          2h
prometheus-node-exporter-stg8f   1/1       Running   2          2h

# oc -n openshift-metrics get route
NAME           HOST/PORT                                                     PATH      SERVICES       PORT      TERMINATION   WILDCARD
alertmanager   alertmanager-openshift-metrics.apps.0301-xvz.qe.rhcloud.com             alertmanager   <all>     reencrypt     None
alerts         alerts-openshift-metrics.apps.0301-xvz.qe.rhcloud.com                   alerts         <all>     reencrypt     None
prometheus     prometheus-openshift-metrics.apps.0301-xvz.qe.rhcloud.com               prometheus     <all>     reencrypt     None


Change  TERMINATION from reencrypt to passthrough can login all routes 

images
oauth-proxy-v3.9.70-1
prometheus-v3.9.70-1
prometheus-alert-buffer-v3.9.70-1
prometheus-alertmanager-v3.9.70-1
prometheus-node-exporter-v3.9.71-1

openshift-ansible version: v3.9.70
Comment 7 Junqi Zhao 2019-03-01 08:20:37 UTC 
Created attachment 1539749 [details]
503 error for prometheus route
Comment 8 Junqi Zhao 2019-03-04 03:23:37 UTC 
Tested with prometheus v3.9.71 and openshift-ansible v3.9.71, still can not login all routes 

alerts-proxy/alert-buffer/prom-proxy container reports error
2019/03/04 03:12:40 server.go:2753: http: TLS handshake error from 10.129.0.1:35806: remote error: tls: unknown certificate authority
Comment 9 Junqi Zhao 2019-03-04 03:32:48 UTC 
Created attachment 1540442 [details]
pods logs
Comment 15 Stephen Cuppett 2019-11-20 19:04:57 UTC 
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift
Comment 16 Stephen Cuppett 2019-11-20 19:06:35 UTC 
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift