Bug 1859156

Summary: "http: server gave HTTP response to HTTPS client" for thanos-ruler endpoints
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Simon Pasquier <spasquie>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.6CC: alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, spasquie, surbania
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:16:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
openshift-user-workload-monitoring dump file
none
"http: server gave HTTP response to HTTPS client" for thanos-ruler endpoints none

Description Junqi Zhao 2020-07-21 11:04:17 UTC 
Created attachment 1701878 [details]
openshift-user-workload-monitoring dump file

Description of problem:
enabled UserWorkload
*************************
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
*************************
see from the prometheus /targets page, "http: server gave HTTP response to HTTPS client" for thanos-ruler endpoints
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -g -H "Authorization: Bearer $token" 'https://10.128.2.16:10902/metrics'
curl: (35) SSL received a record that exceeded the maximum permissible length.
command terminated with exit code 35

# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -g -H "Authorization: Bearer $token" 'http://10.128.2.16:10902/metrics' | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 9.913e-06
go_gc_duration_seconds{quantile="0.25"} 3.5879e-05
go_gc_duration_seconds{quantile="0.5"} 4.1316e-05
go_gc_duration_seconds{quantile="0.75"} 4.586e-05
go_gc_duration_seconds{quantile="1"} 0.005978851
go_gc_duration_seconds_sum 0.031030653
go_gc_duration_seconds_count 224
# HELP go_goroutines Number of goroutines that currently exist.


Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-07-20-183524

How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Junqi Zhao 2020-07-21 11:05:05 UTC 
Created attachment 1701880 [details]
"http: server gave HTTP response to HTTPS client" for thanos-ruler endpoints
Comment 7 Junqi Zhao 2020-07-24 03:06:34 UTC 
issue is fixed with 4.6.0-0.nightly-2020-07-23-194834, thanos-ruler endpoints are UP
# oc -n openshift-user-workload-monitoring get thanosruler/user-workload -oyaml | grep listenLocal
  listenLocal: true
Comment 9 errata-xmlrpc 2020-10-27 16:16:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196