Bug 1950430

Summary: CVO serves metrics over HTTP, despite a lack of consumers
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: Cluster Version OperatorAssignee: W. Trevor King <wking>
Status: CLOSED ERRATA QA Contact: Johnny Liu <jialiu>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.7CC: aos-bugs, jokerman, spasquie
Target Milestone: ---Keywords: Upgrades
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:01:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description W. Trevor King 2021-04-16 16:21:19 UTC
We began serving metrics over HTTPS with 6132bc3 (cvo#358), which also requested monitoring to scrape us over HTTPS.  Now that that is all in place in 4.6, we no longer need to serve over HTTP in 4.7 and later.  We should drop HTTP and only serve metrics over HTTPS.

This might also fix recent occurrences of ClusterVersionOperatorDown, where Prometheus is failing to scrape some CVO pods, despite those pods claiming to be serving metrics.  For example [1] has:

  [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]	1h15m24s
  [sig-arch] Check if alerts are firing during or after upgrade success	1h15m13s
    Apr 15 17:03:27.273: Unexpected alerts fired or pending during the upgrade:

    alert ClusterVersionOperatorDown fired for 2280 seconds with labels: {severity="critical"}
    alert TargetDown fired for 1770 seconds with labels: {job="cluster-version-operator", namespace="openshift-cluster-version", service="cluster-version-operator", severity="warning"}

The current suspicion is that the issue is a bug or incompatibility between the scraper's cmux and our metric server's cmux, and going to HTTPS-only would remove our server's cmux.  Recent frequency:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=24h&type=junit&search=alert+ClusterVersionOperatorDown+fired' | grep 'failures match' | sort
periodic-ci-openshift-release-master-ci-4.8-e2e-azure-compact-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
periodic-ci-openshift-release-master-ci-4.8-e2e-gcp-upgrade (all) - 17 runs, 76% failed, 8% of failures match = 6% impact
periodic-ci-openshift-release-master-nightly-4.8-e2e-azure (all) - 11 runs, 64% failed, 14% of failures match = 9% impact
periodic-ci-openshift-release-master-nightly-4.8-upgrade-from-stable-4.7-e2e-metal-ipi-upgrade (all) - 11 runs, 100% failed, 9% of failures match = 9% impact
pull-ci-openshift-cluster-config-operator-master-e2e-upgrade (all) - 5 runs, 40% failed, 50% of failures match = 20% impact
pull-ci-openshift-router-master-e2e-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact
release-openshift-okd-installer-e2e-aws-upgrade (all) - 16 runs, 63% failed, 10% of failures match = 6% impact

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-gcp-upgrade/1382704884662407168

Comment 2 Johnny Liu 2021-04-21 09:59:19 UTC
Verified this bug with 4.8.0-0.nightly-2021-04-20-195442, and PASS.

[root@preserve-jialiu-ansible ~]# oc -n openshift-cluster-version rsh cluster-version-operator-68f556c79-n6lm8
sh-4.4# cat /proc/1/cmdline 

sh-4.4# /usr/bin/cluster-version-operator start --help
Starts Cluster Version Operator
      --serving-cert-file string         The X.509 certificate file for serving metrics over HTTPS.  You must set both --serving-cert-file and --serving-key-file unless you set --listen empty.
      --serving-key-file string          The X.509 key file for serving metrics over HTTPS.  You must set both --serving-cert-file and --serving-key-file unless you set --listen empty.

sh-4.4# /usr/bin/cluster-version-operator start --release-image=registry.ci.openshift.org/ocp/release@sha256:a4c4f8797512b71c84ee9899fbbec6713db67d490e9b82596888f23dd1ccd9a4 --enable-auto-update=false --enable-default-cluster-version=true --listen= --v=5
I0421 09:45:40.940525      49 start.go:21] ClusterVersionOperator 4.8.0-202104201727.p0-6fdd1e0
F0421 09:45:40.940609      49 start.go:24] error: --listen was not set empty, so --serving-cert-file must be set
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000012001, 0xc00045c000, 0x71, 0xb4)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x27910e0, 0xc000000003, 0x0, 0x0, 0xc000590540, 0x202645e, 0x8, 0x18, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:975 +0x191
k8s.io/klog/v2.(*loggingT).printf(0x27910e0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1a5ceae, 0x9, 0xc000136310, 0x1, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:750 +0x191
main.init.3.func1(0xc000454000, 0xc00042e140, 0x0, 0x5)
	/go/src/github.com/openshift/cluster-version-operator/cmd/start.go:24 +0x1ed
github.com/spf13/cobra.(*Command).execute(0xc000454000, 0xc00042e0f0, 0x5, 0x5, 0xc000454000, 0xc00042e0f0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x277d600, 0xc000000180, 0xc00005c740, 0x46eb45)
	/go/src/github.com/openshift/cluster-version-operator/vendor/github.com/spf13/cobra/command.go:958 +0x375
	/go/src/github.com/openshift/cluster-version-operator/cmd/main.go:26 +0x53

goroutine 6 [chan receive]:
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/klog/v2/klog.go:417 +0xdf

sh-4.4# curl
Client sent an HTTP request to an HTTPS server.

Comment 3 W. Trevor King 2021-04-21 23:58:21 UTC
Looking a lot better vs. the Prom scraping too:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=24h&type=junit&search=alert+ClusterVersionOperatorDown+fired' | grep 'failures match' | sort
periodic-ci-openshift-release-master-okd-4.8-e2e-vsphere (all) - 11 runs, 100% failed, 9% of failures match = 9% impact

Comment 6 errata-xmlrpc 2021-07-27 23:01:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.