Bug 1793850

Summary:	UsingDeprecatedAPIExtensionsV1Beta1 alert does not clear
Product:	OpenShift Container Platform	Reporter:	Christoph Blecker <cblecker>
Component:	kube-apiserver	Assignee:	Maciej Szulik <maszulik>
Status:	CLOSED ERRATA	QA Contact:	Ke Wang <kewang>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.3.0	CC:	aos-bugs, mfojtik, mmariyan, mrhodes, obockows, xxia
Target Milestone:	---
Target Release:	4.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	1795617 (view as bug list)		Environment:
Last Closed:	2020-05-04 11:25:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1795617, 1816773

Description Christoph Blecker 2020-01-22 04:55:43 UTC

Description of problem:
While testing 4.3 on the OpenShift Dedicated platform, multiple instances of the UsingDeprecatedAPIExtensionsV1Beta1 alert fire.

They were traced down to the backup software that lists all objects in the cluster, including the extensions/v1beta1 versions of Deployments and PodSecurityPolicies that are still advertised via the API.

This alert is problematic, as it fires at a warning level (making the cluster show up as unhealthy), and that it has no clearing condition. As it is currently just a straight counter, once it fires it will never resolve itself.

If this alert is intended to be shipped with 4.3.0, then I'd suggest the following:
- Reduce the alert's severity to none
- Change the alert to use `increase` over a timespan, so that if the requests stop happening, the alert is able to resolve itself.


Version-Release number of selected component (if applicable):
4.3.0-rc.3


How reproducible:
Consistently


Steps to Reproduce:
1. Provision an OSD cluster on 4.3.0-rc.3 (this will automatically set up the backup operator)
2. Wait for cluster to fully provision and initial backups to automatically run
3. Observe the alert firing

Actual results:
Alert fires and never resolves

Expected results:
Alert should not indicate the cluster is unhealthy, and should resolve when the error condition goes away

Additional info:
We can disable this alert on OSD as scraping all api-resources is expected, but this will negatively impact all OCP users.

Comment 5 Ke Wang 2020-02-19 07:11:11 UTC

1. Verified with OCP build 4.4.0-0.nightly-2020-02-18-191238, 

$ oc get PrometheusRule -n  openshift-kube-apiserver -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: PrometheusRule
  metadata:
    annotations:
      exclude.release.openshift.io/internal-openshift-hosted: "true"
    creationTimestamp: "2020-02-19T02:00:26Z"
    generation: 1
    name: kube-apiserver
    namespace: openshift-kube-apiserver
    resourceVersion: "16714"
    selfLink: /apis/monitoring.coreos.com/v1/namespaces/openshift-kube-apiserver/prometheusrules/kube-apiserver
    uid: ff3e8cba-906f-48ee-9361-8e4abbf8785f
  spec:
    groups:
    - name: apiserver-requests-in-flight
      rules:
      - expr: |
          max_over_time(sum(apiserver_current_inflight_requests{apiserver=~"openshift-apiserver|kube-apiserver"}) by (apiserver,requestKind)[2m:])
        record: cluster:apiserver_current_inflight_requests:sum:max_over_time:2m
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The alert: UsingDeprecatedAPIExtensionsV1Beta1 has been remove from PrometheusRule.

2. Created file daemonset-simple-hello-openshift.yaml with the following content,
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: hello-openshift-daemonset
spec:
  selector:
      matchLabels:
        name: hello-openshift-daemonset
  template:
    metadata:
      labels:
        name: hello-openshift-daemonset
    spec:
      containers:
      - image: openshift/hello-openshift
        name: hello-openshift

$ oc create -f daemonset-simple-hello-openshift.yaml
error: unable to recognize "daemonset-simple-hello-openshift.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"

Check it on Prometheus UI -> Alerts page, searching UsingDeprecatedAPIExtensionsV1Beta1, there is no related log.
Also check it on Monitoring -> Alerting page, there is no firing log of UsingDeprecatedAPIExtensionsV1Beta1.

Comment 7 errata-xmlrpc 2020-05-04 11:25:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581