Bug 1793850 - UsingDeprecatedAPIExtensionsV1Beta1 alert does not clear
Summary: UsingDeprecatedAPIExtensionsV1Beta1 alert does not clear
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Maciej Szulik
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On:
Blocks: 1795617 1816773
TreeView+ depends on / blocked
 
Reported: 2020-01-22 04:55 UTC by Christoph Blecker
Modified: 2020-05-04 11:26 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1795617 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:25:48 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-kube-apiserver-operator pull 742 None closed Bug 1793850: Update UsingDeprecatedAPIExtensionsV1Beta1 alert 2020-10-20 16:14:44 UTC
Github openshift cluster-kube-apiserver-operator pull 762 None closed Bug 1793850: Update UsingDeprecatedAPIExtensionsV1Beta1 alert 2020-10-20 16:14:31 UTC
Github openshift cluster-kube-apiserver-operator pull 764 None closed Bug 1793850: Update UsingDeprecatedAPIExtensionsV1Beta1 alert 2020-10-20 16:14:31 UTC
Red Hat Bugzilla 1788812 medium CLOSED Test cluster is firing large number of UsingDeprecatedAPIExtensionsV1Beta1 alerts. 2020-10-14 00:28:05 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:26:26 UTC

Description Christoph Blecker 2020-01-22 04:55:43 UTC
Description of problem:
While testing 4.3 on the OpenShift Dedicated platform, multiple instances of the UsingDeprecatedAPIExtensionsV1Beta1 alert fire.

They were traced down to the backup software that lists all objects in the cluster, including the extensions/v1beta1 versions of Deployments and PodSecurityPolicies that are still advertised via the API.

This alert is problematic, as it fires at a warning level (making the cluster show up as unhealthy), and that it has no clearing condition. As it is currently just a straight counter, once it fires it will never resolve itself.

If this alert is intended to be shipped with 4.3.0, then I'd suggest the following:
- Reduce the alert's severity to none
- Change the alert to use `increase` over a timespan, so that if the requests stop happening, the alert is able to resolve itself.


Version-Release number of selected component (if applicable):
4.3.0-rc.3


How reproducible:
Consistently


Steps to Reproduce:
1. Provision an OSD cluster on 4.3.0-rc.3 (this will automatically set up the backup operator)
2. Wait for cluster to fully provision and initial backups to automatically run
3. Observe the alert firing

Actual results:
Alert fires and never resolves

Expected results:
Alert should not indicate the cluster is unhealthy, and should resolve when the error condition goes away

Additional info:
We can disable this alert on OSD as scraping all api-resources is expected, but this will negatively impact all OCP users.

Comment 5 Ke Wang 2020-02-19 07:11:11 UTC
1. Verified with OCP build 4.4.0-0.nightly-2020-02-18-191238, 

$ oc get PrometheusRule -n  openshift-kube-apiserver -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: PrometheusRule
  metadata:
    annotations:
      exclude.release.openshift.io/internal-openshift-hosted: "true"
    creationTimestamp: "2020-02-19T02:00:26Z"
    generation: 1
    name: kube-apiserver
    namespace: openshift-kube-apiserver
    resourceVersion: "16714"
    selfLink: /apis/monitoring.coreos.com/v1/namespaces/openshift-kube-apiserver/prometheusrules/kube-apiserver
    uid: ff3e8cba-906f-48ee-9361-8e4abbf8785f
  spec:
    groups:
    - name: apiserver-requests-in-flight
      rules:
      - expr: |
          max_over_time(sum(apiserver_current_inflight_requests{apiserver=~"openshift-apiserver|kube-apiserver"}) by (apiserver,requestKind)[2m:])
        record: cluster:apiserver_current_inflight_requests:sum:max_over_time:2m
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The alert: UsingDeprecatedAPIExtensionsV1Beta1 has been remove from PrometheusRule.

2. Created file daemonset-simple-hello-openshift.yaml with the following content,
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: hello-openshift-daemonset
spec:
  selector:
      matchLabels:
        name: hello-openshift-daemonset
  template:
    metadata:
      labels:
        name: hello-openshift-daemonset
    spec:
      containers:
      - image: openshift/hello-openshift
        name: hello-openshift

$ oc create -f daemonset-simple-hello-openshift.yaml
error: unable to recognize "daemonset-simple-hello-openshift.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"

Check it on Prometheus UI -> Alerts page, searching UsingDeprecatedAPIExtensionsV1Beta1, there is no related log.
Also check it on Monitoring -> Alerting page, there is no firing log of UsingDeprecatedAPIExtensionsV1Beta1.

Comment 7 errata-xmlrpc 2020-05-04 11:25:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.