Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2079913

Summary: [4.10] APIRemovedInNextEUSReleaseInUse alert for OVN endpointslices
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: NetworkingAssignee: ffernand <ffernand>
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: ffernand, obraunsh
Version: 4.10Keywords: Triaged
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 11:09:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2022-04-28 13:30:16 UTC 
Description of problem:
4.10.0-0.nightly-2022-04-28-030337 vSphere OVN cluster, APIRemovedInNextEUSReleaseInUse alert for endpointslices, it's caused by openshift-ovn-kubernetes:ovn-kubernetes-controller
maybe affected by code:
https://github.com/openshift/ovn-kubernetes/blob/release-4.10/go-controller/pkg/ovn/controller/services/services_controller.go#L16

# oc get network cluster -o jsonpath="{.spec.networkType}"
OVNKubernetes

# oc -n openshift-ovn-kubernetes get pod
NAME                   READY   STATUS    RESTARTS        AGE
ovnkube-master-5nqp9   6/6     Running   7 (129m ago)    4h46m
ovnkube-master-rp75s   6/6     Running   2 (132m ago)    4h46m
ovnkube-master-zmlmr   6/6     Running   6 (4h45m ago)   4h46m
ovnkube-node-5v6xw     5/5     Running   0               4h46m
ovnkube-node-7qg62     5/5     Running   0               4h46m
ovnkube-node-8n6vv     5/5     Running   0               4h35m
ovnkube-node-fdpl2     5/5     Running   0               4h46m
ovnkube-node-v6gvj     5/5     Running   0               4h35m

# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="endpointslices"}' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "group": "discovery.k8s.io",
          "namespace": "openshift-kube-apiserver",
          "resource": "endpointslices",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1651151845.452,
          "1"
        ]
      }
    ]
  }
}

# oc get apirequestcounts endpointslices.v1beta1.discovery.k8s.io -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-04-28T08:29:08Z"
  generation: 1
  name: endpointslices.v1beta1.discovery.k8s.io
  resourceVersion: "283596"
  uid: 112dace7-8bc5-4bd6-9280-ce19b51a3835
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.126
      requestCount: 1
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 1
  last24h:
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.126
      requestCount: 1
    - byUser:
      - byVerb:
        - requestCount: 3
          verb: watch
        requestCount: 3
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.204
      requestCount: 3
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.229
      requestCount: 2
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 6
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.204
      requestCount: 8
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 3
          verb: watch
        requestCount: 3
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.126
      requestCount: 3
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.204
      requestCount: 4
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 7
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 8
          verb: watch
        requestCount: 8
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.126
      requestCount: 8
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 8
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 7
          verb: watch
        requestCount: 7
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.126
      requestCount: 7
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 7
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: watch
        requestCount: 1
        userAgent: qeci-37349-p29xj-master-2/ovnkube@e36c4d2dcd44
        username: system:serviceaccount:openshift-ovn-kubernetes:ovn-kubernetes-controller
      nodeName: 172.31.249.126
      requestCount: 1
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 1
  - requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 172.31.249.126
      requestCount: 0
    - nodeName: 172.31.249.204
      requestCount: 0
    - nodeName: 172.31.249.229
      requestCount: 0
    - nodeName: 172.31.249.56
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.25"
  requestCount: 37

APIRemovedInNextEUSReleaseInUse alert detail
**********************
        - alert: APIRemovedInNextEUSReleaseInUse
          annotations:
            description: Deprecated API that will be removed in the next EUS version is
              being used. Removing the workload that is using the {{ $labels.group }}.{{
              $labels.version }}/{{ $labels.resource }} API might be necessary for a successful
              upgrade to the next EUS cluster version. Refer to `oc get apirequestcounts
              {{ $labels.resource }}.{{ $labels.version }}.{{ $labels.group }} -o yaml`
              to identify the workload.
            summary: Deprecated API that will be removed in the next EUS version is being
              used.
          expr: |
            group(apiserver_requested_deprecated_apis{removed_release=~"1\\.2[45]"}) by (group,version,resource) and (sum by(group,version,resource) (rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h]))) > 0
          for: 1h
          labels:
            namespace: openshift-kube-apiserver
            severity: info
**********************


Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-04-28-030337 OVN cluster
kubernetes v1.23.5+b463d71

How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
APIRemovedInNextEUSReleaseInUse alert for endpointslices

Expected results:
no such alert

Additional info:
Comment 1 ffernand 2022-06-23 19:01:10 UTC 
Fixed upstream and backported to 4.11 via PR:  https://github.com/openshift/ovn-kubernetes/pull/1078

Commit:  https://github.com/openshift/ovn-kubernetes/commit/ed24bd4117e97c04959ac2e0f2e04533bb973e8f
Comment 5 errata-xmlrpc 2022-08-10 11:09:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069