Bug 2092193

Summary: [4.10] APIRemovedInNextEUSReleaseInUse alert for OVN poddisruptionbudgets
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: NetworkingAssignee: obraunsh
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: anusaxen, mkennell, surya
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-15 15:44:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2079422    
Bug Blocks:    

Description Junqi Zhao 2022-06-01 04:04:44 UTC 
Description of problem:
upi-on-azure OVN 4.10.16 cluster, APIRemovedInNextEUSReleaseInUse alert fired for poddisruptionbudgets, checked the apirequest, it is from openshift-network-operator
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname="APIRemovedInNextEUSReleaseInUse",resource="poddisruptionbudgets"}' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "ALERTS",
          "alertname": "APIRemovedInNextEUSReleaseInUse",
          "alertstate": "firing",
          "environment": "Azure",
          "group": "policy",
          "namespace": "openshift-kube-apiserver",
          "prometheus": "openshift-monitoring/k8s",
          "region": "mtcazs",
          "resource": "poddisruptionbudgets",
          "severity": "info",
          "version": "v1beta1"
        },
        "value": [
          1654055786.192,
          "1"
        ]
      }
    ]
  }
}

# oc get apirequestcounts poddisruptionbudgets.v1beta1.policy -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-05-31T21:56:57Z"
  generation: 1
  name: poddisruptionbudgets.v1beta1.policy
  resourceVersion: "142700"
  uid: f38c8fa7-a9da-4945-b9b4-7ff0542ba9af
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 13
          verb: get
        - requestCount: 13
          verb: update
        requestCount: 26
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.8
      requestCount: 26
    requestCount: 26
  last24h:
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 20
          verb: get
        - requestCount: 20
          verb: update
        requestCount: 40
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.8
      requestCount: 40
    requestCount: 40
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 20
          verb: get
        - requestCount: 20
          verb: update
        requestCount: 40
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.8
      requestCount: 40
    requestCount: 40
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 20
          verb: get
        - requestCount: 20
          verb: update
        requestCount: 40
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.8
      requestCount: 40
    requestCount: 40
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 13
          verb: get
        - requestCount: 13
          verb: update
        requestCount: 26
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.8
      requestCount: 26
    requestCount: 26
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.0.5
      requestCount: 0
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: create
        - requestCount: 30
          verb: get
        - requestCount: 29
          verb: update
        requestCount: 60
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.5
      requestCount: 60
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - nodeName: 10.0.0.8
      requestCount: 0
    requestCount: 60
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 19
          verb: get
        - requestCount: 19
          verb: update
        requestCount: 38
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.6
      requestCount: 38
    - nodeName: 10.0.0.7
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 16
          verb: get
        - requestCount: 16
          verb: update
        requestCount: 32
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.8
      requestCount: 32
    requestCount: 70
  - byNode:
    - nodeName: 10.0.0.6
      requestCount: 0
    - nodeName: 10.0.0.7
      requestCount: 0
    - byUser:
      - byVerb:
        - requestCount: 20
          verb: get
        - requestCount: 20
          verb: update
        requestCount: 40
        userAgent: cluster-network-operator/v0.0.0
        username: system:serviceaccount:openshift-network-operator:default
      nodeName: 10.0.0.8
      requestCount: 40
    requestCount: 40
  removedInRelease: "1.25"
  requestCount: 316

# oc -n openshift-network-operator get po
NAME                                READY   STATUS    RESTARTS   AGE
network-operator-6ddb6d99bc-hlg2v   1/1     Running   0          6h4m

# oc -n openshift-network-operator logs network-operator-6ddb6d99bc-hlg2v | grep -i poddisruptionbudget | tail
I0601 03:26:58.392985       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:29:59.402334       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:33:00.425388       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:36:01.481744       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:39:02.516711       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:42:03.559903       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:45:04.593258       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:48:05.610182       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:51:06.642777       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard
I0601 03:54:07.719012       1 log.go:184] reconciling (policy/v1beta1, Kind=PodDisruptionBudget) openshift-ovn-kubernetes/ovn-raft-quorum-guard


Version-Release number of selected component (if applicable):
# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.16   True        False         5h33m   Cluster version is 4.10.16
# oc get network cluster -o jsonpath="{.spec.networkType}"
OVNKubernetes

How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
APIRemovedInNextEUSReleaseInUse alert fired for poddisruptionbudgets

Expected results:
no such alert

Additional info:
maybe caused by https://github.com/openshift/ovn-kubernetes/blob/release-4.10/go-controller/vendor/modules.txt#L388
Comment 5 Martin Kennelly 2023-01-30 16:21:59 UTC 
Anurag, we are looking to move this bz forward. The code is merged but we are on QA. Can you PTAL and advance this issues status?
Comment 11 errata-xmlrpc 2023-02-15 15:44:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.52 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0698