Bug 1947789

Summary: Console: check (see bug 1947801#c4 steps) audit log to find deprecated API access related to this component to ensure this component won't access APIs that trigger APIRemovedInNextReleaseInUse alert
Product: OpenShift Container Platform Reporter: Stefan Schimanski <sttts>
Component: Management ConsoleAssignee: Jakub Hadvig <jhadvig>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: high Docs Contact:
Priority: high    
Version: 4.8CC: alegrand, anpicker, aos-bugs, erooth, hongyli, jhadvig, jokerman, juzhao, kakkoyun, kewang, lcosic, mfojtik, pkrupa, schituku, spadgett, surbania, xxia, yapei
Target Milestone: ---Keywords: Reopened
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1947719 Environment:
Last Closed: 2021-07-27 22:58:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1947719    
Attachments:
Description Flags
Api removed in next release
none
Alert not referring to a resource none

Description Stefan Schimanski 2021-04-09 09:23:36 UTC
This component accesses APIs that will be removed in 4.9 (Kubernetes 1.22). It is causing the DeprecatedAPIInUse alert to fire in every 4.8 clusters permanently and hence must be fixed in 4.8 (blocker+).

The raw audit data can be found at https://gist.github.com/sttts/50a1429837f2448ce07f30174fa73cdb.

Here are the observed requests for this component:

system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consoleclidownloads.console.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consoleexternalloglinks.console.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consolelinks.console.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consolenotifications.console.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consoleplugins.console.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consolequickstarts.console.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consoles.config.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/consoleyamlsamples.console.openshift.io

+++ This bug was initially created as a clone of Bug #1947719 +++

Created attachment 1770482 [details]
alert screen shot

Created attachment 1770482 [details]
alert screen shot

Description of problem:
8 DeprecatedAPIInUse info alerts display

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-08-200632

How reproducible:
always

Steps to Reproduce:
1. open console-monitoring-alerts
2.
3.

Actual results:
8 DeprecatedAPIInUse info alerts display

Expected results:
No other alerts display except watchdog

Additional info:

alert rule metrics:
group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0

Element	Value:
{group="rbac.authorization.k8s.io",resource="roles",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="mutatingwebhookconfigurations",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="validatingwebhookconfigurations",version="v1beta1"}	1
{group="apiextensions.k8s.io",resource="customresourcedefinitions",version="v1beta1"}	1
{group="certificates.k8s.io",resource="certificatesigningrequests",version="v1beta1"}	1
{group="extensions",resource="ingresses",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="clusterrolebindings",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="rolebindings",version="v1beta1"}	1

----------------
# for i in roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions certificatesigningrequests ingresses clusterrolebindings rolebindings; do oc api-resources | grep $i; echo -e "\n"; done
clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role
mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration
validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration
customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest
ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding

--- Additional comment from Junqi Zhao on 2021-04-09 05:28:56 CEST ---

alert details
alert:DeprecatedAPIInUse
expr:group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0
for: 1h
labels:
  severity: info
annotations:
  message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the audit logs to identify the workload.

--- Additional comment from hongyan li on 2021-04-09 05:37:17 CEST ---



--- Additional comment from hongyan li on 2021-04-09 05:44:46 CEST ---

Different issue from bug 1932165 which is about variable not translated to value

--- Additional comment from Junqi Zhao on 2021-04-09 06:04:30 CEST ---

# oc version
Client Version: 4.8.0-0.nightly-2021-04-08-200632
Server Version: 4.8.0-0.nightly-2021-04-08-200632
Kubernetes Version: v1.21.0-rc.0+6d27558

checked from prometheus, query parameter:
count(apiserver_requested_deprecated_apis{removed_release="1.22"}) by(instance,version,group,resource)
version is v1beta1
{group="certificates.k8s.io", instance="10.0.160.188:6443", resource="certificatesigningrequests", version="v1beta1"} 1
{group="extensions", instance="10.0.160.188:6443", resource="ingresses", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="clusterrolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="rolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="roles", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="mutatingwebhookconfigurations", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="validatingwebhookconfigurations", version="v1beta1"} 1
{group="apiextensions.k8s.io", instance="10.0.160.188:6443", resource="customresourcedefinitions", version="v1beta1"} 1

but the api versions are all actually v1, which means apiserver_requested_deprecated_apis may post the wrong result
# for i in certificatesigningrequests ingresses clusterrolebindings rolebindings roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions; do oc api-resources | grep $i; echo -e "\n"; done
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest


ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding


clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role


mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration


validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration


customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition

Comment 2 Siva Reddy 2021-04-20 13:44:16 UTC
Now the deprecatedAPIInUse alerts are not displayed in the alerts

Version:
 4.8.0-0.nightly-2021-04-18-101412

Comment 3 Ke Wang 2021-04-27 11:01:16 UTC
@schituku, I still can see the alerts in web-console, the requests of the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1947785#c0 are gone for the given component, you won't see the related alert in web-console.

Comment 4 Ke Wang 2021-04-27 11:03:52 UTC
Verification steps, you can refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4

Comment 5 Siva Reddy 2021-04-28 22:36:33 UTC
Created attachment 1776939 [details]
Api removed in next release

version:
  4.8.0-0.nightly-2021-04-26-151924

  The initial verification was done for "DeprecatedAPIInUse" alerts and these are not there any more but still the overall issue is not resolved as there are still 
"APIRemovedInNextReleaseInUse" 
   so re-opening the bug 

   attached screen shot

Comment 6 Samuel Padgett 2021-04-29 21:34:17 UTC
Siva, can you tell us what APIs specifically are being alerted on? Are they for resources in the console.openshift.io API group or another?

Comment 7 Ke Wang 2021-04-30 03:29:32 UTC
There is a bug 1949593 - rename DeprecatedAPIInUse alert to APIRemovedInNextReleaseInUse.

Comment 8 Jakub Hadvig 2021-05-03 12:06:18 UTC
Does that mean that there is an naming issue? or that we are missing moving any resources from v1beta1 to v1?
Im a bit confused here.

Comment 9 Jakub Hadvig 2021-05-03 12:16:19 UTC
Created attachment 1778952 [details]
Alert not referring to a resource

Also when checking a `4.8.0-0.nightly-2021-04-30-102231` cluster the alert does not make any sense to me, since its not referring to any particular resource.

Comment 10 Jakub Hadvig 2021-05-03 13:38:03 UTC
Based on discussion with Stefan, the alert won't go away until all components in the system have been updated their manifests.

For verification we should look into audit logs:
```
$ oc adm node-logs ci-ln-cp5t82t-f76d1-v8r86-master-{0,1,2} --path=kube-apiserver/audit.log --raw | grep -e '"k8s.io/removed-release":"1.22"' > audit.json
$ cat audit.json | jq -r '.user.username+": "+.requestURI' | sort | uniq
```
and check that the console API requests are gone.

Comment 13 Ke Wang 2021-05-13 07:58:45 UTC
Refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4, tried to verify on latest OCP4.8,

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-05-13-002125   True        False         76m     Cluster version is 4.8.0-0.nightly-2021-05-13-002125

$ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')

$ oc adm node-logs $masters --path=kube-apiserver/audit.log --raw | grep -e '"k8s.io/removed-release":"1.22"' | tee dep.json

$ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep 'console'
No results found.

Comment 16 errata-xmlrpc 2021-07-27 22:58:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Comment 17 Red Hat Bugzilla 2023-09-15 01:04:52 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days