Bug 1947794 - OLM: check (see bug 1947801#c4 steps) audit log to find deprecated API access related to this component to ensure this component does not trigger APIRemovedInNextReleaseInUse alert
Summary: OLM: check (see bug 1947801#c4 steps) audit log to find deprecated API access...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Kevin Rizza
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1947719 1952049
TreeView+ depends on / blocked
 
Reported: 2021-04-09 09:32 UTC by Stefan Schimanski
Modified: 2021-07-27 22:59 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1947719
: 1952049 1966508 (view as bug list)
Environment:
Last Closed: 2021-07-27 22:58:29 UTC
Target Upstream Version:


Attachments (Terms of Use)
Verified the alert (185.91 KB, image/png)
2021-04-20 22:01 UTC, Salvatore Colangelo
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift api pull 896 0 None open Bug 1947794: Update operatorhub config object to v1 2021-04-14 15:45:19 UTC
Github openshift api pull 907 0 None open Bug 1947794: HelmChartRepository CRD v1 API 2021-04-22 20:52:50 UTC
Github openshift console-operator pull 535 0 None open Bug 1947794: Pull HelmChartRepository CRD v1 2021-04-23 12:45:50 UTC
Github operator-framework operator-marketplace pull 390 0 None open Bug 1947794: Update openshift rolebindings to v1 2021-04-14 14:48:26 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:59:13 UTC

Internal Links: 1958296

Description Stefan Schimanski 2021-04-09 09:32:38 UTC
This component accesses APIs that will be removed in 4.9 (Kubernetes 1.22). It is causing the DeprecatedAPIInUse alert to fire in every 4.8 clusters permanently and hence must be fixed in 4.8 (blocker+).

The raw audit data can be found at https://gist.github.com/sttts/50a1429837f2448ce07f30174fa73cdb.

Here are the observed requests for this component:

system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/helmchartrepositories.helm.openshift.io
system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/operatorhubs.config.openshift.io

system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/marketplace-operator
system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-marketplace/rolebindings/marketplace-operator

+++ This bug was initially created as a clone of Bug #1947719 +++

Created attachment 1770482 [details]
alert screen shot

Created attachment 1770482 [details]
alert screen shot

Description of problem:
8 DeprecatedAPIInUse info alerts display

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-08-200632

How reproducible:
always

Steps to Reproduce:
1. open console-monitoring-alerts
2.
3.

Actual results:
8 DeprecatedAPIInUse info alerts display

Expected results:
No other alerts display except watchdog

Additional info:

alert rule metrics:
group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0

Element	Value:
{group="rbac.authorization.k8s.io",resource="roles",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="mutatingwebhookconfigurations",version="v1beta1"}	1
{group="admissionregistration.k8s.io",resource="validatingwebhookconfigurations",version="v1beta1"}	1
{group="apiextensions.k8s.io",resource="customresourcedefinitions",version="v1beta1"}	1
{group="certificates.k8s.io",resource="certificatesigningrequests",version="v1beta1"}	1
{group="extensions",resource="ingresses",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="clusterrolebindings",version="v1beta1"}	1
{group="rbac.authorization.k8s.io",resource="rolebindings",version="v1beta1"}	1

----------------
# for i in roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions certificatesigningrequests ingresses clusterrolebindings rolebindings; do oc api-resources | grep $i; echo -e "\n"; done
clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role
mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration
validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration
customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest
ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding

--- Additional comment from Junqi Zhao on 2021-04-09 05:28:56 CEST ---

alert details
alert:DeprecatedAPIInUse
expr:group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0
for: 1h
labels:
  severity: info
annotations:
  message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the audit logs to identify the workload.

--- Additional comment from hongyan li on 2021-04-09 05:37:17 CEST ---



--- Additional comment from hongyan li on 2021-04-09 05:44:46 CEST ---

Different issue from bug 1932165 which is about variable not translated to value

--- Additional comment from Junqi Zhao on 2021-04-09 06:04:30 CEST ---

# oc version
Client Version: 4.8.0-0.nightly-2021-04-08-200632
Server Version: 4.8.0-0.nightly-2021-04-08-200632
Kubernetes Version: v1.21.0-rc.0+6d27558

checked from prometheus, query parameter:
count(apiserver_requested_deprecated_apis{removed_release="1.22"}) by(instance,version,group,resource)
version is v1beta1
{group="certificates.k8s.io", instance="10.0.160.188:6443", resource="certificatesigningrequests", version="v1beta1"} 1
{group="extensions", instance="10.0.160.188:6443", resource="ingresses", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="clusterrolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="rolebindings", version="v1beta1"} 1
{group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="roles", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="mutatingwebhookconfigurations", version="v1beta1"} 1
{group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="validatingwebhookconfigurations", version="v1beta1"} 1
{group="apiextensions.k8s.io", instance="10.0.160.188:6443", resource="customresourcedefinitions", version="v1beta1"} 1

but the api versions are all actually v1, which means apiserver_requested_deprecated_apis may post the wrong result
# for i in certificatesigningrequests ingresses clusterrolebindings rolebindings roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions; do oc api-resources | grep $i; echo -e "\n"; done
certificatesigningrequests            csr              certificates.k8s.io/v1                        false        CertificateSigningRequest


ingresses                                              config.openshift.io/v1                        false        Ingress
ingresses                             ing              extensions/v1beta1                            true         Ingress
ingresses                             ing              networking.k8s.io/v1                          true         Ingress


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding


clusterrolebindings                                    authorization.openshift.io/v1                 false        ClusterRoleBinding
rolebindings                                           authorization.openshift.io/v1                 true         RoleBinding
clusterrolebindings                                    rbac.authorization.k8s.io/v1                  false        ClusterRoleBinding
rolebindings                                           rbac.authorization.k8s.io/v1                  true         RoleBinding


clusterroles                                           authorization.openshift.io/v1                 false        ClusterRole
roles                                                  authorization.openshift.io/v1                 true         Role
clusterroles                                           rbac.authorization.k8s.io/v1                  false        ClusterRole
roles                                                  rbac.authorization.k8s.io/v1                  true         Role


mutatingwebhookconfigurations                          admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration


validatingwebhookconfigurations                        admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration


customresourcedefinitions             crd,crds         apiextensions.k8s.io/v1                       false        CustomResourceDefinition

Comment 1 Kevin Rizza 2021-04-14 14:47:11 UTC
Hi Stefan,

I'm going to work on getting these updated, but I don't believe that that Helm CRD /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/helmchartrepositories.helm.openshift.io is owned or managed by any of our components. I'm not sure where it came from.

Comment 3 Salvatore Colangelo 2021-04-20 22:01:48 UTC
Created attachment 1773949 [details]
Verified the alert

Comment 4 Salvatore Colangelo 2021-04-20 22:22:02 UTC
The alert still remain as see in the attachement

Alert From "DeprecatedAPIinUse" to "APIRemovedInNextReleasiInUse" but the alert is present



[scolange@scolange go]$ oc -n openshift-monitoring get routes
NAME                HOST/PORT                                                                            PATH   SERVICES            PORT    TERMINATION          WILDCARD
alertmanager-main   alertmanager-main-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com          alertmanager-main   web     reencrypt/Redirect   None
grafana             grafana-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com                    grafana             https   reencrypt/Redirect   None
prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com             prometheus-k8s      web     reencrypt/Redirect   None
thanos-querier      thanos-querier-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com             thanos-querier      web     reencrypt/Redirect   None

[scolange@scolange go]$ curl -k -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)"  https://alertmanager-main-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com/api/v1/alerts

{"status":"success","data":[{"labels":{"alertname":"AlertmanagerReceiversNotConfigured","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"message":"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager."},"startsAt":"2021-04-20T20:56:52.563Z","endsAt":"2021-04-20T22:12:52.563Z","generatorURL":"https://prometheus-k8s-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com/graph?g0.expr=cluster%3Aalertmanager_routing_enabled%3Amax+%3D%3D+0\u0026g0.tab=1","status":{"state":"active","silencedBy":[],"inhibitedBy":[]},"receivers":["Default"],"fingerprint":"14298351083980ef"},{"labels":{"alertname":"APIRemovedInNextReleaseInUse","group":"apiextensions.k8s.io","prometheus":"openshift-monitoring/k8s","resource":"customresourcedefinitions","severity":"info","version":"v1beta1"},"annotations":{"message":"Deprecated API ......



Could you please check?

Comment 5 Kevin Rizza 2021-04-21 12:15:58 UTC
This bug will not by itself resolve the alert. The alert will only disappear when all related bugs for each OCP component that is tagged in the set of bugs produced by the API server team are resolved. The resolution from the OLM team was just to update the following APIs to v1:


system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/operatorhubs.config.openshift.io

system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/marketplace-operator
system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-marketplace/rolebindings/marketplace-operator

As long as the cluster has those object versions updated to v1 this bug should be verified.

Comment 7 Jian Zhang 2021-04-23 02:20:22 UTC
I'm not sure why the Helm fixed PR(https://github.com/openshift/api/pull/907) add in this bug. Anyway, I change the status to POST since this helm PR hasn't been merged.

Comment 9 Kevin Rizza 2021-04-26 18:04:44 UTC
Seems like the helm folks didn't create their PR with the cloned bz I created for them: https://bugzilla.redhat.com/show_bug.cgi?id=1952049

I would say we should modify that pull request and point it to another bug, but it does seem like it's about to merge so it seems like it's lower effort for us to just let it merge.

I do question what the QE ownership of that is, but in this case the test should be very straightforward.

Comment 10 Ke Wang 2021-04-27 11:07:02 UTC
The requests of the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1947785#c0 are gone for the given component, you won't see the related alert in web-console. Verification steps, you can refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4

Comment 12 Jian Zhang 2021-04-28 01:24:29 UTC
Hi Kevin,

> I would say we should modify that pull request and point it to another bug, but it does seem like it's about to merge so it seems like it's lower effort for us to just let it merge.
 I do question what the QE ownership of that is, but in this case the test should be very straightforward.

Yes, thanks for the explanation. I know it's a low effort for us to verify this bug. Even if it's a big effort, if you needed, we(QE) can still do the work. I just curious why the Helm PR was here, I thought we(OLM) would take responsibility for fixing Helm issues in the future, it was confusing. Besides, if one bug failed QA, it should be changed to "ASSIGNED" status, not "NEW". 

Hi Ke,

Thanks for your information!


Verify steps:
cluster version is 4.8.0-0.nightly-2021-04-26-151924

1, Get the alert route.
[jzhang@dhcp-140-36 ~]$ oc -n openshift-monitoring get routes
NAME                HOST/PORT                                                                              PATH   SERVICES            PORT    TERMINATION          WILDCARD
alertmanager-main   alertmanager-main-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com          alertmanager-main   web     reencrypt/Redirect   None
grafana             grafana-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com                    grafana             https   reencrypt/Redirect   None
prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com             prometheus-k8s      web     reencrypt/Redirect   None
thanos-querier      thanos-querier-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com             thanos-querier      web     reencrypt/Redirect   None

2, Check the "DeprecatedAPIInUse" alert.
[jzhang@dhcp-140-36 ~]$ curl -k -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)" https://alertmanager-main-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com/api/v1/alerts | jq | grep -i "DeprecatedAPIInUse" 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4644    0  4644    0     0   2672      0 --:--:--  0:00:01 --:--:--  2670

I didn't find any "DeprecatedAPIInUse" alert, LGTM, verify it.

Comment 13 Ke Wang 2021-04-28 09:27:29 UTC
@jiazha@redhat.com, there is a bug 1949593 - rename DeprecatedAPIInUse alert to APIRemovedInNextReleaseInUse, so you cannot see DeprecatedAPIInUse alert.

Comment 14 Ke Wang 2021-04-28 12:34:50 UTC
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-28-090319   True        False         74m     Cluster version is 4.8.0-0.nightly-2021-04-28-090319

$ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')

$ oc adm node-logs $masters --path=kube-apiserver/audit.log --raw | grep -e '"k8s.io/removed-release":"1.22"' | tee dep.json

$ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep -E 'operatorhubs|helm|marketplace'
Nothing can be found.

No longer see the requests of the bug comment#0

Comment 15 Jian Zhang 2021-04-30 02:54:16 UTC
Hi Ke,

Thanks for your information! Added it and no found the related OLM, Helm alerts.
[jzhang@dhcp-140-36 ~]$ curl -sk -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)" https://alertmanager-main-openshift-monitoring.apps.jiazha29.qe.devcluster.openshift.com/api/v1/alerts | jq |grep -i "APIRemovedInNextReleaseInUse" -A2
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "apiextensions.k8s.io",
        "prometheus": "openshift-monitoring/k8s",
--
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "admissionregistration.k8s.io",
        "prometheus": "openshift-monitoring/k8s",
--
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "rbac.authorization.k8s.io",
        "prometheus": "openshift-monitoring/k8s",
--
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "rbac.authorization.k8s.io",
        "prometheus": "openshift-monitoring/k8s",
--
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "admissionregistration.k8s.io",
        "prometheus": "openshift-monitoring/k8s",
--
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "rbac.authorization.k8s.io",
        "prometheus": "openshift-monitoring/k8s",
--
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "extensions",
        "prometheus": "openshift-monitoring/k8s",

Comment 16 Stefan Schimanski 2021-05-07 10:24:33 UTC
Reopening. We still see:

user/system:serviceaccount:openshift-operator-lifecycle-manager:olm-operator-serviceaccount accessed customresourcedefinitions.v1beta1.apiextensions.k8s.io 20 times

in [sig-arch][Late] clients should not use APIs that are removed in upcoming releases [Suite:openshift/conformance/parallel].

Comment 17 Kevin Rizza 2021-05-07 14:56:03 UTC
Per this conversation: https://coreos.slack.com/archives/CB48XQ4KZ/p1620383097397200

I am closing this and re marking as VERIFIED. OLM needs to be able to write v1beta1 crds for the 4.8 release, and it appears that OLM will need to subsume the alerting in that case. Creating a separate BZ to track that issue.

Comment 20 errata-xmlrpc 2021-07-27 22:58:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.