This component accesses APIs that will be removed in 4.9 (Kubernetes 1.22). It is causing the DeprecatedAPIInUse alert to fire in every 4.8 clusters permanently and hence must be fixed in 4.8 (blocker+). The raw audit data can be found at https://gist.github.com/sttts/50a1429837f2448ce07f30174fa73cdb. Here are the observed requests for this component: system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/helmchartrepositories.helm.openshift.io system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/operatorhubs.config.openshift.io system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/marketplace-operator system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-marketplace/rolebindings/marketplace-operator +++ This bug was initially created as a clone of Bug #1947719 +++ Created attachment 1770482 [details] alert screen shot Created attachment 1770482 [details] alert screen shot Description of problem: 8 DeprecatedAPIInUse info alerts display Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-04-08-200632 How reproducible: always Steps to Reproduce: 1. open console-monitoring-alerts 2. 3. Actual results: 8 DeprecatedAPIInUse info alerts display Expected results: No other alerts display except watchdog Additional info: alert rule metrics: group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0 Element Value: {group="rbac.authorization.k8s.io",resource="roles",version="v1beta1"} 1 {group="admissionregistration.k8s.io",resource="mutatingwebhookconfigurations",version="v1beta1"} 1 {group="admissionregistration.k8s.io",resource="validatingwebhookconfigurations",version="v1beta1"} 1 {group="apiextensions.k8s.io",resource="customresourcedefinitions",version="v1beta1"} 1 {group="certificates.k8s.io",resource="certificatesigningrequests",version="v1beta1"} 1 {group="extensions",resource="ingresses",version="v1beta1"} 1 {group="rbac.authorization.k8s.io",resource="clusterrolebindings",version="v1beta1"} 1 {group="rbac.authorization.k8s.io",resource="rolebindings",version="v1beta1"} 1 ---------------- # for i in roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions certificatesigningrequests ingresses clusterrolebindings rolebindings; do oc api-resources | grep $i; echo -e "\n"; done clusterroles authorization.openshift.io/v1 false ClusterRole roles authorization.openshift.io/v1 true Role clusterroles rbac.authorization.k8s.io/v1 false ClusterRole roles rbac.authorization.k8s.io/v1 true Role mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest ingresses config.openshift.io/v1 false Ingress ingresses ing extensions/v1beta1 true Ingress ingresses ing networking.k8s.io/v1 true Ingress clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding rolebindings authorization.openshift.io/v1 true RoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding rolebindings rbac.authorization.k8s.io/v1 true RoleBinding --- Additional comment from Junqi Zhao on 2021-04-09 05:28:56 CEST --- alert details alert:DeprecatedAPIInUse expr:group by(group, version, resource) (apiserver_requested_deprecated_apis{removed_release="1.22"}) and (sum by(group, version, resource) (rate(apiserver_request_total[10m]))) > 0 for: 1h labels: severity: info annotations: message: Deprecated API that will be removed in the next version is being used. Removing the workload that is using the {{"{{$labels.group}}"}}.{{"{{$labels.version}}"}}/{{"{{$labels.resource}}"}} API might be necessary for a successful upgrade to the next cluster version. Refer to the audit logs to identify the workload. --- Additional comment from hongyan li on 2021-04-09 05:37:17 CEST --- --- Additional comment from hongyan li on 2021-04-09 05:44:46 CEST --- Different issue from bug 1932165 which is about variable not translated to value --- Additional comment from Junqi Zhao on 2021-04-09 06:04:30 CEST --- # oc version Client Version: 4.8.0-0.nightly-2021-04-08-200632 Server Version: 4.8.0-0.nightly-2021-04-08-200632 Kubernetes Version: v1.21.0-rc.0+6d27558 checked from prometheus, query parameter: count(apiserver_requested_deprecated_apis{removed_release="1.22"}) by(instance,version,group,resource) version is v1beta1 {group="certificates.k8s.io", instance="10.0.160.188:6443", resource="certificatesigningrequests", version="v1beta1"} 1 {group="extensions", instance="10.0.160.188:6443", resource="ingresses", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="clusterrolebindings", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="rolebindings", version="v1beta1"} 1 {group="rbac.authorization.k8s.io", instance="10.0.160.188:6443", resource="roles", version="v1beta1"} 1 {group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="mutatingwebhookconfigurations", version="v1beta1"} 1 {group="admissionregistration.k8s.io", instance="10.0.160.188:6443", resource="validatingwebhookconfigurations", version="v1beta1"} 1 {group="apiextensions.k8s.io", instance="10.0.160.188:6443", resource="customresourcedefinitions", version="v1beta1"} 1 but the api versions are all actually v1, which means apiserver_requested_deprecated_apis may post the wrong result # for i in certificatesigningrequests ingresses clusterrolebindings rolebindings roles mutatingwebhookconfigurations validatingwebhookconfigurations customresourcedefinitions; do oc api-resources | grep $i; echo -e "\n"; done certificatesigningrequests csr certificates.k8s.io/v1 false CertificateSigningRequest ingresses config.openshift.io/v1 false Ingress ingresses ing extensions/v1beta1 true Ingress ingresses ing networking.k8s.io/v1 true Ingress clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding clusterrolebindings authorization.openshift.io/v1 false ClusterRoleBinding rolebindings authorization.openshift.io/v1 true RoleBinding clusterrolebindings rbac.authorization.k8s.io/v1 false ClusterRoleBinding rolebindings rbac.authorization.k8s.io/v1 true RoleBinding clusterroles authorization.openshift.io/v1 false ClusterRole roles authorization.openshift.io/v1 true Role clusterroles rbac.authorization.k8s.io/v1 false ClusterRole roles rbac.authorization.k8s.io/v1 true Role mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition
Hi Stefan, I'm going to work on getting these updated, but I don't believe that that Helm CRD /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/helmchartrepositories.helm.openshift.io is owned or managed by any of our components. I'm not sure where it came from.
Created attachment 1773949 [details] Verified the alert
The alert still remain as see in the attachement Alert From "DeprecatedAPIinUse" to "APIRemovedInNextReleasiInUse" but the alert is present [scolange@scolange go]$ oc -n openshift-monitoring get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD alertmanager-main alertmanager-main-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com alertmanager-main web reencrypt/Redirect None grafana grafana-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com grafana https reencrypt/Redirect None prometheus-k8s prometheus-k8s-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com prometheus-k8s web reencrypt/Redirect None thanos-querier thanos-querier-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com thanos-querier web reencrypt/Redirect None [scolange@scolange go]$ curl -k -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)" https://alertmanager-main-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com/api/v1/alerts {"status":"success","data":[{"labels":{"alertname":"AlertmanagerReceiversNotConfigured","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"message":"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager."},"startsAt":"2021-04-20T20:56:52.563Z","endsAt":"2021-04-20T22:12:52.563Z","generatorURL":"https://prometheus-k8s-openshift-monitoring.apps.qeci-19375.qe.devcluster.openshift.com/graph?g0.expr=cluster%3Aalertmanager_routing_enabled%3Amax+%3D%3D+0\u0026g0.tab=1","status":{"state":"active","silencedBy":[],"inhibitedBy":[]},"receivers":["Default"],"fingerprint":"14298351083980ef"},{"labels":{"alertname":"APIRemovedInNextReleaseInUse","group":"apiextensions.k8s.io","prometheus":"openshift-monitoring/k8s","resource":"customresourcedefinitions","severity":"info","version":"v1beta1"},"annotations":{"message":"Deprecated API ...... Could you please check?
This bug will not by itself resolve the alert. The alert will only disappear when all related bugs for each OCP component that is tagged in the set of bugs produced by the API server team are resolved. The resolution from the OLM team was just to update the following APIs to v1: system:serviceaccount:openshift-cluster-version:default: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/operatorhubs.config.openshift.io system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/marketplace-operator system:serviceaccount:openshift-cluster-version:default: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/openshift-marketplace/rolebindings/marketplace-operator As long as the cluster has those object versions updated to v1 this bug should be verified.
I'm not sure why the Helm fixed PR(https://github.com/openshift/api/pull/907) add in this bug. Anyway, I change the status to POST since this helm PR hasn't been merged.
Seems like the helm folks didn't create their PR with the cloned bz I created for them: https://bugzilla.redhat.com/show_bug.cgi?id=1952049 I would say we should modify that pull request and point it to another bug, but it does seem like it's about to merge so it seems like it's lower effort for us to just let it merge. I do question what the QE ownership of that is, but in this case the test should be very straightforward.
The requests of the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1947785#c0 are gone for the given component, you won't see the related alert in web-console. Verification steps, you can refer to https://bugzilla.redhat.com/show_bug.cgi?id=1947801#c4
Hi Kevin, > I would say we should modify that pull request and point it to another bug, but it does seem like it's about to merge so it seems like it's lower effort for us to just let it merge. I do question what the QE ownership of that is, but in this case the test should be very straightforward. Yes, thanks for the explanation. I know it's a low effort for us to verify this bug. Even if it's a big effort, if you needed, we(QE) can still do the work. I just curious why the Helm PR was here, I thought we(OLM) would take responsibility for fixing Helm issues in the future, it was confusing. Besides, if one bug failed QA, it should be changed to "ASSIGNED" status, not "NEW". Hi Ke, Thanks for your information! Verify steps: cluster version is 4.8.0-0.nightly-2021-04-26-151924 1, Get the alert route. [jzhang@dhcp-140-36 ~]$ oc -n openshift-monitoring get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD alertmanager-main alertmanager-main-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com alertmanager-main web reencrypt/Redirect None grafana grafana-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com grafana https reencrypt/Redirect None prometheus-k8s prometheus-k8s-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com prometheus-k8s web reencrypt/Redirect None thanos-querier thanos-querier-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com thanos-querier web reencrypt/Redirect None 2, Check the "DeprecatedAPIInUse" alert. [jzhang@dhcp-140-36 ~]$ curl -k -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)" https://alertmanager-main-openshift-monitoring.apps.jianl-042801.qe.devcluster.openshift.com/api/v1/alerts | jq | grep -i "DeprecatedAPIInUse" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4644 0 4644 0 0 2672 0 --:--:-- 0:00:01 --:--:-- 2670 I didn't find any "DeprecatedAPIInUse" alert, LGTM, verify it.
@jiazha, there is a bug 1949593 - rename DeprecatedAPIInUse alert to APIRemovedInNextReleaseInUse, so you cannot see DeprecatedAPIInUse alert.
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-04-28-090319 True False 74m Cluster version is 4.8.0-0.nightly-2021-04-28-090319 $ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}') $ oc adm node-logs $masters --path=kube-apiserver/audit.log --raw | grep -e '"k8s.io/removed-release":"1.22"' | tee dep.json $ cat dep.json | jq -r '.user.username+": "+.requestURI' | sort | uniq | grep -E 'operatorhubs|helm|marketplace' Nothing can be found. No longer see the requests of the bug comment#0
Hi Ke, Thanks for your information! Added it and no found the related OLM, Helm alerts. [jzhang@dhcp-140-36 ~]$ curl -sk -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)" https://alertmanager-main-openshift-monitoring.apps.jiazha29.qe.devcluster.openshift.com/api/v1/alerts | jq |grep -i "APIRemovedInNextReleaseInUse" -A2 "alertname": "APIRemovedInNextReleaseInUse", "group": "apiextensions.k8s.io", "prometheus": "openshift-monitoring/k8s", -- "alertname": "APIRemovedInNextReleaseInUse", "group": "admissionregistration.k8s.io", "prometheus": "openshift-monitoring/k8s", -- "alertname": "APIRemovedInNextReleaseInUse", "group": "rbac.authorization.k8s.io", "prometheus": "openshift-monitoring/k8s", -- "alertname": "APIRemovedInNextReleaseInUse", "group": "rbac.authorization.k8s.io", "prometheus": "openshift-monitoring/k8s", -- "alertname": "APIRemovedInNextReleaseInUse", "group": "admissionregistration.k8s.io", "prometheus": "openshift-monitoring/k8s", -- "alertname": "APIRemovedInNextReleaseInUse", "group": "rbac.authorization.k8s.io", "prometheus": "openshift-monitoring/k8s", -- "alertname": "APIRemovedInNextReleaseInUse", "group": "extensions", "prometheus": "openshift-monitoring/k8s",
Reopening. We still see: user/system:serviceaccount:openshift-operator-lifecycle-manager:olm-operator-serviceaccount accessed customresourcedefinitions.v1beta1.apiextensions.k8s.io 20 times in [sig-arch][Late] clients should not use APIs that are removed in upcoming releases [Suite:openshift/conformance/parallel].
Per this conversation: https://coreos.slack.com/archives/CB48XQ4KZ/p1620383097397200 I am closing this and re marking as VERIFIED. OLM needs to be able to write v1beta1 crds for the 4.8 release, and it appears that OLM will need to subsume the alerting in that case. Creating a separate BZ to track that issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438