Bug 2041540
| Summary: | RHACM 2.4 using deprecated APIs in managed clusters | ||
|---|---|---|---|
| Product: | Red Hat Advanced Cluster Management for Kubernetes | Reporter: | Simon Krenger <skrenger> |
| Component: | Console | Assignee: | Kevin Cormier <kcormier> |
| Status: | CLOSED ERRATA | QA Contact: | dhuynh |
| Severity: | high | Docs Contact: | Christopher Dawson <cdawson> |
| Priority: | high | ||
| Version: | rhacm-2.4 | CC: | apitt, ashafi, chuyang, cqu, daliu, dho, dhuynh, efried, huichen, jpadilla, juhsu, nagbetra, shaising, xiangli, yuhe |
| Target Milestone: | --- | Flags: | ashafi:
qe_test_coverage-
bot-tracker-sync: rhacm-2.4.z+ bot-tracker-sync: needinfo+ |
| Target Release: | rhacm-2.4.6 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-09-26 14:52:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
G2Bsync 1026278973 comment gparvin Mon, 31 Jan 2022 22:32:30 UTC G2Bsync The `cluster-policy-controller` identified in this report is not part of ACM GRC, but is part of OpenShift's base components: https://github.com/openshift/cluster-policy-controller The search-collector dynamically discovers all the APIs available in the cluster to index the resources. This process doesn't exclude deprecated APIs. When an API is removed we'll simply stop watching it and delete any indexed resources. What release were these changes delivered to? Was it ACM 2.4.3? @kcormier The Console, App, and Hive service accounts seem to be using deprecated APIs. Could you assign to the correct person to work on this? @efried Could you help to take a look about this comments https://bugzilla.redhat.com/show_bug.cgi?id=2041540#c9 It looks like hive use the deprecated APIs in ocm 2.4 branch. @daliu I'm looking at the tip of the ocm-2.4 branch and not seeing any use of apiextensions v1beta1. We did work, largely starting with https://github.com/openshift/hive/pull/1395, to remove these deprecated APIs. Can you please confirm the hive commit level built into the ACM in use here? @efried I could reproduce this using the latest ocm-2.4 hive. I could reproduce it with the following command, I will send you the env in slack. oc get apirequestcounts customresourcedefinitions.v1beta1.apiextensions.k8s.io -oyaml I checked the code (again) and we're definitely not using this API _directly_. It still exists buried in vendored dependencies though. Could it be an artifact of some k8s-y library validating that there are no such APIs in use? I even took a look on my cluster running the latest master hive, and am seeing these same apirequests in small numbers. ...and that cluster is running k8s 1.23 and OCP 4.10.6; so it can't actually be causing any problems. Is this really blocking an upgrade, or is it just cosmetic? Unless there's something performing an artificial check and blocking the upgrade, I am confident that these mysterious requests will not prevent you from running under OCP versions with post-1.22 k8s (at least for hive). Please let me know if that's not the case. I would still like to get to the bottom of the requests themselves, though. I think this is just cosmetic, as I haven't seen it actually block an upgrade. FYI I've opened https://issues.redhat.com/browse/HIVE-1911 for the hive team to investigate why these zombie apirequests still exist. But wrt this BZ I'm considering the matter resolved. Please let me know if that's not the case. Thanks! Could you please specify which APIs are deprecated? I created a new cluster and this output for api request count
oc get apirequestcounts ingresses.v1beta1.extensions -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
creationTimestamp: "2022-09-19T19:22:09Z"
generation: 1
name: ingresses.v1beta1.extensions
resourceVersion: "38465"
uid: 2ce02749-54a0-440c-aa24-60e9e7c30688
spec:
numberOfUsersToReport: 10
status:
currentHour:
byNode:
- byUser:
- byVerb:
- requestCount: 4
verb: watch
requestCount: 4
userAgent: cluster-policy-controller/v0.0.0
username: system:kube-controller-manager
nodeName: 10.0.152.44
requestCount: 4
- byUser:
- byVerb:
- requestCount: 2
verb: list
- requestCount: 5
verb: watch
requestCount: 7
userAgent: kube-controller-manager/v1.21.11+31d53a1
username: system:kube-controller-manager
- byVerb:
- requestCount: 3
verb: list
- requestCount: 3
verb: watch
requestCount: 6
userAgent: cluster-policy-controller/v0.0.0
username: system:kube-controller-manager
nodeName: 10.0.173.217
requestCount: 13
- byUser:
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: kube-controller-manager/v1.21.11+31d53a1
username: system:kube-controller-manager
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: main/v0.0.0
username: system:serviceaccount:open-cluster-management-agent-addon:klusterlet-addon-search
nodeName: 10.0.209.53
requestCount: 4
- byUser:
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: cluster-policy-controller/v0.0.0
username: system:admin
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: kube-controller-manager/v1.21.11+31d53a1
username: system:admin
nodeName: 10.0.4.198
requestCount: 4
requestCount: 25
last24h:
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- byUser:
- byVerb:
- requestCount: 4
verb: watch
requestCount: 4
userAgent: cluster-policy-controller/v0.0.0
username: system:kube-controller-manager
nodeName: 10.0.152.44
requestCount: 4
- byUser:
- byVerb:
- requestCount: 2
verb: list
- requestCount: 5
verb: watch
requestCount: 7
userAgent: kube-controller-manager/v1.21.11+31d53a1
username: system:kube-controller-manager
- byVerb:
- requestCount: 3
verb: list
- requestCount: 3
verb: watch
requestCount: 6
userAgent: cluster-policy-controller/v0.0.0
username: system:kube-controller-manager
nodeName: 10.0.173.217
requestCount: 13
- byUser:
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: kube-controller-manager/v1.21.11+31d53a1
username: system:kube-controller-manager
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: main/v0.0.0
username: system:serviceaccount:open-cluster-management-agent-addon:klusterlet-addon-search
nodeName: 10.0.209.53
requestCount: 4
- byUser:
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: cluster-policy-controller/v0.0.0
username: system:admin
- byVerb:
- requestCount: 1
verb: list
- requestCount: 1
verb: watch
requestCount: 2
userAgent: kube-controller-manager/v1.21.11+31d53a1
username: system:admin
nodeName: 10.0.4.198
requestCount: 4
requestCount: 25
- requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
- byNode:
- nodeName: 10.0.152.44
requestCount: 0
- nodeName: 10.0.173.217
requestCount: 0
- nodeName: 10.0.209.53
requestCount: 0
- nodeName: 10.0.4.198
requestCount: 0
requestCount: 0
removedInRelease: "1.22"
requestCount: 25
Based on the output, there are no deprecated apis used now. Verified on v2.4.6 (2.2.13 --> 2.3.12 --> 2.4.6) OCP 4.10.32 acm-custom-registry:2.4.6-DOWNSTREAM-2022-09-12-21-17-47 (In reply to Napoco Agbetra from comment #21) > Verified on v2.4.6 (2.2.13 --> 2.3.12 --> 2.4.6) OCP 4.8.49 > acm-custom-registry:2.4.6-DOWNSTREAM-2022-09-12-21-17-47 Verified on OCP 4.8.49 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Critical: Red Hat Advanced Cluster Management 2.4.6 security update and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:6696 |
G2Bsync 1020338140 comment xiangjingli Mon, 24 Jan 2022 17:12:27 UTC G2Bsync For app lifecycle, I have confirmed no deprecated APIs owned by app lifecycle is used. From the attached output for "oc get apirequestcounts ingresses.v1beta1.extensions -o yaml", it turns out the argocd-application-controller pod is using the deprecated api. ``` userAgent: argocd-application-controller/v0.0.0 username: system:serviceaccount:open-cluster-management-agent-addon:klusterlet-addon-appmgr ``` Please log a new issue for argocd team for their investigation.