Bug 2041540 - RHACM 2.4 using deprecated APIs in managed clusters
Summary: RHACM 2.4 using deprecated APIs in managed clusters
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Advanced Cluster Management for Kubernetes
Classification: Red Hat
Component: Console
Version: rhacm-2.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: rhacm-2.4.6
Assignee: Kevin Cormier
QA Contact: dhuynh
Christopher Dawson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-17 16:25 UTC by Simon Krenger
Modified: 2022-09-26 14:53 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-26 14:52:19 UTC
Target Upstream Version:
ashafi: qe_test_coverage-
bot-tracker-sync: rhacm-2.4.z+
bot-tracker-sync: needinfo+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github stolostron backlog issues 19175 0 None None None 2022-01-17 17:03:32 UTC
Red Hat Issue Tracker GITOPS-1716 0 None Waiting on Red Hat After satellite 6.10 upgrade insights are not showing on host level recomendation 2022-06-27 13:40:02 UTC
Red Hat Product Errata RHSA-2022:6696 0 None None None 2022-09-26 14:53:33 UTC

Comment 3 bot-tracker-sync 2022-01-24 17:42:11 UTC 
G2Bsync 1020338140 comment 
 xiangjingli Mon, 24 Jan 2022 17:12:27 UTC 
 G2Bsync

For app lifecycle, I have confirmed no deprecated APIs  owned by app lifecycle is used. 

From the attached output for "oc get apirequestcounts ingresses.v1beta1.extensions -o yaml",  it turns out the argocd-application-controller pod is using the deprecated api. 
```
        userAgent: argocd-application-controller/v0.0.0
        username: system:serviceaccount:open-cluster-management-agent-addon:klusterlet-addon-appmgr
```
Please log a new issue for argocd team for their investigation.
Comment 5 bot-tracker-sync 2022-01-31 23:23:15 UTC 
G2Bsync 1026278973 comment 
 gparvin Mon, 31 Jan 2022 22:32:30 UTC 
 G2Bsync The `cluster-policy-controller` identified in this report is not part of ACM GRC, but is part of OpenShift's base components: https://github.com/openshift/cluster-policy-controller
Comment 7 Jorge Padilla 2022-02-10 01:24:48 UTC 
The search-collector dynamically discovers all the APIs available in the cluster to index the resources. This process doesn't exclude deprecated APIs. When an API is removed we'll simply stop watching it and delete any indexed resources.
Comment 8 juhsu 2022-05-09 18:09:38 UTC 
What release were these changes delivered to?  Was it ACM 2.4.3?
Comment 10 Jorge Padilla 2022-06-02 14:21:49 UTC 
@kcormier The Console, App, and Hive service accounts seem to be using deprecated APIs. Could you assign to the correct person to work on this?
Comment 11 daliu 2022-06-06 02:47:43 UTC 
@efried Could you help to take a look about this comments https://bugzilla.redhat.com/show_bug.cgi?id=2041540#c9
It looks like hive use the deprecated APIs in ocm 2.4 branch.
Comment 12 Eric Fried 2022-06-06 15:20:01 UTC 
@daliu I'm looking at the tip of the ocm-2.4 branch and not seeing any use of apiextensions v1beta1. We did work, largely starting with https://github.com/openshift/hive/pull/1395, to remove these deprecated APIs. Can you please confirm the hive commit level built into the ACM in use here?
Comment 13 daliu 2022-06-07 03:32:50 UTC 
@efried 
I could reproduce this using the latest ocm-2.4 hive.
I could reproduce it with the following command, I will send you the env in slack.

oc get apirequestcounts customresourcedefinitions.v1beta1.apiextensions.k8s.io -oyaml
Comment 14 Eric Fried 2022-06-07 22:00:25 UTC 
I checked the code (again) and we're definitely not using this API _directly_. It still exists buried in vendored dependencies though. Could it be an artifact of some k8s-y library validating that there are no such APIs in use?

I even took a look on my cluster running the latest master hive, and am seeing these same apirequests in small numbers.

...and that cluster is running k8s 1.23 and OCP 4.10.6; so it can't actually be causing any problems.

Is this really blocking an upgrade, or is it just cosmetic?
Comment 16 Eric Fried 2022-06-09 15:11:56 UTC 
Unless there's something performing an artificial check and blocking the upgrade, I am confident that these mysterious requests will not prevent you from running under OCP versions with post-1.22 k8s (at least for hive). Please let me know if that's not the case.

I would still like to get to the bottom of the requests themselves, though.
Comment 17 Andrew Pitt 2022-06-09 16:09:07 UTC 
I think this is just cosmetic, as I haven't seen it actually block an upgrade.
Comment 18 Eric Fried 2022-06-09 22:09:22 UTC 
FYI I've opened https://issues.redhat.com/browse/HIVE-1911 for the hive team to investigate why these zombie apirequests still exist. But wrt this BZ I'm considering the matter resolved. Please let me know if that's not the case. Thanks!
Comment 19 Napoco Agbetra 2022-09-19 20:40:22 UTC 
Could you please specify which APIs are deprecated? I created a new cluster and this output for api request count

oc get apirequestcounts ingresses.v1beta1.extensions -o yaml
apiVersion: apiserver.openshift.io/v1
kind: APIRequestCount
metadata:
  creationTimestamp: "2022-09-19T19:22:09Z"
  generation: 1
  name: ingresses.v1beta1.extensions
  resourceVersion: "38465"
  uid: 2ce02749-54a0-440c-aa24-60e9e7c30688
spec:
  numberOfUsersToReport: 10
status:
  currentHour:
    byNode:
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: cluster-policy-controller/v0.0.0
        username: system:kube-controller-manager
      nodeName: 10.0.152.44
      requestCount: 4
    - byUser:
      - byVerb:
        - requestCount: 2
          verb: list
        - requestCount: 5
          verb: watch
        requestCount: 7
        userAgent: kube-controller-manager/v1.21.11+31d53a1
        username: system:kube-controller-manager
      - byVerb:
        - requestCount: 3
          verb: list
        - requestCount: 3
          verb: watch
        requestCount: 6
        userAgent: cluster-policy-controller/v0.0.0
        username: system:kube-controller-manager
      nodeName: 10.0.173.217
      requestCount: 13
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: kube-controller-manager/v1.21.11+31d53a1
        username: system:kube-controller-manager
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: main/v0.0.0
        username: system:serviceaccount:open-cluster-management-agent-addon:klusterlet-addon-search
      nodeName: 10.0.209.53
      requestCount: 4
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: cluster-policy-controller/v0.0.0
        username: system:admin
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: kube-controller-manager/v1.21.11+31d53a1
        username: system:admin
      nodeName: 10.0.4.198
      requestCount: 4
    requestCount: 25
  last24h:
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - byUser:
      - byVerb:
        - requestCount: 4
          verb: watch
        requestCount: 4
        userAgent: cluster-policy-controller/v0.0.0
        username: system:kube-controller-manager
      nodeName: 10.0.152.44
      requestCount: 4
    - byUser:
      - byVerb:
        - requestCount: 2
          verb: list
        - requestCount: 5
          verb: watch
        requestCount: 7
        userAgent: kube-controller-manager/v1.21.11+31d53a1
        username: system:kube-controller-manager
      - byVerb:
        - requestCount: 3
          verb: list
        - requestCount: 3
          verb: watch
        requestCount: 6
        userAgent: cluster-policy-controller/v0.0.0
        username: system:kube-controller-manager
      nodeName: 10.0.173.217
      requestCount: 13
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: kube-controller-manager/v1.21.11+31d53a1
        username: system:kube-controller-manager
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: main/v0.0.0
        username: system:serviceaccount:open-cluster-management-agent-addon:klusterlet-addon-search
      nodeName: 10.0.209.53
      requestCount: 4
    - byUser:
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: cluster-policy-controller/v0.0.0
        username: system:admin
      - byVerb:
        - requestCount: 1
          verb: list
        - requestCount: 1
          verb: watch
        requestCount: 2
        userAgent: kube-controller-manager/v1.21.11+31d53a1
        username: system:admin
      nodeName: 10.0.4.198
      requestCount: 4
    requestCount: 25
  - requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  - byNode:
    - nodeName: 10.0.152.44
      requestCount: 0
    - nodeName: 10.0.173.217
      requestCount: 0
    - nodeName: 10.0.209.53
      requestCount: 0
    - nodeName: 10.0.4.198
      requestCount: 0
    requestCount: 0
  removedInRelease: "1.22"
  requestCount: 25
Comment 20 daliu 2022-09-20 01:07:05 UTC 
Based on the output, there are no deprecated apis used now.
Comment 21 Napoco Agbetra 2022-09-20 12:53:25 UTC 
Verified on v2.4.6 (2.2.13 --> 2.3.12 --> 2.4.6) OCP 4.10.32  acm-custom-registry:2.4.6-DOWNSTREAM-2022-09-12-21-17-47
Comment 22 Napoco Agbetra 2022-09-20 12:56:45 UTC 
(In reply to Napoco Agbetra from comment #21)
> Verified on v2.4.6 (2.2.13 --> 2.3.12 --> 2.4.6) OCP 4.8.49 
> acm-custom-registry:2.4.6-DOWNSTREAM-2022-09-12-21-17-47

Verified on OCP 4.8.49
Comment 27 errata-xmlrpc 2022-09-26 14:52:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Critical: Red Hat Advanced Cluster Management 2.4.6 security update and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6696
Note You need to log in before you can comment on or make changes to this bug.