Bug 1958296

Summary: OLM must explicitly alert on deprecated APIs in use
Product: OpenShift Container Platform Reporter: Kevin Rizza <krizza>
Component: OLMAssignee: Ben Luddy <bluddy>
OLM sub component: OLM QA Contact: xzha
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: bluddy, davegord, htariq, sttts, tflannag, xxia
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:07:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kevin Rizza 2021-05-07 15:02:20 UTC 
Description of problem:

After resolving all other know v1beta1 APIs that were being deprecated and updating them to v1 apis (as part of this BZ https://bugzilla.redhat.com/show_bug.cgi?id=1947794) the API server was still firing alerts for OLM as a result of v1beta1 crds still being in use. The level of granularity of those alerts is not valuable enough for an end user to actually be aware of Operators installed by OLM that are going to prevent cluster upgrade, so OLM must subsume those alerts when it installs and uses v1beta1 CRDs. Because the API server can only ignore these alerts at the granularity of a service account, OLM must also generically alert on any other deprecated API in use.

Version-Release number of selected component (if applicable):

4.8


How reproducible:

Always


Steps to Reproduce:
1. Install an operator that uses v1beta1crds

Actual results:

DeprecatedAPIInUse alert fires from the apiserver


Expected results:

Deprecated API message fires from OLM and points to the specific operator package that installed it.

Additional info:
Comment 1 Stefan Schimanski 2021-05-07 15:20:49 UTC 
Note that there are more APIs being removed in Kube 1.22, compare https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22. And even more in the following versions.

Please do not hardcode the APIs in your code, but rather use one of the generic ways to notice that an API is deprecated. For instance, every request against an API gives you headers in return that warn the client (kubectl prints these even to the user as warnings).
Comment 7 Kevin Rizza 2021-06-01 15:32:46 UTC 
*** Bug 1966508 has been marked as a duplicate of this bug. ***
Comment 10 Stefan Schimanski 2021-06-09 06:21:15 UTC 
The additional metrics/alerts don't block https://bugzilla.redhat.com/show_bug.cgi?id=1947719. Leaving the blocker+- judgement to the OLM team.
Comment 12 xzha 2021-06-11 07:16:38 UTC 
verify:

zhaoxia@xzha-mac community-operators % oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-10-224448   True        False         39m     Cluster version is 4.8.0-0.nightly-2021-06-10-224448
zhaoxia@xzha-mac community-operators % oc exec catalog-operator-6f7cfc6977-qgxqd -- olm --version 
OLM version: 0.17.0
git commit: 59934e50ebe3d59344cb8fe15f5fbba6b1b0219a

1) install strimzi-kafka-operator with channel strimzi-0.19.x
zhaoxia@xzha-mac community-operators % oc get sub
NAME                     PACKAGE                  SOURCE                CHANNEL
strimzi-kafka-operator   strimzi-kafka-operator   community-operators   strimzi-0.19.x
zhaoxia@xzha-mac community-operators % oc get csv 
NAME                               DISPLAY                            VERSION    REPLACES                           PHASE
elasticsearch-operator.5.1.0-52    OpenShift Elasticsearch Operator   5.1.0-52                                      Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi                            0.19.0     strimzi-cluster-operator.v0.18.0   Succeeded

zhaoxia@xzha-mac community-operators % oc api-resources | grep v1beta1 | grep str                                                                                          
kafkaconnects                         kc               kafka.strimzi.io/v1beta1                      true         KafkaConnect
kafkaconnects2is                      kcs2i            kafka.strimzi.io/v1beta1                      true         KafkaConnectS2I
kafkamirrormakers                     kmm              kafka.strimzi.io/v1beta1                      true         KafkaMirrorMaker
kafkas                                k                kafka.strimzi.io/v1beta1                      true         Kafka
kafkatopics                           kt               kafka.strimzi.io/v1beta1                      true         KafkaTopic
kafkausers                            ku               kafka.strimzi.io/v1beta1                      true         KafkaUser

2) check alert

https://user-images.githubusercontent.com/77608951/121646415-e0938900-cac7-11eb-9642-eec8122447da.png

there is no alert DeprecatedAPIInUse
Comment 13 Xingxing Xia 2021-06-15 08:15:28 UTC 
> there is no alert DeprecatedAPIInUse
Xia Zhao, DeprecatedAPIInUse is renamed as APIRemovedInNextReleaseInUse for months. Could you please re-check?
Comment 15 xzha 2021-06-16 02:57:27 UTC 
verify

[root@preserve-olm-agent-test ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-15-181825   True        False         96m     Cluster version is 4.8.0-0.nightly-2021-06-15-181825
[root@preserve-olm-agent-test ~]# oc exec  catalog-operator-5c8bb7d95f-82gwl -- olm --version 
OLM version: 0.17.0
git commit: f25f670c03e849ba0fd53a56daa0d8a697f68d16

1) install strimzi-kafka-operator with channel strimzi-0.19.x and 3scale-community-operator
[root@preserve-olm-agent-test ~]# oc get sub
NAME                        PACKAGE                     SOURCE                CHANNEL
3scale-community-operator   3scale-community-operator   community-operators   threescale-2.10
strimzi-kafka-operator      strimzi-kafka-operator      community-operators   strimzi-0.19.x
[root@preserve-olm-agent-test ~]# oc get csv
NAME                               DISPLAY                 VERSION   REPLACES                           PHASE
3scale-community-operator.v0.7.0   3scale API Management   0.7.0     3scale-community-operator.v0.6.0   Succeeded
strimzi-cluster-operator.v0.19.0   Strimzi                 0.19.0    strimzi-cluster-operator.v0.18.0   Succeeded
[root@preserve-olm-agent-test ~]# oc api-resources | grep v1beta1 | grep str 
kafkaconnects                         kc               kafka.strimzi.io/v1beta1                      true         KafkaConnect
kafkaconnects2is                      kcs2i            kafka.strimzi.io/v1beta1                      true         KafkaConnectS2I
kafkamirrormakers                     kmm              kafka.strimzi.io/v1beta1                      true         KafkaMirrorMaker
kafkas                                k                kafka.strimzi.io/v1beta1                      true         Kafka
kafkatopics                           kt               kafka.strimzi.io/v1beta1                      true         KafkaTopic
kafkausers                            ku               kafka.strimzi.io/v1beta1                      true         KafkaUser
[root@preserve-olm-agent-test ~]#  oc api-resources | grep v1beta1 | grep scale
machineautoscalers                    ma               autoscaling.openshift.io/v1beta1              true         MachineAutoscaler
backends                                               capabilities.3scale.net/v1beta1               true         Backend
products                                               capabilities.3scale.net/v1beta1               true         Product

2) check alert, OLM do not report APIRemovedInNextReleaseInUse alert, OLM report alert "InstallPlanStepAppliedWithWarnings"
[root@preserve-olm-agent-test ~]# curl -k -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)" https://alertmanager-main-openshift-monitoring.apps.ci-ln-qc7dj6b-f76d1.origin-ci-int-gce.dev.openshift.com/api/v1/alerts | jq | grep -i "APIRemovedInNext" -A2
        "alertname": "APIRemovedInNextReleaseInUse",
        "group": "apiextensions.k8s.io",
        "prometheus": "openshift-monitoring/k8s",
--
        "alertname": "APIRemovedInNextEUSReleaseInUse",
        "group": "apiextensions.k8s.io",
        "prometheus": "openshift-monitoring/k8s",

[root@preserve-olm-agent-test ~]# curl -k -H "Authorization: Bearer $(oc -n openshift-monitoring sa get-token prometheus-k8s)" https://alertmanager-main-openshift-monitoring.apps.ci-ln-qc7dj6b-f76d1.origin-ci-int-gce.dev.openshift.com/api/v1/alerts | jq | grep -i "InstallPlanStepAppliedWithWarnings" -A6
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5086    0  5086    0     0  43844      0 --:--:-- --:--:-- --:--:-- 44226
        "alertname": "InstallPlanStepAppliedWithWarnings",
        "prometheus": "openshift-monitoring/k8s",
        "severity": "warning"
      },
      "annotations": {
        "message": "The API server returned a warning during installation or upgrade of an operator. An Event with reason \"AppliedWithWarnings\" has been created with complete details, including a reference to the InstallPlan step that generated the warning."
      },


3) check event
[root@preserve-olm-agent-test ~]# oc get events --sort-by='.lastTimestamp' | grep AppliedWithWarnings
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkausers.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkaconnects.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkamirrormaker2s.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkas.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkaconnects2is.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkabridges.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkatopics.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkaconnectors.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                (combined from similar events): 1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkamirrormakers.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
85m         Warning   AppliedWithWarnings   installplan/install-x5plr                                1 warning(s) generated during installation of operator "strimzi-cluster-operator.v0.19.0" (CustomResourceDefinition "kafkarebalances.kafka.strimzi.io"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
45m         Warning   AppliedWithWarnings   installplan/install-fmg4l                                1 warning(s) generated during installation of operator "3scale-community-operator.v0.7.0" (CustomResourceDefinition "apimanagerrestores.apps.3scale.net"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
45m         Warning   AppliedWithWarnings   installplan/install-fmg4l                                1 warning(s) generated during installation of operator "3scale-community-operator.v0.7.0" (CustomResourceDefinition "products.capabilities.3scale.net"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
45m         Warning   AppliedWithWarnings   installplan/install-fmg4l                                1 warning(s) generated during installation of operator "3scale-community-operator.v0.7.0" (CustomResourceDefinition "apimanagerbackups.apps.3scale.net"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
45m         Warning   AppliedWithWarnings   installplan/install-fmg4l                                1 warning(s) generated during installation of operator "3scale-community-operator.v0.7.0" (CustomResourceDefinition "backends.capabilities.3scale.net"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
45m         Warning   AppliedWithWarnings   installplan/install-fmg4l                                1 warning(s) generated during installation of operator "3scale-community-operator.v0.7.0" (CustomResourceDefinition "tenants.capabilities.3scale.net"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
45m         Warning   AppliedWithWarnings   installplan/install-fmg4l                                1 warning(s) generated during installation of operator "3scale-community-operator.v0.7.0" (CustomResourceDefinition "apimanagers.apps.3scale.net"): apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition


lgtm, verifed.
Comment 17 errata-xmlrpc 2021-07-27 23:07:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438