Bug 1826372

Summary: Invalid CatalogSourceConfig does not trigger alert that deprecated API is in use
Product: OpenShift Container Platform Reporter: Shawn Hurley <shurley>
Component: OLMAssignee: Anik <anbhatta>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: anbhatta, ecordell, jiazha, nhale, schoudha
Version: 4.4   
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1826360 Environment:
Last Closed: 2020-06-29 15:33:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1826360    
Bug Blocks:    

Comment 1 Anik 2020-04-27 16:12:05 UTC 
https://github.com/operator-framework/operator-marketplace/pull/303
Comment 9 Jian Zhang 2020-06-22 10:02:44 UTC 
Cluster version is 4.4.0-0.nightly-2020-06-21-210301

Marketplace-operator version:
[root@preserve-olm-env data]#  oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-06-21-210301 |grep marketplace
  operator-marketplace                           https://github.com/operator-framework/operator-marketplace                  18ec31ba6b228dc63b637db840b6667580888d79

1, Create a CSC with bogus values
mac:~ jianzhang$ oc create -f csc-invalid.yaml 
catalogsourceconfig.operators.coreos.com/csc-invaild created

mac:~ jianzhang$ cat csc-invalid.yaml 
apiVersion: operators.coreos.com/v1
kind: CatalogSourceConfig
metadata:
  name: csc-invaild
  namespace: openshift-marketplace
spec:
  source: ""
  packages: etcd-test
  targetNamespace: openshift-operators
  csDisplayName: CSC Operators
  csPublisher: CSC

mac:~ jianzhang$ oc get csc
NAME          STATUS        MESSAGE                                                                                  AGE
csc-invaild   Configuring   Unable to resolve the source - no source contains the requested package(s) [etcd-test]   8m36s

Check the Prometheus alert, but I couldn't find the related CSC alert. 
Maybe the above CSC object is inappropriate. If yes, could you help show an example with me? @Shawn@Anik Thanks!

mac:~ jianzhang$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
mac:~ jianzhang$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/alerts' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5223    0  5223    0     0  86107      0 --:--:-- --:--:-- --:--:-- 87050
{
  "status": "success",
  "data": {
    "alerts": [
      {
        "labels": {
          "alertname": "ServiceCatalogAPIServerEnabled",
          "endpoint": "https",
          "instance": "10.128.0.4:8443",
          "job": "metrics",
          "namespace": "openshift-service-catalog-apiserver-operator",
          "pod": "openshift-service-catalog-apiserver-operator-84594497f5-9ck8j",
          "service": "metrics",
          "severity": "warning"
        },
        "annotations": {
          "message": "Indicates whether Service Catalog API Server is enabled",
          "summary": "Indicates whether Service Catalog API Server is enabled"
        },
        "state": "firing",
        "activeAt": "2020-06-22T03:24:21.310548511Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "ServiceCatalogControllerManagerEnabled",
          "endpoint": "https",
          "instance": "10.128.0.9:8443",
          "job": "metrics",
          "namespace": "openshift-service-catalog-controller-manager-operator",
          "pod": "openshift-service-catalog-controller-manager-operator-74945j729",
          "service": "metrics",
          "severity": "warning"
        },
        "annotations": {
          "message": "Indicates whether Service Catalog Controller Manager is enabled",
          "summary": "Indicates whether Service Catalog Controller Manager is enabled"
        },
        "state": "firing",
        "activeAt": "2020-06-22T03:24:32.506627385Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "CustomResourceDetected",
          "customResourceType": "OperatorSource",
          "endpoint": "https-metrics",
          "instance": "10.129.0.23:8081",
          "job": "marketplace-operator-metrics",
          "namespace": "openshift-marketplace",
          "pod": "marketplace-operator-664774d49-jhqkf",
          "service": "marketplace-operator-metrics",
          "severity": "warning"
        },
        "annotations": {
          "message": "The cluster has custom OperatorSource/CatalogSourceConfig, which are deprecated from OCP 4.5. Upgrade to OCP 4.5 will not be possible unless those resources are removed. Please visit https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated for further details.",
          "summary": "Indicates the presence of a custom OperatorSource or a CatalogSourceConifg in the cluster"
        },
        "state": "firing",
        "activeAt": "2020-06-22T03:24:13.459916592Z",
        "value": "3e+00"
      },
      {
        "labels": {
          "alertname": "CustomResourceDetected",
          "customResourceType": "CatalogSourceConfig",
          "endpoint": "https-metrics",
          "instance": "10.129.0.23:8081",
          "job": "marketplace-operator-metrics",
          "namespace": "openshift-marketplace",
          "pod": "marketplace-operator-664774d49-jhqkf",
          "service": "marketplace-operator-metrics",
          "severity": "warning"
        },
        "annotations": {
          "message": "The cluster has custom OperatorSource/CatalogSourceConfig, which are deprecated from OCP 4.5. Upgrade to OCP 4.5 will not be possible unless those resources are removed. Please visit https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated for further details.",
          "summary": "Indicates the presence of a custom OperatorSource or a CatalogSourceConifg in the cluster"
        },
        "state": "firing",
        "activeAt": "2020-06-22T09:31:13.459916592Z",
        "value": "2.7e+01"
      },
      {
        "labels": {
          "alertname": "Watchdog",
          "severity": "none"
        },
        "annotations": {
          "message": "This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n"
        },
        "state": "firing",
        "activeAt": "2020-06-22T03:20:30.163677339Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "TemplateServiceBrokerEnabled",
          "endpoint": "cr-metrics",
          "exported_namespace": "openshift-template-service-broker",
          "instance": "10.131.0.16:8686",
          "job": "openshift-template-service-broker-operator-metrics",
          "namespace": "openshift-template-service-broker",
          "pod": "openshift-template-service-broker-operator-64b47bd668-wsd8k",
          "service": "openshift-template-service-broker-operator-metrics",
          "severity": "warning",
          "templateservicebroker": "template-service-broker"
        },
        "annotations": {
          "summary": "Indicates whether the Template Service Broker is enabled"
        },
        "state": "firing",
        "activeAt": "2020-06-22T03:32:46.217512756Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "ImagePruningDisabled",
          "endpoint": "60000",
          "instance": "10.129.0.18:60000",
          "job": "image-registry-operator",
          "namespace": "openshift-image-registry",
          "pod": "cluster-image-registry-operator-765df5b6f9-dhzb8",
          "service": "image-registry-operator",
          "severity": "warning"
        },
        "annotations": {
          "message": "Automatic image pruning is not enabled. Regular pruning of images\nno longer referenced by ImageStreams is strongly recommended to\nensure your cluster remains healthy.\n\nTo remove this warning, install the image pruner by creating an\nimagepruner.imageregistry.operator.openshift.io resource with the\nname `cluster`. Ensure that the `suspend` field is set to `false`.\n"
        },
        "state": "firing",
        "activeAt": "2020-06-22T03:22:38.020409993Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "AlertmanagerReceiversNotConfigured",
          "severity": "warning"
        },
        "annotations": {
          "message": "Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager."
        },
        "state": "firing",
        "activeAt": "2020-06-22T03:22:36.662770393Z",
        "value": "0e+00"
      }
    ]
  }
}
Comment 10 Anik 2020-06-25 22:01:27 UTC 
Hi Jian,

This bug fix was accepted into a z stream release. Notice the Target Release 4.4.z

I noticed that you were using a 4.4.0 cluster for testing this. 

Just to double check I deployed a image I built with the commit the PR this bz is attached to introduced, and I can see the alert: 

$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/alerts' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3991    0  3991    0     0  73359      0 --:--:-- --:--:-- --:--:-- 73907
{
  "status": "success",
  "data": {
    "alerts": [
      {
        "labels": {
          "alertname": "KubePodNotReady",
          "namespace": "openshift-marketplace",
          "pod": "marketplace-operator-f7f869c8b-p6bwg",
          "severity": "critical"
        },
        "annotations": {
          "message": "Pod openshift-marketplace/marketplace-operator-f7f869c8b-p6bwg has been in a non-ready state for longer than 15 minutes."
        },
        "state": "pending",
        "activeAt": "2020-06-25T21:51:10.59085788Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "KubeDeploymentReplicasMismatch",
          "deployment": "marketplace-operator",
          "endpoint": "https-main",
          "instance": "10.128.2.8:8443",
          "job": "kube-state-metrics",
          "namespace": "openshift-marketplace",
          "pod": "kube-state-metrics-7c95677f97-sh6fc",
          "service": "kube-state-metrics",
          "severity": "critical"
        },
        "annotations": {
          "message": "Deployment openshift-marketplace/marketplace-operator has not matched the expected number of replicas for longer than 15 minutes."
        },
        "state": "pending",
        "activeAt": "2020-06-25T21:51:10.59085788Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "KubeContainerWaiting",
          "container": "marketplace-operator",
          "namespace": "openshift-marketplace",
          "pod": "marketplace-operator-f7f869c8b-p6bwg",
          "severity": "warning"
        },
        "annotations": {
          "message": "Pod openshift-marketplace/marketplace-operator-f7f869c8b-p6bwg container marketplace-operator has been in waiting state for longer than 1 hour."
        },
        "state": "pending",
        "activeAt": "2020-06-25T21:51:10.59085788Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "AlertmanagerReceiversNotConfigured",
          "severity": "warning"
        },
        "annotations": {
          "message": "Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager."
        },
        "state": "firing",
        "activeAt": "2020-06-25T19:34:06.662770393Z",
        "value": "0e+00"
      },
      {
        "labels": {
          "alertname": "ImagePruningDisabled",
          "endpoint": "60000",
          "instance": "10.130.0.17:60000",
          "job": "image-registry-operator",
          "namespace": "openshift-image-registry",
          "pod": "cluster-image-registry-operator-6878bf8f75-wx99n",
          "service": "image-registry-operator",
          "severity": "warning"
        },
        "annotations": {
          "message": "Automatic image pruning is not enabled. Regular pruning of images\nno longer referenced by ImageStreams is strongly recommended to\nensure your cluster remains healthy.\n\nTo remove this warning, install the image pruner by creating an\nimagepruner.imageregistry.operator.openshift.io resource with the\nname `cluster`. Ensure that the `suspend` field is set to `false`.\n"
        },
        "state": "firing",
        "activeAt": "2020-06-25T19:34:08.020409993Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "Watchdog",
          "severity": "none"
        },
        "annotations": {
          "message": "This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n"
        },
        "state": "firing",
        "activeAt": "2020-06-25T19:33:30.163677339Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "CustomResourceDetected",
          "customResourceType": "CatalogSourceConfig",
          "endpoint": "https-metrics",
          "instance": "10.129.0.48:8081",
          "job": "marketplace-operator-metrics",
          "namespace": "openshift-marketplace",
          "pod": "marketplace-operator-f7f869c8b-gbjjb",
          "service": "marketplace-operator-metrics",
          "severity": "warning"
        },
        "annotations": {
          "message": "The cluster has custom OperatorSource/CatalogSourceConfig, which are deprecated from OCP 4.5. Upgrade to OCP 4.5 will not be possible unless those resources are removed. Please visit https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated for further details.",
          "summary": "Indicates the presence of a custom OperatorSource or a CatalogSourceConifg in the cluster"
        },
        "state": "firing",
        "activeAt": "2020-06-25T21:56:13.459916592Z",
        "value": "2e+00"
      }
    ]
  }
}


Double checked the marketplace logs too to see if the metrics were being emitted: 

$ oc logs marketplace-operator-f7f869c8b-gbjjb
.
.
.
INFO[0194] Reconciling CatalogSourceConfig openshift-marketplace/csc-invaild 
WARN[0194] DEPRECATION NOTICE: The CatalogSourceConfig API is deprecated in future versions. Please visit this link for futher details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated 
.
.
.

Could you use a z stream release that contains this PR to test this please? 

Thanks!
Comment 11 Sunil Choudhary 2020-06-26 10:21:09 UTC 
I checked on 4.4.10 build and see the deprecation notice in the alerts as per comment #10. Marking it Verified.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.10    True        False         3h57m   Cluster version is 4.4.10

$ oc get csc
NAME          STATUS        MESSAGE                                                                                  AGE
csc-invaild   Configuring   Unable to resolve the source - no source contains the requested package(s) [etcd-test]   4m32s


$ oc get csc -o yaml
apiVersion: v1
items:
- apiVersion: operators.coreos.com/v2
  kind: CatalogSourceConfig
  metadata:
    creationTimestamp: "2020-06-26T10:13:30Z"
    finalizers:
    - finalizer.catalogsourceconfigs.operators.coreos.com
    generation: 3
    name: csc-invaild
    namespace: openshift-marketplace
    resourceVersion: "92589"
    selfLink: /apis/operators.coreos.com/v2/namespaces/openshift-marketplace/catalogsourceconfigs/csc-invaild
    uid: 3ad87f91-074b-43a2-b81c-3ad227e5153e
  spec:
    csDisplayName: CSC Operators
    csPublisher: CSC
    packages: etcd-test
    source: ""
    targetNamespace: openshift-operators
  status:
    currentPhase:
      lastTransitionTime: "2020-06-26T10:13:30Z"
      lastUpdateTime: "2020-06-26T10:13:30Z"
      phase:
        message: Unable to resolve the source - no source contains the requested package(s)
          [etcd-test]
        name: Configuring
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/alerts' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2639    0  2639    0     0  13337      0 --:--:-- --:--:-- --:--:-- 13395
{
  "status": "success",
  "data": {
    "alerts": [
      {
        "labels": {
          "alertname": "AlertmanagerReceiversNotConfigured",
          "severity": "warning"
        },
        "annotations": {
          "message": "Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager."
        },
        "state": "firing",
        "activeAt": "2020-06-26T06:08:06.662770393Z",
        "value": "0e+00"
      },
      {
        "labels": {
          "alertname": "CustomResourceDetected",
          "customResourceType": "OperatorSource",
          "endpoint": "https-metrics",
          "instance": "10.129.0.18:8081",
          "job": "marketplace-operator-metrics",
          "namespace": "openshift-marketplace",
          "pod": "marketplace-operator-79454c5bcb-wzfg2",
          "service": "marketplace-operator-metrics",
          "severity": "warning"
        },
        "annotations": {
          "message": "The cluster has custom OperatorSource/CatalogSourceConfig, which are deprecated from OCP 4.5. Upgrade to OCP 4.5 will not be possible unless those resources are removed. Please visit https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated for further details.",
          "summary": "Indicates the presence of a custom OperatorSource or a CatalogSourceConifg in the cluster"
        },
        "state": "firing",
        "activeAt": "2020-06-26T06:15:43.459916592Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "ImagePruningDisabled",
          "endpoint": "60000",
          "instance": "10.130.0.23:60000",
          "job": "image-registry-operator",
          "namespace": "openshift-image-registry",
          "pod": "cluster-image-registry-operator-77899994d8-crx4f",
          "service": "image-registry-operator",
          "severity": "warning"
        },
        "annotations": {
          "message": "Automatic image pruning is not enabled. Regular pruning of images\nno longer referenced by ImageStreams is strongly recommended to\nensure your cluster remains healthy.\n\nTo remove this warning, install the image pruner by creating an\nimagepruner.imageregistry.operator.openshift.io resource with the\nname `cluster`. Ensure that the `suspend` field is set to `false`.\n"
        },
        "state": "firing",
        "activeAt": "2020-06-26T06:08:08.020409993Z",
        "value": "1e+00"
      },
      {
        "labels": {
          "alertname": "Watchdog",
          "severity": "none"
        },
        "annotations": {
          "message": "This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n"
        },
        "state": "firing",
        "activeAt": "2020-06-26T06:08:00.163677339Z",
        "value": "1e+00"
      }
    ]
  }
}


$ oc logs marketplace-operator-79454c5bcb-wzfg2
...
time="2020-06-26T10:16:20Z" level=info msg="Reconciling CatalogSourceConfig openshift-marketplace/csc-invaild\n"
time="2020-06-26T10:16:20Z" level=warning msg="DEPRECATION NOTICE: The CatalogSourceConfig API is deprecated in future versions. Please visit this link for futher details: https://docs.openshift.com/container-platform/4.4/release_notes/ocp-4-4-release-notes.html#ocp-4-4-marketplace-apis-deprecated"
Comment 13 Jian Zhang 2020-06-29 08:25:32 UTC 
Thanks @Sunil

Hi, Anik

> Could you use a z stream release that contains this PR to test this please? 

As you can see from comment 9:

>> Cluster version is 4.4.0-0.nightly-2020-06-21-210301

>> Marketplace-operator version:
>> [root@preserve-olm-env data]#  oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-06-21-210301 |grep marketplace
  operator-marketplace                           https://github.com/operator-framework/operator-marketplace                  18ec31ba6b228dc63b637db840b6667580888d79

I used the right cluster version that contains your fixed PR.

Actually, in comment 9, already list the alert info:
  "message": "The cluster has custom OperatorSource/CatalogSourceConfig, which are deprecated from OCP 4.5. Upgrade to OCP 4.5 ...

Sorry for that, my mistake, I should have verified it.
Comment 14 errata-xmlrpc 2020-06-29 15:33:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2713