Description of problem: The openshift-marketplace community operator pod keeps crashing with Liveness and Readiness probe failures. The failures are: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 26m (x941 over 10h) kubelet, 10.195.94.74 Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s Warning BackOff 6m5s (x1631 over 10h) kubelet, 10.195.94.74 Back-off restarting failed container Warning Unhealthy 71s (x1403 over 10h) kubelet, 10.195.94.74 Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s Version-Release number of selected component (if applicable): OCP 4.6, but we have also recreated the problem in OpenShift 4.3, 4.4 and 4.5. How reproducible: Not always Steps to Reproduce: 1. Create an OpenShift 4.6 cluster 2. Observe the community-operator pods in openshift-marketplace, after a few minutes they will fail. Actual results: The community operator pod failed to run. Expected results: We need the openshift-marketplace community operator pods to function in a running state. Additional info:
Can you provide the logs of these pods that eventually crash? If the liveness and readiness probes are failing it most likely means that the pod is unable to start up.
Here are the logs: describe pod -n openshift-marketplace community-operators-7897f56dbd-z7ns6 Name: community-operators-7897f56dbd-z7ns6 Namespace: openshift-marketplace Priority: 0 Node: 10.195.94.74/10.195.94.74 Start Time: Mon, 05 Apr 2021 01:25:53 +0000 Labels: marketplace.operatorSource=community-operators pod-template-hash=7897f56dbd Annotations: cni.projectcalico.org/podIP: 172.30.134.12/32 cni.projectcalico.org/podIPs: 172.30.134.12/32 k8s.v1.cni.cncf.io/networks-status: [{ "name": "k8s-pod-network", "ips": [ "172.30.134.12" ], "dns": {} }] openshift.io/scc: restricted Status: Running IP: 172.30.134.12 IPs: IP: 172.30.134.12 Controlled By: ReplicaSet/community-operators-7897f56dbd Containers: community-operators: Container ID: cri-o://c04f12c76403f776a657f904fe6a0679a4e2f8ddd266c31b76a0e5f6fdf6942d Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2d8d4dc5e56a1ffdac9a63fa4517ff6cac55ca082be3eabc05c18a3d5251ca6b Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2d8d4dc5e56a1ffdac9a63fa4517ff6cac55ca082be3eabc05c18a3d5251ca6b Port: 50051/TCP Host Port: 0/TCP Command: appregistry-server -r https://quay.io/cnr|community-operators -o hive-operator,nexus-operator-m88i,mta-operator,atlasmap-operator,victoriametrics-operator,spark-gcp,cockroachdb,opsmx-spinnaker-operator,infinispan,dell-csi-operator,planetscale,keepalived-operator,eunomia,ibmcloud-iam-operator,cluster-manager,proactive-node-scaling-operator,resource-locker-operator,openebs,kubestone,argocd-operator,vault-helm,etcd,datadog-operator,knative-kafka-operator,3scale-community-operator,keda,event-streams-topic,syndesis,percona-xtradb-cluster-operator,enc-key-sync,global-load-balancer-operator,tidb-operator,istio-workspace-operator,camel-k,eclipse-che,ibmcloud-operator,enmasse,grafana-operator,ibm-spectrum-scale-csi-operator,neuvector-community-operator,reportportal-operator,jaeger,akka-cluster-operator,esindex-operator,maistraoperator,service-binding-operator,codeready-toolchain-operator,cost-mgmt-operator,prisma-cloud-compute-console-operator,metering,argocd-operator-helm,lib-bucket-provisioner,sosreport-operator,traefikee-operator,hazelcast-operator,openshift-ibm-quantum-operator,multicluster-operators-subscription,teiid,splunk,podium-operator-bundle,hawkbit-operator,kubefed,integrity-shield-operator,mongodb-atlas-kubernetes,gitlab-runner-operator,volume-expander-operator,openshift-qiskit-operator,apicurito,keycloak-operator,aws-efs-operator,konveyor-virt-operator,node-problem-detector,microcks,t8c,hawtio-operator,egressip-ipam-operator,horreum-operator,group-sync-operator,community-windows-machine-config-operator,prometheus,knative-camel-operator,namespace-configuration-operator,redis-operator,kubernetes-imagepuller-operator,hazelcast-jet-operator,opendatahub-operator,cert-utils-operator,ham-deploy,federation,api-operator,sysflow-operator,snapscheduler,portworx-essentials,oadp-operator,hedvig-operator,kogito-operator,hyperfoil-bundle,pystol,klusterlet,konveyor-forklift-operator,microsegmentation-operator,ember-csi-operator,apicurio-registry,ditto-operator,jupyterlab-operator,iot-simulator,nsm-operator-registry,must-gather-operator,ibm-block-csi-operator-community,myvirtualdirectory,jenkins-operator,special-resource-operator,konveyor-operator,buildv2-operator,openshift-nfd-operator,strimzi-kafka-operator,radanalytics-spark,skydive-operator,snyk-operator,starter-kit-operator,prometheus-exporter-operator,yaks,opentelemetry-operator,spinnaker-operator,community-kubevirt-hyperconverged,federatorai,seldon-operator,kiali,kubeturbo,postgresql,wso2am-operator,sealed-secrets-operator-helm,percona-server-mongodb-operator,mcad-operator,koku-metrics-operator,assisted-service-operator,crossplane,awss3-operator-registry,postgresql-operator-dev4devs-com,data-explorer-operator,aqua,lightbend-console-operator,composable-operator,apicast-community-operator,azure-service-operator,ibm-quantum-operator,ripsaw State: Running Started: Mon, 05 Apr 2021 12:12:13 +0000 Last State: Terminated Reason: Error Exit Code: 2 Started: Mon, 05 Apr 2021 12:10:03 +0000 Finished: Mon, 05 Apr 2021 12:12:13 +0000 Ready: False Restart Count: 141 Requests: cpu: 10m memory: 100Mi Liveness: exec [grpc_health_probe -addr=localhost:50051] delay=30s timeout=1s period=10s #success=1 #failure=10 Readiness: exec [grpc_health_probe -addr=localhost:50051] delay=60s timeout=1s period=10s #success=1 #failure=10 Environment: HTTP_PROXY: HTTPS_PROXY: NO_PROXY: Mounts: /etc/pki/ca-trust/extracted/pem/ from marketplace-trusted-ca (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-gzwln (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: marketplace-trusted-ca: Type: ConfigMap (a volume populated by a ConfigMap) Name: marketplace-trusted-ca Optional: false default-token-gzwln: Type: Secret (a volume populated by a Secret) SecretName: default-token-gzwln Optional: false QoS Class: Burstable Node-Selectors: beta.kubernetes.io/os=linux Tolerations: node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 26m (x941 over 10h) kubelet, 10.195.94.74 Readiness probe failed: timeout: failed to connect service "localhost:50051" within 1s Warning BackOff 6m5s (x1631 over 10h) kubelet, 10.195.94.74 Back-off restarting failed container Warning Unhealthy 71s (x1403 over 10h) kubelet, 10.195.94.74 Liveness probe failed: timeout: failed to connect service "localhost:50051" within 1s(In reply to Kevin Rizza from comment #1) > Can you provide the logs of these pods that eventually crash? If the > liveness and readiness probes are failing it most likely means that the pod > is unable to start up.
Couple of followup questions: First off, it looks like this pod is running with app-registry and the old marketplace-operator apis. We have deprecated those in favor of public catalog images built upfront from registry.redhat.io. Is there a reason that this 4.6 cluster is still attempting to use the old APIs? That is definitely not supported if it is the configuration that is being used on a 4.6 cluster. Secondly, that is the description of the pod, which is helpful, but I'm actually looking for the pod's logs itself before it crashes. If it's retrying, you should be able to pull the logs while it is still in progress. This is presumably because the size of the community catalog is too much for the memory constraints defined by that api, and I want to confirm that there isn't another underlying issue here. I also want to level set that this particular configuration isn't something that is supported on a 4.6 cluster, so if there is something that needs to be increased for a 4.5 and earlier cluster we can investigate but a 4.6 cluster should just use the new pubic index images which shouldn't run into these same limitations. As a result, I'm lowering the severity and priority of this bug down to high.
I have a couple more details to add. Logs from when the probe failed: Found kubeconfig in kubeconfigs secret time="2021-04-05T12:12:13Z" level=info msg="Using in-cluster kube client config" time="2021-04-05T12:12:13Z" level=info msg="operator source(s) specified are - [https://quay.io/cnr|community-operators]" port=50051 type=appregistry time="2021-04-05T12:12:13Z" level=info msg="package(s) specified are - hive-operator,nexus-operator-m88i,mta-operator,atlasmap-operator,victoriametrics-operator,spark-gcp,cockroachdb,opsmx-spinnaker-operator,infinispan,dell-csi-operator,planetscale,keepalived-operator,eunomia,ibmcloud-iam-operator,cluster-manager,proactive-node-scaling-operator,resource-locker-operator,openebs,kubestone,argocd-operator,vault-helm,etcd,datadog-operator,knative-kafka-operator,3scale-community-operator,keda,event-streams-topic,syndesis,percona-xtradb-cluster-operator,enc-key-sync,global-load-balancer-operator,tidb-operator,istio-workspace-operator,camel-k,eclipse-che,ibmcloud-operator,enmasse,grafana-operator,ibm-spectrum-scale-csi-operator,neuvector-community-operator,reportportal-operator,jaeger,akka-cluster-operator,esindex-operator,maistraoperator,service-binding-operator,codeready-toolchain-operator,cost-mgmt-operator,prisma-cloud-compute-console-operator,metering,argocd-operator-helm,lib-bucket-provisioner,sosreport-operator,traefikee-operator,hazelcast-operator,openshift-ibm-quantum-operator,multicluster-operators-subscription,teiid,splunk,podium-operator-bundle,hawkbit-operator,kubefed,integrity-shield-operator,mongodb-atlas-kubernetes,gitlab-runner-operator,volume-expander-operator,openshift-qiskit-operator,apicurito,keycloak-operator,aws-efs-operator,konveyor-virt-operator,node-problem-detector,microcks,t8c,hawtio-operator,egressip-ipam-operator,horreum-operator,group-sync-operator,community-windows-machine-config-operator,prometheus,knative-camel-operator,namespace-configuration-operator,redis-operator,kubernetes-imagepuller-operator,hazelcast-jet-operator,opendatahub-operator,cert-utils-operator,ham-deploy,federation,api-operator,sysflow-operator,snapscheduler,portworx-essentials,oadp-operator,hedvig-operator,kogito-operator,hyperfoil-bundle,pystol,klusterlet,konveyor-forklift-operator,microsegmentation-operator,ember-csi-operator,apicurio-registry,ditto-operator,jupyterlab-operator,iot-simulator,nsm-operator-registry,must-gather-operator,ibm-block-csi-operator-community,myvirtualdirectory,jenkins-operator,special-resource-operator,konveyor-operator,buildv2-operator,openshift-nfd-operator,strimzi-kafka-operator,radanalytics-spark,skydive-operator,snyk-operator,starter-kit-operator,prometheus-exporter-operator,yaks,opentelemetry-operator,spinnaker-operator,community-kubevirt-hyperconverged,federatorai,seldon-operator,kiali,kubeturbo,postgresql,wso2am-operator,sealed-secrets-operator-helm,percona-server-mongodb-operator,mcad-operator,koku-metrics-operator,assisted-service-operator,crossplane,awss3-operator-registry,postgresql-operator-dev4devs-com,data-explorer-operator,aqua,lightbend-console-operator,composable-operator,apicast-community-operator,azure-service-operator,ibm-quantum-operator,ripsaw" port=50051 type=appregistry time="2021-04-05T12:12:13Z" level=info msg="input has been sanitized" port=50051 type=appregistry time="2021-04-05T12:12:13Z" level=info msg="sources: [https://quay.io/cnr/community-operators]" port=50051 type=appregistry time="2021-04-05T12:12:13Z" level=info msg="packages: [hive-operator nexus-operator-m88i mta-operator atlasmap-operator victoriametrics-operator spark-gcp cockroachdb opsmx-spinnaker-operator infinispan dell-csi-operator planetscale keepalived-operator eunomia ibmcloud-iam-operator cluster-manager proactive-node-scaling-operator resource-locker-operator openebs kubestone argocd-operator vault-helm etcd datadog-operator knative-kafka-operator 3scale-community-operator keda event-streams-topic syndesis percona-xtradb-cluster-operator enc-key-sync global-load-balancer-operator tidb-operator istio-workspace-operator camel-k eclipse-che ibmcloud-operator enmasse grafana-operator ibm-spectrum-scale-csi-operator neuvector-community-operator reportportal-operator jaeger akka-cluster-operator esindex-operator maistraoperator service-binding-operator codeready-toolchain-operator cost-mgmt-operator prisma-cloud-compute-console-operator metering argocd-operator-helm lib-bucket-provisioner sosreport-operator traefikee-operator hazelcast-operator openshift-ibm-quantum-operator multicluster-operators-subscription teiid splunk podium-operator-bundle hawkbit-operator kubefed integrity-shield-operator mongodb-atlas-kubernetes gitlab-runner-operator volume-expander-operator openshift-qiskit-operator apicurito keycloak-operator aws-efs-operator konveyor-virt-operator node-problem-detector microcks t8c hawtio-operator egressip-ipam-operator horreum-operator group-sync-operator community-windows-machine-config-operator prometheus knative-camel-operator namespace-configuration-operator redis-operator kubernetes-imagepuller-operator hazelcast-jet-operator opendatahub-operator cert-utils-operator ham-deploy federation api-operator sysflow-operator snapscheduler portworx-essentials oadp-operator hedvig-operator kogito-operator hyperfoil-bundle pystol klusterlet konveyor-forklift-operator microsegmentation-operator ember-csi-operator apicurio-registry ditto-operator jupyterlab-operator iot-simulator nsm-operator-registry must-gather-operator ibm-block-csi-operator-community myvirtualdirectory jenkins-operator special-resource-operator konveyor-operator buildv2-operator openshift-nfd-operator strimzi-kafka-operator radanalytics-spark skydive-operator snyk-operator starter-kit-operator prometheus-exporter-operator yaks opentelemetry-operator spinnaker-operator community-kubevirt-hyperconverged federatorai seldon-operator kiali kubeturbo postgresql wso2am-operator sealed-secrets-operator-helm percona-server-mongodb-operator mcad-operator koku-metrics-operator assisted-service-operator crossplane awss3-operator-registry postgresql-operator-dev4devs-com data-explorer-operator aqua lightbend-console-operator composable-operator apicast-community-operator azure-service-operator ibm-quantum-operator ripsaw]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package openshift-ibm-quantum-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package kubernetes-imagepuller-operator -> [0.0.5 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package iot-simulator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package infinispan -> [2.1.0 2.0.9 2.0.8 2.0.7 2.0.6 2.0.5 2.0.4 2.0.3 2.0.2 2.0.1 2.0.0 1.1.2 1.1.1 1.1.0 1.0.2 1.0.1 1.0.0 0.3.3 0.3.2 0.3.1 0.3.0 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package oadp-operator -> [0.1.7 0.1.6 0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package spinnaker-operator -> [1.17.8 1.17.7 1.17.6 1.17.5 1.17.4 1.17.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package keda -> [1.5.3 1.5.2 1.5.1 1.5.0 1.4.1 1.3.0 1.2.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package hawkbit-operator -> [0.1.4 0.1.3 0.1.2 0.1.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package mcad-operator -> [0.1.9]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ibmcloud-iam-operator -> [0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package neuvector-community-operator -> [1.0.6 1.0.5 1.0.4 1.0.3 1.0.2 1.0.1 1.0.0 0.9.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package cert-utils-operator -> [0.2.6 0.2.5 0.2.4 0.2.3 0.2.2 0.2.1 0.2.0 0.1.1 0.1.0 0.0.3 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ibm-quantum-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package 3scale-community-operator -> [0.5.2 0.5.1 0.4.0 0.3.1 0.3.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package integrity-shield-operator -> [0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package sysflow-operator -> [0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package konveyor-forklift-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package postgresql-operator-dev4devs-com -> [0.1.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package nexus-operator-m88i -> [0.2.5 0.2.4 0.2.3 0.2.2 0.2.1 0.2.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package global-load-balancer-operator -> [0.1.6 0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package gitlab-runner-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package klusterlet -> [0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package radanalytics-spark -> [1.0.11 1.0.10 1.0.9 1.0.8 1.0.7 1.0.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package nsm-operator-registry -> [0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package planetscale -> [0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package prisma-cloud-compute-console-operator -> [1.0.1 1.0.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package podium-operator-bundle -> [1.0.6 1.0.5 1.0.4 1.0.3 1.0.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package hazelcast-jet-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package kogito-operator -> [0.15.0 0.14.9 0.14.8 0.14.7 0.14.6 0.14.5 0.14.4 0.14.3 0.14.2 0.14.1 0.14.0 0.13.0 0.12.1 0.12.0 0.11.0 0.10.0 0.9.1 0.9.0 0.8.1 0.8.0 0.7.0 0.7.0-rc4 0.6.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package microsegmentation-operator -> [0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ember-csi-operator -> [0.9.5 0.9.4 0.9.3 0.9.2 0.9.1 0.9.0 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package wso2am-operator -> [1.0.2 1.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package sealed-secrets-operator-helm -> [0.0.3 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package spark-gcp -> [2.4.0 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package kubestone -> [0.4.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package syndesis -> [1.9.1 1.9.0 1.7.0 0.1.7]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package percona-xtradb-cluster-operator -> [1.4.1 1.4.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package eclipse-che -> [7.20.6 7.20.5 7.20.4 7.20.3 7.20.2 7.20.1 7.20.0 7.19.9 7.19.8 7.19.7 7.19.6 7.19.5 7.19.4 7.19.3 7.19.2 7.19.1 7.19.0 7.18.9 7.18.8 7.18.7 7.18.6 7.18.5 7.18.4 7.18.3 7.18.2 7.17.2 7.17.1 7.17.0 7.16.2 7.15.3 7.15.2 7.15.1 7.15.0 7.14.2 7.14.1 7.13.2 7.13.1 7.12.1 7.12.0 7.11.0 7.10.0 7.9.0 7.8.0 7.7.3 7.7.2 7.7.1 7.7.0 7.6.0 7.5.2 7.5.1 7.4.0 7.3.1 7.3.0 7.2.0 7.1.0 7.0.5 7.0.4 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ibmcloud-operator -> [0.1.14 0.1.13 0.1.12 0.1.11 0.1.10 0.1.9 0.1.8 0.1.7 0.1.6 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package azure-service-operator -> [0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package cost-mgmt-operator -> [0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package t8c -> [7.21.1 7.21.0 7.17.0 7.16.0 0.7.17]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package konveyor-operator -> [1.2.6 1.2.5 1.2.4 1.2.3 1.2.2 1.2.1 1.2.0 1.1.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package strimzi-kafka-operator -> [0.19.3 0.19.2 0.19.1 0.19.0 0.18.9 0.18.8 0.18.7 0.18.6 0.18.5 0.18.4 0.18.3 0.18.2 0.18.1 0.18.0 0.17.0 0.16.2 0.15.0 0.14.4 0.14.3 0.14.2 0.14.1 0.14.0 0.13.0 0.12.2 0.12.1 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package crossplane -> [0.11.0 0.9.0 0.8.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package multicluster-operators-subscription -> [0.2.8 0.2.7 0.2.6 0.2.5 0.2.4 0.2.3 0.2.2 0.2.1 0.1.11 0.1.10 0.1.9 0.1.8 0.1.7 0.1.6 0.1.5 0.1.4 0.1.3 0.1.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package prometheus-exporter-operator -> [0.2.1 0.2.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package proactive-node-scaling-operator -> [0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package argocd-operator-helm -> [0.0.7 0.0.6 0.0.4 0.0.3]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package kubefed -> [0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package seldon-operator -> [1.2.5 1.2.4 1.2.3 1.2.2 1.2.1 1.2.0 1.1.1 1.1.0 0.1.6 0.1.5 0.1.4 0.1.3 0.1.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package apicast-community-operator -> [0.2.4 0.2.3 0.2.2 0.2.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package etcd -> [0.0.13 0.0.12 0.0.11 0.0.10 0.0.9 0.0.8 0.0.6 0.0.5 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package event-streams-topic -> [0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package konveyor-virt-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package jenkins-operator -> [0.4.8 0.4.7 0.4.6 0.4.5 0.4.4 0.4.3 0.4.2 0.4.1 0.4.0 0.3.31 0.3.3 0.3.0 0.2.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package opentelemetry-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package community-kubevirt-hyperconverged -> [0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package cluster-manager -> [0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package tidb-operator -> [1.1.6 1.1.5 1.1.4 1.1.3 1.1.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package openshift-nfd-operator -> [4.5.4 4.5.3 4.5.2 4.5.1 4.5.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package pystol -> [0.8.17]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package apicurio-registry -> [0.0.4 0.0.3 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ripsaw -> [0.1.2 0.1.1 0.1.0 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package jupyterlab-operator -> [0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package hive-operator -> [1.2.4 1.2.3 1.2.2 1.2.1 1.2.0 1.1.9 1.1.8 1.1.7 1.1.6 1.1.5 1.1.4 1.1.3 1.1.2 1.1.1 1.1.0 1.0.9 1.0.8 1.0.7 1.0.6 1.0.5 1.0.4 1.0.3 1.0.2 1.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package atlasmap-operator -> [0.3.0 0.2.0 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package openebs -> [1.5.2 1.5.1 1.5.0 1.4.0 1.3.0 1.1.1 1.1.0 1.0.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package service-binding-operator -> [0.1.1-364 0.1.1-354 0.1.1-352 0.1.1-282 0.1.0-267 0.1.0-254 0.1.0-249 0.0.26 0.0.25 0.0.24 0.0.24-214 0.0.23 0.0.20]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package traefikee-operator -> [2.1.1 2.0.2 0.4.1 0.3.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package hawtio-operator -> [0.2.2 0.2.1 0.2.0 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package snapscheduler -> [1.1.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package hazelcast-operator -> [0.3.5 0.3.4 0.3.3 0.3.2 0.3.1 0.3.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package openshift-qiskit-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package special-resource-operator -> [0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package lightbend-console-operator -> [0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package vault-helm -> [0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package splunk -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package volume-expander-operator -> [0.1.6 0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package aws-efs-operator -> [0.0.6 0.0.5 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package buildv2-operator -> [0.1.1 0.1.0 0.0.9 0.0.8 0.0.7 0.0.6]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package composable-operator -> [0.1.3]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package opsmx-spinnaker-operator -> [1.20.6 1.20.5 1.20.4 1.20.3 1.17.6 1.17.5 1.17.4 1.17.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package knative-kafka-operator -> [0.16.2 0.16.1 0.16.0 0.15.1 0.15.0 0.14.1 0.13.3 0.13.2 0.12.2 0.12.1 0.11.2 0.10.0 0.9.0 0.8.2 0.7.1 0.6.1 0.6.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package akka-cluster-operator -> [1.0.1 1.0.0 0.2.3 0.2.0 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package must-gather-operator -> [0.1.6 0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ibm-block-csi-operator-community -> [1.2.4 1.2.3 1.2.2 1.2.1 1.2.0 1.0.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package kubeturbo -> [7.22.1 7.22.0 7.21.0 6.4.0 6.3.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package keycloak-operator -> [11.0.2 11.0.1 11.0.0 10.0.0 9.0.4 9.0.3 9.0.2 9.0.1 9.0.0 8.0.2 8.0.1 7.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package myvirtualdirectory -> [0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package federatorai -> [4.2.756 4.2.755 4.2.557 4.2.556 4.2.553 4.2.552 4.2.551 4.2.301 4.1.20 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package dell-csi-operator -> [1.0.3 1.0.2 1.0.1 1.0.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package argocd-operator -> [0.0.14 0.0.13 0.0.12 0.0.11 0.0.9 0.0.8 0.0.6 0.0.5 0.0.4]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package enmasse -> [0.32.2 0.31.4 0.31.3 0.31.2 0.31.1 0.31.0 0.30.2 0.29.1 0.28.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ibm-spectrum-scale-csi-operator -> [2.0.1 2.0.0 1.1.0 1.0.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package apicurito -> [0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package reportportal-operator -> [0.0.6 0.0.5]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package esindex-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package namespace-configuration-operator -> [0.3.1 0.3.0 0.2.9 0.2.8 0.2.7 0.2.6 0.2.5 0.2.4 0.2.3 0.2.2 0.2.1 0.2.0 0.1.0 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package opendatahub-operator -> [0.7.6 0.7.5 0.7.4 0.7.3 0.7.2 0.7.1 0.7.0 0.6.2 0.6.1 0.6.0 0.5.2 0.5.1 0.5.0 0.4.2 0.4.1 0.4.0 0.2.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package data-explorer-operator -> [0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package istio-workspace-operator -> [0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package camel-k -> [1.1.6 1.1.5 1.1.4 1.1.3 1.1.2 1.1.1 1.1.0 1.0.1 1.0.0 1.0.0-rc2 1.0.0-rc1 1.0.0-m4 1.0.0-m3 1.0.0-m2 1.0.0-m1 0.3.5 0.3.4 0.3.0 0.2.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package postgresql -> [4.4.8 4.4.7 4.4.6 4.4.5 4.4.4 4.4.3 4.4.2 4.4.1 4.4.0 4.3.2 4.2.3 4.2.2 4.2.1 4.2.0 4.1.0 4.0.1 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package percona-server-mongodb-operator -> [1.4.5 1.4.4 1.4.3 1.4.2 1.4.1 1.4.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package assisted-service-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package victoriametrics-operator -> [0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package jaeger -> [1.18.4 1.18.3 1.18.2 1.18.1 1.18.0 1.17.2 1.17.1 1.17.0 1.16.0 1.15.0 1.14.1 1.14.0 1.13.1 1.13.0 1.12.1 1.12.0 0.0.7 0.0.6 0.0.5 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package group-sync-operator -> [0.1.1 0.1.0 0.0.9 0.0.8 0.0.7 0.0.6 0.0.5 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package prometheus -> [0.37.1 0.37.0 0.32.0 0.22.4 0.22.3 0.22.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package kiali -> [1.22.7 1.22.6 1.22.5 1.22.4 1.22.3 1.22.2 1.22.1 1.22.0 1.19.0 1.15.1 1.13.0 1.11.0 1.9.1 1.7.0 1.4.3 1.4.2 1.3.1 1.1.1 1.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package eunomia -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package grafana-operator -> [3.5.5 3.5.4 3.5.3 3.5.2 3.5.1 3.5.0 2.0.0 1.3.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package metering -> [4.1.1 4.1.0 0.14.1 0.14.0 0.13.5 0.13.4 0.13.3 0.13.2 0.13.1 0.13.0 0.12.0 0.4.3 0.4.2 0.4.1 0.0.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package lib-bucket-provisioner -> [2.0.0 1.0.2 1.0.1 1.0.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package teiid -> [0.3.1 0.3.0 0.2.0 0.1.1 0.1.0 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package node-problem-detector -> [0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package awss3-operator-registry -> [1.0.6 1.0.5 1.0.4 1.0.3 1.0.2 1.0.1 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package microcks -> [1.0.2 1.0.1 1.0.0 0.3.0 0.2.1 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package horreum-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package federation -> [0.0.6 0.0.5 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package resource-locker-operator -> [0.2.4 0.2.3 0.2.2 0.2.1 0.2.0 0.1.9 0.1.8 0.1.7 0.1.6 0.1.5 0.1.4 0.1.3 0.1.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package datadog-operator -> [0.3.3 0.3.2 0.3.1 0.3.0 0.2.3 0.2.2 0.2.0 0.1.3]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package egressip-ipam-operator -> [0.2.2 0.2.1 0.2.0 0.1.9 0.1.8 0.1.7 0.1.6 0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ham-deploy -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package portworx-essentials -> [1.3.9 1.3.8 1.3.7 1.3.6 1.3.5 1.3.4 1.3.3]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package koku-metrics-operator -> [0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package sosreport-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package skydive-operator -> [0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package yaks -> [0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package mta-operator -> [0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package maistraoperator -> [1.1.6 1.1.5 1.1.4 1.1.3 1.1.2 1.1.1 1.1.0 1.0.8 1.0.6 1.0.2 1.0.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package mongodb-atlas-kubernetes -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package community-windows-machine-config-operator -> [0.1.4 0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package cockroachdb -> [3.0.16 3.0.15 3.0.14 3.0.13 3.0.12 3.0.11 3.0.10 3.0.9 3.0.8 3.0.7 2.1.12 2.1.11 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package starter-kit-operator -> [0.2.1 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package enc-key-sync -> [0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package codeready-toolchain-operator -> [0.1.3 0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package api-operator -> [1.1.3 1.1.2 1.1.1 1.1.0 1.0.2 1.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package hedvig-operator -> [0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package snyk-operator -> [1.32.7 1.32.6 1.32.5 1.32.4 1.32.3 1.30.6 1.30.0 1.25.0 1.21.4]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package aqua -> [1.1.1 1.1.0 1.0.9 1.0.8 1.0.7 1.0.6 1.0.5 1.0.4 1.0.3 1.0.2 1.0.1 1.0.0 0.0.4 0.0.3 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package keepalived-operator -> [0.2.8 0.2.7 0.2.6 0.2.5 0.2.4 0.2.3 0.2.2 0.2.0 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package knative-camel-operator -> [0.15.3 0.15.2 0.15.1 0.15.0 0.13.2 0.12.1 0.10.0 0.9.0 0.8.2 0.7.1 0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package redis-operator -> [0.0.2 0.0.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package hyperfoil-bundle -> [0.5.3 0.5.2 0.5.1]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="the following releases are available for package ditto-operator -> [0.1.2 0.1.1 0.1.0]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="resolved the following packages: [openshift-ibm-quantum-operator:0.1.0 kubernetes-imagepuller-operator:0.0.5 iot-simulator:0.1.0 infinispan:2.1.0 oadp-operator:0.1.7 spinnaker-operator:1.17.8 keda:1.5.3 hawkbit-operator:0.1.4 mcad-operator:0.1.9 ibmcloud-iam-operator:0.1.3 neuvector-community-operator:1.0.6 cert-utils-operator:0.2.6 ibm-quantum-operator:0.1.0 3scale-community-operator:0.5.2 integrity-shield-operator:0.1.4 sysflow-operator:0.0.1 konveyor-forklift-operator:0.1.0 postgresql-operator-dev4devs-com:0.1.1 nexus-operator-m88i:0.2.5 global-load-balancer-operator:0.1.6 gitlab-runner-operator:0.1.0 klusterlet:0.1.2 radanalytics-spark:1.0.11 nsm-operator-registry:0.0.1 planetscale:0.0.3 prisma-cloud-compute-console-operator:1.0.1 podium-operator-bundle:1.0.6 hazelcast-jet-operator:0.1.0 kogito-operator:0.15.0 microsegmentation-operator:0.0.1 ember-csi-operator:0.9.5 wso2am-operator:1.0.2 sealed-secrets-operator-helm:0.0.3 spark-gcp:2.4.0 kubestone:0.4.0 syndesis:1.9.1 percona-xtradb-cluster-operator:1.4.1 eclipse-che:7.20.6 ibmcloud-operator:0.1.14 azure-service-operator:0.1.1 cost-mgmt-operator:0.0.4 t8c:7.21.1 konveyor-operator:1.2.6 strimzi-kafka-operator:0.19.3 crossplane:0.11.0 multicluster-operators-subscription:0.2.8 prometheus-exporter-operator:0.2.1 proactive-node-scaling-operator:0.1.5 argocd-operator-helm:0.0.7 kubefed:0.1.3 seldon-operator:1.2.5 apicast-community-operator:0.2.4 etcd:0.0.13 event-streams-topic:0.1.1 konveyor-virt-operator:0.1.0 jenkins-operator:0.4.8 opentelemetry-operator:0.1.0 community-kubevirt-hyperconverged:0.1.1 cluster-manager:0.1.1 tidb-operator:1.1.6 openshift-nfd-operator:4.5.4 pystol:0.8.17 apicurio-registry:0.0.4 ripsaw:0.1.2 jupyterlab-operator:0.1.2 hive-operator:1.2.4 atlasmap-operator:0.3.0 openebs:1.5.2 service-binding-operator:0.1.1-364 traefikee-operator:2.1.1 hawtio-operator:0.2.2 snapscheduler:1.1.1 hazelcast-operator:0.3.5 openshift-qiskit-operator:0.1.0 special-resource-operator:0.0.3 lightbend-console-operator:0.0.1 vault-helm:0.1.3 splunk:0.1.0 volume-expander-operator:0.1.6 aws-efs-operator:0.0.6 buildv2-operator:0.1.1 composable-operator:0.1.3 opsmx-spinnaker-operator:1.20.6 knative-kafka-operator:0.16.2 akka-cluster-operator:1.0.1 must-gather-operator:0.1.6 ibm-block-csi-operator-community:1.2.4 kubeturbo:7.22.1 keycloak-operator:11.0.2 myvirtualdirectory:0.0.1 federatorai:4.2.756 dell-csi-operator:1.0.3 argocd-operator:0.0.14 enmasse:0.32.2 ibm-spectrum-scale-csi-operator:2.0.1 apicurito:0.0.1 reportportal-operator:0.0.6 esindex-operator:0.1.0 namespace-configuration-operator:0.3.1 opendatahub-operator:0.7.6 data-explorer-operator:0.1.2 istio-workspace-operator:0.1.4 camel-k:1.1.6 postgresql:4.4.8 percona-server-mongodb-operator:1.4.5 assisted-service-operator:0.1.0 victoriametrics-operator:0.1.5 jaeger:1.18.4 group-sync-operator:0.1.1 prometheus:0.37.1 kiali:1.22.7 eunomia:0.1.0 grafana-operator:3.5.5 metering:4.1.1 lib-bucket-provisioner:2.0.0 teiid:0.3.1 node-problem-detector:0.0.2 awss3-operator-registry:1.0.6 microcks:1.0.2 horreum-operator:0.1.0 federation:0.0.6 resource-locker-operator:0.2.4 datadog-operator:0.3.3 egressip-ipam-operator:0.2.2 ham-deploy:0.1.0 portworx-essentials:1.3.9 koku-metrics-operator:0.1.5 sosreport-operator:0.1.0 skydive-operator:0.0.2 yaks:0.1.2 mta-operator:0.1.3 maistraoperator:1.1.6 mongodb-atlas-kubernetes:0.1.0 community-windows-machine-config-operator:0.1.4 cockroachdb:3.0.16 starter-kit-operator:0.2.1 enc-key-sync:0.0.2 codeready-toolchain-operator:0.1.3 api-operator:1.1.3 hedvig-operator:0.1.0 snyk-operator:1.32.7 aqua:1.1.1 keepalived-operator:0.2.8 knative-camel-operator:0.15.3 redis-operator:0.0.2 hyperfoil-bundle:0.5.3 ditto-operator:0.1.2]" port=50051 type=appregistry time="2021-04-05T12:12:16Z" level=info msg="downloading repository: openshift-ibm-quantum-operator:0.1.0 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:17Z" level=info msg="downloading repository: kubernetes-imagepuller-operator:0.0.5 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:17Z" level=info msg="downloading repository: iot-simulator:0.1.0 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:18Z" level=info msg="downloading repository: infinispan:2.1.0 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:18Z" level=info msg="downloading repository: oadp-operator:0.1.7 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:19Z" level=info msg="downloading repository: spinnaker-operator:1.17.8 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:19Z" level=info msg="downloading repository: keda:1.5.3 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:20Z" level=info msg="downloading repository: hawkbit-operator:0.1.4 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:21Z" level=info msg="downloading repository: mcad-operator:0.1.9 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:21Z" level=info msg="downloading repository: ibmcloud-iam-operator:0.1.3 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:22Z" level=info msg="downloading repository: neuvector-community-operator:1.0.6 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:22Z" level=info msg="downloading repository: cert-utils-operator:0.2.6 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:23Z" level=info msg="downloading repository: ibm-quantum-operator:0.1.0 from https://quay.io/cnr" port=50051 type=appregistry time="2021-04-05T12:12:23Z" level=info msg="downloading repository: 3scale-community-operator:0.5.2 from https://quay.io/cnr" port=50051 type=appregistry Updated liveness probe that fixed the failure: livenessProbe: exec: command: - grpc_health_probe - -addr=localhost:50051 failureThreshold: 10 initialDelaySeconds: 120 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10
We've seen this problem on OpenShift versions 4.3, 4.4 and 4.5. The logs and pod details that Smitha and I posted were for an OpenShift version 4.4 cluster. Sorry about the confusion here. While we've had some problems on the community operator pods on OpenShift version 4.6, we don't have enough data to confirm it is the same problem.
So it seems like the problem is that the probes are done too quickly for the older appregistry based community operator catalog sources? Then this would require going back and making changes to the marketplace operator on those older release branches. That being said, 4.4 is well passed EOL (and using a deprecated operator packaging format) so I'm not sure we can get an exception, let's confirm.
The issue is ultimately that the node pulling down the catalog is too slow to finish downloading all the content before the probes start to fail. This isn't a problem for 4.6 and later because we don't download anything, it's just packaged in the image itself. Because this older format is deprecated, and we cannot backport anything beyond urgent bugfixes to 4.5 and lower clusters, closing this as WONTFIX.