Bug 1779043
| Summary: | The jaeger operator couldn't be deployed due to the CatalogSource pods fail to respond | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Anping Li <anli> | ||||
| Component: | OLM | Assignee: | Bowen Song <bsong> | ||||
| OLM sub component: | OLM | QA Contact: | Jian Zhang <jiazha> | ||||
| Status: | CLOSED WONTFIX | Docs Contact: | |||||
| Severity: | medium | ||||||
| Priority: | high | CC: | bsong, dageoffr, ecordell, eparis | ||||
| Version: | 4.1.z | ||||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.1.z | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1781261 (view as bug list) | Environment: | |||||
| Last Closed: | 2020-02-18 13:46:51 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1781261 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
Created attachment 1641560 [details]
csc pod manifest and catalog operators logs
There are similar error in the csc pod cluster-logging-operator-75597bc855-6x2qb/installed-custom-openshift-ansible-service-broker-59664cdft6dtr, community-operators-556477f67f-6wtbb and redhat-operators-78d75587b5-fsjhm . the openshift-ansible-service-broker operator couldn't be deployed too. [anli@preserve-docker-slave ocp41_73053]$ oc logs cluster-logging-operator-75597bc855-6x2qb time="2019-12-03T06:04:55Z" level=info msg="Using in-cluster kube client config" port=50051 type=appregistry time="2019-12-03T06:04:55Z" level=info msg="operator source(s) specified are - [https://quay.io/cnr|redhat-operators]" port=50051 type=appregistry time="2019-12-03T06:04:55Z" level=info msg="package(s) specified are - cluster-logging" port=50051 type=appregistry time="2019-12-03T06:04:55Z" level=info msg="input has been sanitized" port=50051 type=appregistry time="2019-12-03T06:04:55Z" level=info msg="sources: [https://quay.io/cnr/redhat-operators]" port=50051 type=appregistry time="2019-12-03T06:04:55Z" level=info msg="packages: [cluster-logging]" port=50051 type=appregistry time="2019-12-03T06:04:55Z" level=info msg="resolved the following packages: [redhat-operators/cluster-logging:18.0.0]" port=50051 type=appregistry time="2019-12-03T06:04:55Z" level=info msg="downloading repository: redhat-operators/cluster-logging:18.0.0 from https://quay.io/cnr" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="download complete - 1 repositories have been downloaded" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="decoding the downloaded operator manifest(s)" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="manifest format is - nested" port=50051 repository="redhat-operators/cluster-logging:18.0.0" type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt - type=directory" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt/4.1 - type=directory" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt/4.1/cluster-logging.v4.1.0.clusterserviceversion.yaml - type=file" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt/4.1/cluster-loggings.crd.yaml - type=file" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt/4.2 - type=directory" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt/4.2/cluster-logging.v4.2.0.clusterserviceversion.yaml - type=file" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt/4.2/cluster-loggings.crd.yaml - type=file" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="downloaded/cluster-logging-w7o263kt/cluster-logging.package.yaml - type=file" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="decoded successfully" port=50051 repository="redhat-operators/cluster-logging:18.0.0" type=appregistry time="2019-12-03T06:04:56Z" level=info msg="decoded 0 flattened and 1 nested operator manifest(s)" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="loading nested operator bundle(s) from downloaded into sqlite" port=50051 type=appregistry time="2019-12-03T06:04:56Z" level=info msg="Loading bundles" dir=downloaded time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=downloaded load=bundles time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=cluster-logging-w7o263kt load=bundles time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=4.1 load=bundles time="2019-12-03T06:04:56Z" level=info msg="found csv, loading bundle" dir=downloaded file=cluster-logging.v4.1.0.clusterserviceversion.yaml load=bundles time="2019-12-03T06:04:56Z" level=info msg="loading bundle file" dir=downloaded file=cluster-logging.v4.1.0.clusterserviceversion.yaml load=bundle time="2019-12-03T06:04:56Z" level=info msg="loading bundle file" dir=downloaded file=cluster-loggings.crd.yaml load=bundle time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=4.2 load=bundles time="2019-12-03T06:04:56Z" level=info msg="found csv, loading bundle" dir=downloaded file=cluster-logging.v4.2.0.clusterserviceversion.yaml load=bundles time="2019-12-03T06:04:56Z" level=info msg="loading bundle file" dir=downloaded file=cluster-logging.v4.2.0.clusterserviceversion.yaml load=bundle time="2019-12-03T06:04:56Z" level=info msg="loading bundle file" dir=downloaded file=cluster-loggings.crd.yaml load=bundle time="2019-12-03T06:04:56Z" level=info msg="could not decode contents of file downloaded/cluster-logging-w7o263kt/cluster-logging.package.yaml into csv: error unmarshaling JSON: Object 'Kind' is missing in '{\"channels\":[{\"currentCSV\":\"clusterlogging.4.1.25-201911190028\",\"name\":\"preview\"},{\"currentCSV\":\"clusterlogging.4.2.8-201911190952\",\"name\":\"4.2\"}],\"defaultChannel\":\"4.2\",\"packageName\":\"cluster-logging\"}'" dir=downloaded file=cluster-logging.package.yaml load=bundles time="2019-12-03T06:04:56Z" level=info msg="Loading packages and entries" dir=downloaded time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=downloaded load=package time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=cluster-logging-w7o263kt load=package time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=4.1 load=package time="2019-12-03T06:04:56Z" level=info msg=directory dir=downloaded file=4.2 load=package time="2019-12-03T06:04:56Z" level=info msg="Extracting provided API information" dir=downloaded time="2019-12-03T06:04:56Z" level=info msg="serving registry" port=50051 type=appregistry The cluster network works well, but that pods(installed-redhat-openshift-operators-55f75f7dc8-7vctk) doesn't work well.
mac:~ jianzhang$ oc get pods -n openshift-marketplace -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
certified-operators-5c755d7664-wz79w 1/1 Running 0 3h9m 10.129.2.6 ip-10-0-143-220.us-east-2.compute.internal <none> <none>
cluster-logging-operator-5cc95f8dd9-m6qcn 1/1 Running 0 6m25s 10.128.2.27 ip-10-0-175-24.us-east-2.compute.internal <none> <none>
community-operators-556477f67f-6wtbb 1/1 Running 0 3h9m 10.129.2.5 ip-10-0-143-220.us-east-2.compute.internal <none> <none>
elasticsearch-775cdd5dcb-ms4nq 1/1 Running 0 6m25s 10.128.2.28 ip-10-0-175-24.us-east-2.compute.internal <none> <none>
installed-community-openshift-operators-64bd5849c6-8chkd 1/1 Running 0 72s 10.131.0.31 ip-10-0-149-237.us-east-2.compute.internal <none> <none>
installed-redhat-openshift-operators-55f75f7dc8-7vctk 1/1 Running 0 6m24s 10.129.2.23 ip-10-0-143-220.us-east-2.compute.internal <none> <none>
marketplace-operator-5b8cf489b6-zhlf4 1/1 Running 0 3h9m 10.129.0.26 ip-10-0-162-42.us-east-2.compute.internal <none> <none>
redhat-operators-6cffbfd58d-6mtl2 1/1 Running 0 6m24s 10.131.0.28 ip-10-0-149-237.us-east-2.compute.internal <none> <none>
As we can see, in the same node, the pod certified-operators-5c755d7664-wz79w (10.129.2.6) can respond well, but that installed-redhat-openshift-operators-55f75f7dc8-7vctk (10.129.0.23) failed to response.
mac:~ jianzhang$ oc rsh -n openshift-sdn sdn-zqvfk
sh-4.2# ping 10.129.0.23
PING 10.129.0.23 (10.129.0.23) 56(84) bytes of data.
From 10.130.0.1 icmp_seq=1 Destination Host Unreachable
From 10.130.0.1 icmp_seq=2 Destination Host Unreachable
From 10.130.0.1 icmp_seq=3 Destination Host Unreachable
From 10.130.0.1 icmp_seq=4 Destination Host Unreachable
From 10.130.0.1 icmp_seq=5 Destination Host Unreachable
From 10.130.0.1 icmp_seq=6 Destination Host Unreachable
From 10.130.0.1 icmp_seq=7 Destination Host Unreachable
From 10.130.0.1 icmp_seq=8 Destination Host Unreachable
From 10.130.0.1 icmp_seq=9 Destination Host Unreachable
^c
sh-4.2# ping 10.129.2.6
PING 10.129.2.6 (10.129.2.6) 56(84) bytes of data.
64 bytes from 10.129.2.6: icmp_seq=1 ttl=64 time=1.48 ms
64 bytes from 10.129.2.6: icmp_seq=2 ttl=64 time=0.748 ms
64 bytes from 10.129.2.6: icmp_seq=3 ttl=64 time=0.753 ms
exi64 bytes from 10.129.2.6: icmp_seq=4 ttl=64 time=0.744 ms
...
But, we didn't get any errors and warnings from that pod logs. Only get the info message:
time="2019-12-03T05:40:51Z" level=info msg="could not decode contents of file downloaded/jaeger-product-aiiua3zr/jaeger.package.yaml into csv: error unmarshaling JSON: Object 'Kind' is missing in '{\"channels\":[{\"currentCSV\":\"jaeger-operator.v1.13.1\",\"name\":\"stable\"}],\"packageName\":\"jaeger-product\"}'" dir=downloaded file=jaeger.package.yaml load=bundles
time="2019-12-03T05:40:51Z" level=info msg="Loading packages and entries" dir=downloaded
So, here, I think we should fix two problems:
1, the above info: "Object 'Kind' is missing". I think there is no `Kind` object in the package. It's very confusing!
2, If the pod fails to respond, please print the error message.
One more question, the database is empty, is it as expected? mac:~ jianzhang$ oc rsh installed-redhat-openshift-operators-565cdc5448-cvb9n ... sh-4.2$ ls bundles.db downloaded sh-4.2$ sh-4.2$ sh-4.2$ sqlite3 bundles.db SQLite version 3.7.17 2013-05-20 00:56:22 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> .table sqlite> By the ways, the community jaeger can be deployed in this cluster. |
Description of problem: The stable jaeger operator couldn't be deployed in 4.1. the catalogsource raise the following error time="2019-12-03T05:40:51Z" level=info msg="loading bundle file" dir=downloaded file=jaeger-v1.crd.yaml load=bundle time="2019-12-03T05:40:51Z" level=info msg="loading bundle file" dir=downloaded file=jaeger.v1.13.1.clusterserviceversion.yaml load=bundle time="2019-12-03T05:40:51Z" level=info msg="could not decode contents of file downloaded/jaeger-product-aiiua3zr/jaeger.package.yaml into csv: error unmarshaling JSON: Object 'Kind' is missing in '{\"channels\":[{\"currentCSV\":\"jaeger-operator.v1.13.1\",\"name\":\"stable\"}],\"packageName\":\"jaeger-product\"}'" dir=downloaded file=jaeger.package.yaml load=bundles time="2019-12-03T05:40:51Z" level=info msg="Loading packages and entries" dir=downloaded Version-Release number of selected component (if applicable): OLM version: io.openshift.build.commit.url=https://github.com/operator-framework/operator-lifecycle-manager/commit/1bd1fe142d7ae74fd53829132b66e0455716e035 Cluster version: 4.1.0-0.nightly-2019-11-28-220857 Jadger: "currentCSV":"jaeger-operator.v1.13.1" How reproducible: This time Steps to Reproduce: 1. Deploy elasticsearch operators 2. Deploy cluster-logging operators 3. Deploy stable jaeger Actual results: 1. The catalogsource are created and running [ocp41_73053]$ oc get pods NAME READY STATUS RESTARTS AGE certified-operators-5c755d7664-wz79w 1/1 Running 0 138m cluster-logging-operator-75597bc855-6x2qb 1/1 Running 0 17m community-operators-556477f67f-6wtbb 1/1 Running 0 138m elasticsearch-74f7959575-4dcrz 1/1 Running 0 17m installed-custom-openshift-ansible-service-broker-59664cdft6dtr 1/1 Running 0 17m installed-custom-openshift-template-service-broker-59df8c8pttsp 1/1 Running 0 17m installed-redhat-openshift-operators-565cdc5448-cvb9n 1/1 Running 0 41m marketplace-operator-5b8cf489b6-zhlf4 1/1 Running 0 138m 2. The catalogsoruce pod report :could not decode contents of file downloaded/jaeger-product-aiiua3zr/jaeger.package.yaml into csv: error unmarshaling JSON: Object 'Kind' is missing in '{\"channels\":[{\"currentCSV\":\"jaeger-operator.v1.13.1\",\"name\":\"stable\"}],\"packageName\":\"jaeger-product\"}'" dir=downloaded file=jaeger.package.yaml load=bundles 3. oc describe sub jaeger-product -n openshift-operators Name: jaeger-product Namespace: openshift-operators Labels: csc-owner-name=installed-redhat-openshift-operators csc-owner-namespace=openshift-marketplace Annotations: <none> API Version: operators.coreos.com/v1alpha1 Kind: Subscription Metadata: Creation Timestamp: 2019-12-03T05:24:50Z Generation: 1 Resource Version: 41994 Self Link: /apis/operators.coreos.com/v1alpha1/namespaces/openshift-operators/subscriptions/jaeger-product UID: 3ab3fff8-158d-11ea-89b4-0a7160cfddb4 Spec: Channel: stable Install Plan Approval: Automatic Name: jaeger-product Source: installed-redhat-openshift-operators Source Namespace: openshift-operators Starting CSV: jaeger-operator.v1.13.1 Events: <none> 3. oc describe csc installed-redhat-openshift-operators Name: installed-redhat-openshift-operators Namespace: openshift-marketplace Labels: <none> Annotations: <none> API Version: operators.coreos.com/v1 Kind: CatalogSourceConfig Metadata: Creation Timestamp: 2019-12-03T05:24:50Z Finalizers: finalizer.catalogsourceconfigs.operators.coreos.com Generation: 3 Resource Version: 41983 Self Link: /apis/operators.coreos.com/v1/namespaces/openshift-marketplace/catalogsourceconfigs/installed-redhat-openshift-operators UID: 3a815016-158d-11ea-89b4-0a7160cfddb4 Spec: Cs Display Name: Red Hat Operators Cs Publisher: Red Hat Packages: jaeger-product Target Namespace: openshift-operators Status: Current Phase: Last Transition Time: 2019-12-03T05:24:50Z Last Update Time: 2019-12-03T05:24:50Z Phase: Message: The object has been successfully reconciled Name: Succeeded Package Repositiory Versions: Jaeger - Product: 3.0.0 Events: <none> Expected results: The jaeger operator can be deployed in 4.1 Additional info: