Bug 2020486
Summary: | still check the deleted custom catsrc of openshift-marketplace found regression auto case | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | kuiwang |
Component: | OLM | Assignee: | Ankita Thomas <ankithom> |
OLM sub component: | OLM | QA Contact: | kuiwang |
Status: | CLOSED DEFERRED | Docs Contact: | |
Severity: | high | ||
Priority: | medium | CC: | ankithom, davegord, jiazha, krizza, pegoncal, xzha |
Version: | 4.10 | Keywords: | Reopened |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2023-03-09 01:08:46 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
kuiwang
2021-11-05 03:15:08 UTC
Hi, Another error. I think it is related to this. it is also found in automation regression, and impacts the automation. --- message: '[error using catalog community-operators (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.54.152:50051: connect: connection refused", error using catalog redhat-operators (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.140.255:50051: connect: connection refused", error using catalog redhat-marketplace (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.92.239:50051: connect: connection refused", error using catalog certified-operators (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.5.74:50051: connect: connection refused"]' reason: ErrorPreventedResolution status: "True" type: ResolutionFailed lastUpdated: "2021-11-07T22:12:44Z" --- Actually the catsrcs exist. here is the full of sub output. -- operators.coreos.com/v1alpha1 kind: Subscription metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"operators.coreos.com/v1alpha1","kind":"Subscription","metadata":{"annotations":{},"name":"mta-operator","namespace":"e2e-test-olm-a-s2vh596e-2f99t"},"spec":{"channel":"alpha","installPlanApproval":"Automatic","name":"mta-operator","source":"catsrc-27683-operator","sourceNamespace":"e2e-test-olm-a-s2vh596e-2f99t","startingCSV":"windup-operator.0.0.5"}} creationTimestamp: "2021-11-07T22:09:36Z" generation: 1 labels: operators.coreos.com/mta-operator.e2e-test-olm-a-s2vh596e-2f99t: "" name: mta-operator namespace: e2e-test-olm-a-s2vh596e-2f99t resourceVersion: "377040" uid: 07b02a4b-00a3-4271-9e30-99b2fbcb328c spec: channel: alpha installPlanApproval: Automatic name: mta-operator source: catsrc-27683-operator sourceNamespace: e2e-test-olm-a-s2vh596e-2f99t startingCSV: windup-operator.0.0.5 status: catalogHealth: catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: catsrc-27683-operator namespace: e2e-test-olm-a-s2vh596e-2f99t resourceVersion: "374440" uid: c6514f03-9f99-4274-a392-b473530927fc healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: bucket-test1 namespace: openshift-marketplace resourceVersion: "372867" uid: d8aa3193-25d3-45d6-9248-83129d92c879 healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: bucket-test2 namespace: openshift-marketplace resourceVersion: "374206" uid: e42cb0c3-2cd3-40d0-a688-9d0ba1e8d51c healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: certified-operators namespace: openshift-marketplace resourceVersion: "374036" uid: 2fa9c33e-fb9d-4d9e-9495-e96cae43a11d healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: community-operators namespace: openshift-marketplace resourceVersion: "373379" uid: 4e44075e-76e0-45a0-bb84-038d95bb3f2f healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: ocs-cs namespace: openshift-marketplace resourceVersion: "372740" uid: 21caf2ce-bbca-4829-8d88-2bae1dcf2580 healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: qe-app-registry namespace: openshift-marketplace resourceVersion: "374438" uid: f25480e8-b5d8-4741-9a4b-a92b29a97236 healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-marketplace namespace: openshift-marketplace resourceVersion: "374361" uid: 6f398eae-3c06-4462-b45b-5b4547d6805c healthy: true lastUpdated: "2021-11-07T22:10:01Z" catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-operators namespace: openshift-marketplace resourceVersion: "373900" uid: 970e4b15-7f19-4859-9433-f2f00b20db6e healthy: true lastUpdated: "2021-11-07T22:10:01Z" conditions: lastTransitionTime: "2021-11-07T22:10:01Z" message: all available catalogsources are healthy reason: AllCatalogSourcesHealthy status: "False" type: CatalogSourcesUnhealthy message: '[error using catalog community-operators (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.54.152:50051: connect: connection refused", error using catalog redhat-operators (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.140.255:50051: connect: connection refused", error using catalog redhat-marketplace (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.92.239:50051: connect: connection refused", error using catalog certified-operators (in namespace openshift-marketplace): failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.30.5.74:50051: connect: connection refused"]' reason: ErrorPreventedResolution status: "True" type: ResolutionFailed lastUpdated: "2021-11-07T22:12:44Z" -- Given that we have been unable to build a reproducer for this, I am closing this as CLOSED INSUFFICIENT_DATA. If we can come up with a consistent reproducer, feel free to reopen. Hi, We met this issue on release-4.12 again. prometheus-dependency-cs/openshift-marketplace is deleted, but other sub still check this catsrc. https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-e2e-aws-ipi-proxy-p1/1551628438144552960/artifacts/e2e-aws-ipi-proxy-p1/openshift-extended-test/build-log.txt apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"operators.coreos.com/v1alpha1","kind":"Subscription","metadata":{"annotations":{},"name":"sub-41283","namespace":"e2e-test-default-yslzgqku-t8rpb"},"spec":{"channel":"beta","installPlanApproval":"Automatic","name":"learn","source":"qe-app-registry","sourceNamespace":"openshift-marketplace","startingCSV":""}} creationTimestamp: "2022-07-25T20:25:24Z" generation: 1 labels: operators.coreos.com/learn.e2e-test-default-yslzgqku-t8rpb: "" name: sub-41283 namespace: e2e-test-default-yslzgqku-t8rpb resourceVersion: "379554" uid: ee8728d7-0798-42da-a8ee-947854ca96e4 spec: channel: beta installPlanApproval: Automatic name: learn source: qe-app-registry sourceNamespace: openshift-marketplace startingCSV: "" status: catalogHealth: - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: catsrc-34181 namespace: openshift-marketplace resourceVersion: "378403" uid: 1a8e0068-0c3c-4f9d-8968-b5d600235d48 healthy: true lastUpdated: "2022-07-25T20:25:26Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: certified-operators namespace: openshift-marketplace resourceVersion: "377933" uid: 6a197906-b20f-4931-a514-6cfe237aa6b9 healthy: true lastUpdated: "2022-07-25T20:25:26Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: community-operators namespace: openshift-marketplace resourceVersion: "372177" uid: 789db2b2-1800-4682-ac83-4eb89e9b0395 healthy: true lastUpdated: "2022-07-25T20:25:26Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: qe-app-registry namespace: openshift-marketplace resourceVersion: "351331" uid: 931e65f2-0395-48d0-937c-40ffe96a20fb healthy: true lastUpdated: "2022-07-25T20:25:26Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-marketplace namespace: openshift-marketplace resourceVersion: "356231" uid: b36ef648-bc51-4a9e-9c7f-9137a10f63e1 healthy: true lastUpdated: "2022-07-25T20:25:26Z" - catalogSourceRef: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource name: redhat-operators namespace: openshift-marketplace resourceVersion: "356248" uid: c7c2057f-8d2b-4ed7-8527-7131821f365f healthy: true lastUpdated: "2022-07-25T20:25:26Z" conditions: - lastTransitionTime: "2022-07-25T20:25:26Z" message: all available catalogsources are healthy reason: AllCatalogSourcesHealthy status: "False" type: CatalogSourcesUnhealthy - message: 'failed to populate resolver cache from source prometheus-dependency-cs/openshift-marketplace: failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup prometheus-dependency-cs.openshift-marketplace.svc on 172.30.0.10:53: no such host"' reason: ErrorPreventedResolution status: "True" type: ResolutionFailed lastUpdated: "2022-07-25T20:25:28Z" oc get pod -n openshift-marketplace NAME READY STATUS RESTARTS AGE 396e821f5f897df76f48604a35b951fdf21122004a507827f8d6a5d911z2td2 0/1 Completed 0 60m 4747b56e58799d351945d54657efb0b83a8ca1d00dd7e5380d06d3fa7996wpr 0/1 Completed 0 18m 7287d35a7951dc1c78d66c23cc5830ea6fb2201da8d86b9aae9bee312eb9qlv 0/1 Completed 0 48m 8dadcd4e49a1312f71c2886fc5b7043a4d1fcca18f7deb8fccda62682654gf9 0/1 Completed 0 49m 93f04e698990976394e4a33295a856e5fcea70a1f016cfed3579c24c68fjbzb 0/1 Completed 0 93m 9d3a30bb8b8a8ef71f041a58604ec052ca3b888b279f6ac23de68ca59b8mtr5 0/1 Completed 0 92m a8d0bad6d2efb38a05505c22a61a3610f15ad932bdc7f307071ada39b3z2htn 0/1 Completed 0 94m aaab2b8f65c5454c01522388166d6b659cae1fe2afb06b46c8f2f8880697vnx 0/1 Completed 0 91m b8d8da31d4360c8f21e581d3f9edc4d19e6ec0a87bc240962e4936cbb4qvvk8 0/1 Completed 0 35m c0eafc0942e40dd3fb4cdf25023ca93e2d8b82b4d5cc3b67da4bb1ef3ak77n6 0/1 Completed 0 62m certified-operators-rxmkq 1/1 Running 0 4m25s community-operators-c6q72 1/1 Running 0 112m de8b38fd9ac831c08ce2f4341c50ac2491f605adf23a387e40cc3713cfdcw9j 0/1 Completed 0 6m25s df4206ef9f6068058722977a3e2821d93212a70782e6a82a8a9e79690369ggx 0/1 Completed 0 18m e8c9651078ae45ddb2807e3a07727d459b82d7def5572a7b7ccaae332btqzqz 0/1 Completed 0 73m f1b164aee9615543598c8cb3fd3010b7c98bc29937b4132780ced8d8ferrxjv 0/1 Completed 0 54m fc2985520e4b70156d718aaadf0ad87196b40b20788981bf3bf6cbe09a8rj65 0/1 Completed 0 8m51s marketplace-operator-698bf45f99-wq6jb 1/1 Running 0 129m qe-app-registry-kdnnd 1/1 Running 0 104m redhat-marketplace-5f4hq 1/1 Running 0 127m redhat-operators-f2rhq 1/1 Running 0 127m Get this issue again, automation test detials: https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-e2e-aws-ipi-proxy-cco-manual-security-token-service-p1/1566788424629751808/artifacts/e2e-aws-ipi-proxy-cco-manual-security-token-service-p1/openshift-extended-test/build-log.txt Sep 5 16:40:20.965: INFO: $oc get [sub sub-25922 -n e2e-test-default-plnns5e6-fzpxv -o=jsonpath-as-json={.status}], the returned resource:[ { "catalogHealth": [ { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "certified-operators", "namespace": "openshift-marketplace", "resourceVersion": "191028", "uid": "8873578e-7a94-4b6d-b384-df07c9c388f7" }, "healthy": true, "lastUpdated": "2022-09-05T16:37:14Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "community-operators", "namespace": "openshift-marketplace", "resourceVersion": "191020", "uid": "b37fd169-0300-405a-87e4-2c60efe019e4" }, "healthy": true, "lastUpdated": "2022-09-05T16:37:14Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "qe-app-registry", "namespace": "openshift-marketplace", "resourceVersion": "191021", "uid": "26dda0b5-1733-4595-938a-a72e443f4cf4" }, "healthy": true, "lastUpdated": "2022-09-05T16:37:14Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "redhat-marketplace", "namespace": "openshift-marketplace", "resourceVersion": "191024", "uid": "e300473e-b378-4cdb-aff3-65e094dac6d6" }, "healthy": true, "lastUpdated": "2022-09-05T16:37:14Z" }, { "catalogSourceRef": { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "CatalogSource", "name": "redhat-operators", "namespace": "openshift-marketplace", "resourceVersion": "191023", "uid": "c21c363e-6356-4759-a8c4-d1fe6c6bbcec" }, "healthy": true, "lastUpdated": "2022-09-05T16:37:14Z" } ], "conditions": [ { "lastTransitionTime": "2022-09-05T16:37:14Z", "message": "all available catalogsources are healthy", "reason": "AllCatalogSourcesHealthy", "status": "False", "type": "CatalogSourcesUnhealthy" }, { "message": "failed to populate resolver cache from source catsrc-34181/openshift-marketplace: failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup catsrc-34181.openshift-marketplace.svc on 172.30.0.10:53: no such host\"", "reason": "ErrorPreventedResolution", "status": "True", "type": "ResolutionFailed" } ], "lastUpdated": "2022-09-05T16:37:14Z" } ] OpenShift has moved to Jira for its defect tracking! This bug can now be found in the OCPBUGS project in Jira. https://issues.redhat.com/browse/OCPBUGS-9020 |