Bug 1787653
| Summary: | 4.2.8 -> 4.2.13 update has marketplace operator failing after: dial tcp 172.30.0.1:443: connect: no route to host | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> |
| Component: | Networking | Assignee: | Ben Bennett <bbennett> |
| Networking sub component: | openshift-sdn | QA Contact: | zhaozhanqi <zzhao> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | dsover, scuppett |
| Version: | 4.2.z | ||
| Target Milestone: | --- | ||
| Target Release: | 4.4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-04 16:59:57 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
W. Trevor King
2020-01-03 21:16:34 UTC
In this space are: * Bug 1743699, about CA bundle injection. But that was a 100% failure and a subsequent 4.2.8 -> 4.2.13 GCP CI job passed [1]. * Bug 1744153, about 4.1 marketplace operators vs. v2 CatalogSourceConfig. But this bug is a 4.2->4.2 issue. * Bug 1749643, about marketplace operators vs. non-hyphenated CatalogSourceConfig. But the hyphen guard landed on 2019-09-12 [2], and 4.2.8 was cut on 2019-11-21, so it should include the fix. Checking: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.8 | grep marketplace operator-marketplace https://github.com/operator-framework/operator-marketplace 906bffd1dea6464efd9d73d42f03d411ef6cb158 $ git --no-pager log --oneline -18 origin/release-4.2 | sed 's/^/ /' 2cc8e419 Merge pull request #272 from openshift-cherrypick-robot/cherry-pick-271-to-release-4.2 ... 906bffd1 Merge pull request #263 from Bowenislandsong/upgrade-status-bug-4.2 ... 77108b16 Merge pull request #249 from gallettilance/migration-bug-fix ... 63fdd2cb [migration] - don't crash on non-hyphenated csc so we have the guard from that bug in 4.2.8. [1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/190 [2]: https://github.com/operator-framework/operator-marketplace/pull/249#event-2629670871 [1] is pretty coarse, but has the marketplace-operator deployment breaking updates every few days. It's possible that some of those hits have a different underlying cause. [1]: https://search.svc.ci.openshift.org/?search=Could+not+update+deployment.*openshift-marketplace%2Fmarketplace-operator&maxAge=336h&context=-1&type=all Setting target to active development branch (4.4). Fixes (if any) will result in cloned BZs for backports (where required/requested) for prior versions. Huh. Why does [1] have logs but [2] does not? $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/187/artifacts/e2e-gcp-upgrade/must-gather/must-gather.log | grep 7749fd48dd [must-gather-hc9h5] POD 2020/01/03 19:04:06 Gathering data for pod "marketplace-operator-7749fd48dd-4kfgp" [must-gather-hc9h5] POD 2020/01/03 19:04:06 Skipping container data collection for pod "marketplace-operator-7749fd48dd-4kfgp": Pod not running [must-gather-hc9h5] OUT namespaces/openshift-marketplace/pods/marketplace-operator-7749fd48dd-4kfgp/ [must-gather-hc9h5] OUT namespaces/openshift-marketplace/pods/marketplace-operator-7749fd48dd-4kfgp/marketplace-operator-7749fd48dd-4kfgp.yaml I'll spin that off into a must-gather bug. [1]: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/187/artifacts/e2e-gcp-upgrade/pods/openshift-marketplace_marketplace-operator-7749fd48dd-4kfgp_marketplace-operator.log [2]: https://gcsweb-ci.svc.ci.openshift.org/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/187/artifacts/e2e-gcp-upgrade/must-gather/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-335d7e63146bbd3bb903fb2fd1df042f450661e45be36cef89996badc29bbaf9/namespaces/openshift-marketplace/pods/marketplace-operator-7749fd48dd-4kfgp/ So from [1]: time="2020-01-03T19:01:03Z" level=warning msg="Config API is not available" time="2020-01-03T19:01:06Z" level=fatal msg="Get https://172.30.0.1:443/api?timeout=32s: dial tcp 172.30.0.1:443: connect: no route to host" Sounds like bug 1715515, but that was closed as a dup of bug 1714699, which was verified fixed [2] in 4.2.0: $ curl -sH 'Accept:application/json' 'https://api.openshift.com/api/upgrades_info/v1/graph?channel=stable-4.2' | jq -r '.nodes[] | select(.metadata.url == "https://access.redhat.com/errata/RHBA-2019:2922").version' 4.2.0 But moving to the networking folks so they try and diagnose. [1]: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/187/artifacts/e2e-gcp-upgrade/pods/openshift-marketplace_marketplace-operator-7749fd48dd-4kfgp_marketplace-operator.log [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1714699#c19 (In reply to W. Trevor King from comment #6) > I'll spin that off into a must-gather bug. Filed as bug 1798753. I'm not seeing this happen recently. Please reopen if I am wrong. |