Bug 1829028
| Summary: | 4.2.30 install on GCP fails consistently with machine-api-operator in CrashLoopBackoff | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mike Fiedler <mifiedle> |
| Component: | Release | Assignee: | Luke Meyer <lmeyer> |
| Status: | CLOSED ERRATA | QA Contact: | Milind Yadav <miyadav> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.2.z | CC: | aos-bugs, chezhang, jokerman, jupierce, kgarriso, mmasters, vlaad, wking |
| Target Milestone: | --- | Keywords: | Regression, Reopened, TestBlocker |
| Target Release: | 4.2.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-05-13 11:07:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1827779 | ||
|
Description
Mike Fiedler
2020-04-28 18:03:18 UTC
Must gather is missing logs for machine-controller pod, even though only 1/3 pods is in crashloop backoff. We need to follow up and figure out why the must-gather tool is missing the logs for all the containers. Ideally, we should get the most recent logs from the failed container as well. I used the cluster bot to try to launch a cluster with that release. Logs from machine-controller container panic: semver: Parse(doozer-failure-5ed92c18-055634): No Major.Minor.Patch elements found goroutine 1 [running]: github.com/blang/semver.MustParse(0x19a9890, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/openshift/cluster-api-provider-gcp/vendor/github.com/blang/semver/semver.go:356 +0x1c1 github.com/openshift/cluster-api-provider-gcp/pkg/version.init.ializers() /go/src/github.com/openshift/cluster-api-provider-gcp/pkg/version/version.go:16 +0 Here's some more detailed logs if for some reason they're needed: https://gcsweb-ci.svc.ci.openshift.org/gcs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/615/artifacts/launch/pods/ > panic: semver: Parse(doozer-failure-5ed92c18-055634): No Major.Minor.Patch elements found ART's tooling drops these tags if a build fails for whatever reason. Recommended fix it to have build tooling expect the OS_GIT_VERSION tag [1]. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1827706#c8 Ah, and this bug is a dup. *** This bug has been marked as a duplicate of bug 1827706 *** The image must be rebuilt with a change to ART code: https://bugzilla.redhat.com/show_bug.cgi?id=1827706#c11 Still broken in 4.2.30. Need to re-spin 4.2.30 with a fix for this Since this bug actually has supporting details I'm inverting the duplicate bug relationship. *** Bug 1827706 has been marked as a duplicate of this bug. *** Moving over to release to execute on what Justin prescribes in comment 9. Steps : 1.Ran flexi IPI installation on GCP [registry.svc.ci.openshift.org/ocp/release:4.2.30] 2.Installation failed Errors: oc logs -f machine-api-controllers-9fcf7ff59-q6w5d -c machine-controller panic: semver: Parse(doozer-failure-5ed92c18-055634): No Major.Minor.Patch elements found goroutine 1 [running]: github.com/blang/semver.MustParse(0x19a9890, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/openshift/cluster-api-provider-gcp/vendor/github.com/blang/semver/semver.go:356 +0x1c1 github.com/openshift/cluster-api-provider-gcp/pkg/version.init.ializers() /go/src/github.com/openshift/cluster-api-provider-gcp/pkg/version/version.go:16 +0x78 [miyadav@miyadav bugzhsun1828704]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 79m Unable to apply 4.2.30: some cluster operators have not yet rolled out which is same as what was reported , moving it to ASSIGNED state > 1.Ran flexi IPI installation on GCP [registry.svc.ci.openshift.org/ocp/release:4.2.30] 4.2.30 is the impacted release [1]. Use a more recent nightly to verify. [1]: https://github.com/openshift/cincinnati-graph-data/pull/203#issuecomment-621218274 Thanks Trevor , for providing comment , I used the latest 4.2 nightly - 'registry.svc.ci.openshift.org/ocp/release:4.2.0-0.nightly-2020-05-04-072150' and the installation was successful Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2023 |