Bug 1724155
| Summary: | Upgrade failure with "Marking Degraded due to: during bootstrap: unexpected on-disk state validating" error. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Antonio Murdaca <amurdaca> |
| Component: | Machine Config Operator | Assignee: | Antonio Murdaca <amurdaca> |
| Status: | CLOSED ERRATA | QA Contact: | Micah Abbott <miabbott> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.2.0 | CC: | amurdaca, dmoessne, jmathian, miabbott, mnguyen, mzali |
| Target Milestone: | --- | ||
| Target Release: | 4.2.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1723327 | Environment: | |
| Last Closed: | 2019-10-16 06:32:41 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1723327, 1729345 | ||
| Bug Blocks: | |||
|
Comment 1
Antonio Murdaca
2019-06-26 11:12:03 UTC
To verify this BZ on AWS: - bring up a 4.2 cluster on AWS that already contains https://github.com/openshift/machine-config-operator/pull/887 (anything already since it's already in the latest payloads) - run the following for i in $(oc get pods -n openshift-machine-config-operator -l k8s-app=machine-config-daemon -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | xargs); do oc rsh -n openshift-machine-config-operator $i chroot /rootfs mkdir -p /run/pivot; oc rsh -n openshift-machine-config-operator $i chroot /rootfs touch /run/pivot/reboot-needed; done - disable the CVO oc scale --replicas=0 -n openshift-cluster-version deploy/cluster-version-operator - edit the osimageurl ConfigMap and put a newer oscontainer pullspec in there from https://releases-art-jenkins.cloud.paas.upshift.redhat.com/, releases/rhcos-4.2 from the dropdown (an old oscontainer should work as well by downgrading): oc edit cm/machine-config-osimageurl -nopenshift-machine-config-operartor - the pools should now go through an os update (equal to a real upgrade for the sake of verifying and testing this BZ) - everything reconciles NOTE: make sure you have creds in the pullsecret for the cluster to pull oscontainers from https://releases-art-jenkins.cloud.paas.upshift.redhat.com/ - releases/rhcos-4.2 Verified on 4.2.0-0.nightly-2019-06-27-041730 using the steps above to upgrade to quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0fc729885d9986f3846198a5b8dc3b2603a2fb273c4b5282c9d32ac0f84654b
$ for i in $(oc get pods -n openshift-machine-config-operator -l k8s-app=machine-config-daemon -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | xargs); do oc rsh -n openshift-machine-config-operator $i chroot /rootfs rpm-ostree status; done
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0fc729885d9986f3846198a5b8dc3b2603a2fb273c4b5282c9d32ac0f84654b
CustomOrigin: Managed by pivot tool
Version: 420.8.20190626.0 (2019-06-26T18:35:16Z)
pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d9c4793f2b5a279f0ee0066c474b0d2b0d6f2cb5730d377a7ae144b9f6e1a25
CustomOrigin: Managed by pivot tool
Version: 410.8.20190604.0 (2019-06-04T20:37:00Z)
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0fc729885d9986f3846198a5b8dc3b2603a2fb273c4b5282c9d32ac0f84654b
CustomOrigin: Managed by pivot tool
Version: 420.8.20190626.0 (2019-06-26T18:35:16Z)
pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d9c4793f2b5a279f0ee0066c474b0d2b0d6f2cb5730d377a7ae144b9f6e1a25
CustomOrigin: Managed by pivot tool
Version: 410.8.20190604.0 (2019-06-04T20:37:00Z)
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0fc729885d9986f3846198a5b8dc3b2603a2fb273c4b5282c9d32ac0f84654b
CustomOrigin: Managed by pivot tool
Version: 420.8.20190626.0 (2019-06-26T18:35:16Z)
pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd3bd07c9a4fd39d8039b137fa161c3a5b38a2855e8df9f6c16946ccdf7b3f31
CustomOrigin: Provisioned from oscontainer
Version: 420.8.20190624.0 (2019-06-24T00:25:32Z)
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0fc729885d9986f3846198a5b8dc3b2603a2fb273c4b5282c9d32ac0f84654b
CustomOrigin: Managed by pivot tool
Version: 420.8.20190626.0 (2019-06-26T18:35:16Z)
pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d9c4793f2b5a279f0ee0066c474b0d2b0d6f2cb5730d377a7ae144b9f6e1a25
CustomOrigin: Managed by pivot tool
Version: 410.8.20190604.0 (2019-06-04T20:37:00Z)
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0fc729885d9986f3846198a5b8dc3b2603a2fb273c4b5282c9d32ac0f84654b
CustomOrigin: Managed by pivot tool
Version: 420.8.20190626.0 (2019-06-26T18:35:16Z)
pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d9c4793f2b5a279f0ee0066c474b0d2b0d6f2cb5730d377a7ae144b9f6e1a25
CustomOrigin: Managed by pivot tool
Version: 410.8.20190604.0 (2019-06-04T20:37:00Z)
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0fc729885d9986f3846198a5b8dc3b2603a2fb273c4b5282c9d32ac0f84654b
CustomOrigin: Managed by pivot tool
Version: 420.8.20190626.0 (2019-06-26T18:35:16Z)
pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d9c4793f2b5a279f0ee0066c474b0d2b0d6f2cb5730d377a7ae144b9f6e1a25
CustomOrigin: Managed by pivot tool
Version: 410.8.20190604.0 (2019-06-04T20:37:00Z)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922 |