Recently the s390x 4.5 RHCOS pipeline started failing because the ostree version of fedora32 which is used in the cosa container had the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1867601 Because of this - the root file system is immutable - which is good, but the ostree version in RHCOS 4.5 is an older version which does not have the fix for the above BZ. This causes rollbacks and upgrades to fail. For now, the coreos-assembler is patched to remove the immutable bit, but it would be good if ostree is bumped with the fix for RHCOS 4.5.
This is the said fix that would need to be backported: https://github.com/ostreedev/ostree/pull/2179
Targeting 4.7 (though the fix should already be present there and in 4.6); we will probably need to clone this for 4.5.z
This was fixed in ostree-2020.5-4.el8 which is already included in RHCOS 4.7.
Verified on 4.7.0-0.nightly-2020-10-26-124513. ostree version is ostree-2020.5-4 $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-136-111.us-west-2.compute.internal Ready worker 5h12m v1.19.0+e67f5dc ip-10-0-138-139.us-west-2.compute.internal Ready master 5h17m v1.19.0+e67f5dc ip-10-0-166-15.us-west-2.compute.internal Ready worker 5h12m v1.19.0+e67f5dc ip-10-0-187-209.us-west-2.compute.internal Ready master 5h17m v1.19.0+e67f5dc ip-10-0-218-114.us-west-2.compute.internal Ready worker 5h7m v1.19.0+e67f5dc ip-10-0-219-131.us-west-2.compute.internal Ready master 5h18m v1.19.0+e67f5dc $ oc get pods No resources found. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-10-26-124513 True False 5h4m Cluster version is 4.7.0-0.nightly-2020-10-26-124513 $ oc debug node/ip-10-0-136-111.us-west-2.compute.internal -- chroot /host rpm -q ostree Starting pod/ip-10-0-136-111us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` ostree-2020.5-4.el8.x86_64 Removing debug pod ... $ oc debug node/ip-10-0-136-111.us-west-2.compute.internal -- chroot /host rpm-ostree status Starting pod/ip-10-0-136-111us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` State: idle Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dc9cdce1901a5f5ca6cbab5e435c48d924eb813a01f5c48e49dd91c1d2e340c CustomOrigin: Managed by machine-config-operator Version: 47.82.202010241242-0 (2020-10-24T12:46:24Z) ostree://6dff002defa66e3f9f53b57da7dc3eef37db39762655c00c13b977d72516649c Version: 47.82.202010211043-0 (2020-10-21T10:47:13Z) Removing debug pod ...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633