Description of problem: mkfs.xfs 5.19.0 removes support for creating filesystems smaller than 300 MB unless workarounds are employed, either passing --unsupported or setting three (3) environment variables. This shipped to Fedora 37 in Bodhi update <https://bodhi.fedoraproject.org/updates/FEDORA-2023-e1b65ce960>, which documents the removal. However, since it's a breaking change, it should not have shipped to a Fedora stable release per the Fedora stable update policy. Version-Release number of selected component (if applicable): 6.1.0-1.fc37 How reproducible: Always Steps to Reproduce: 1. Have CI tests that create small filesystems 2. Apply Fedora updates Actual results: mkfs.xfs fails with message: "Filesystem must be larger than 300MB." Expected results: mkfs.xfs succeeds Additional info: https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#stable-releases
For some background on this, Eric built this newer xfsprogs for f37 after discussing with me about XFS log sizes on small filesystems (like the ones we bake into FCOS images) being smaller than optimal for larger sizes (this is similar to https://github.com/coreos/fedora-coreos-tracker/issues/1183). For that purpose, we don't actually need the newer xfsprogs to be in the stable f37 repos. As long as it's in the cosa image, that's all we need (we have a koji tag we use for that). Now that it entered the stable repos though, I'm not sure what the procedure is or whether unpushing it from the repos is the right call. If we decide to keep it in the repos, we can pin xfsprogs in f37-based FCOS to the previous v5.18 build (note this is decoupled from the version used in cosa at image build time).
Can I ask which CI caught this? I guess nothing that runs as part of a Bodhi update :( I was a little concerned about this, but in reality sub-300MB filesystems on XFS make no sense for anyone, and should never be used. I understand that this doesn't absolve me of a "breaking change" but TBH I thought the chance of it mattering to anyone was vanishingly low. I can revert it if we have to, though it's unfortunate because there are a lot of performance advantages to the improvements associated with this change. But yeah, I echo Jonathan's question about how to un-push if we have to.
It was caught by Ignition upstream CI, which only saw the package update when it went stable and was built into an updated coreos-assembler container. We can update the CI tests (and will have to anyway, for Fedora 38). I'm more concerned that users could have an Ignition config that creates small XFS filesystems, so that the change would cause provisioning failures. (E.g., maybe they're creating a separate /var/log partition in a small VM?) We're considering whether to announce this to the Fedora CoreOS userbase (via coreos-status@) as a breaking change. AFAIK there's no way to unpush an update once it's reached Fedora stable, so there'd need to be a new package build. It seems very painful to add an epoch in order to force a downgrade, since then the epoch would have to exist forever. Instead, can the offending check be patched out for Fedora 37? --unsupported exists, so apparently the code is still capable of producing small filesystems.
This also broke ostree's CI tests: https://github.com/ostreedev/ostree/pull/2832 I think the upstream change trying to explicitly detect (x)fstests shows that probably this change should have been "warn only" to start, and --unsupported silences the warning or so.
It seems suggestive that ostree and Ignition independently found it easier to switch affected tests to ext4 where testing XFS was not absolutely required. It'd be great if XFS upstream could reconsider its position on small filesystems.
XFS isn't going to reconsider its position on small filesystems, TBH. "300MB ought to be enough for anybody" ;) Many layered projects are starting with tiny provisioned filesystems, and then massively growing the filesystem on deployment, which causes terrible performance down the line (this sort of problem is not unique to XFS). The minimum fs size change was made specifically to accommodate larger logs even for smaller filesystems, so that subsequent growth would be less painful and better-performing. I can patch out the "--unsupported" requirement if you like. There may still be some changes in heuristics but those will probably be less disruptive.
FEDORA-2023-59700c9754 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-59700c9754
I've submitted an update to bodhi to allow small filesystem creation without the --unsupported command line option. If you could test with the previously failing CI, that would be great. Thanks!
FEDORA-2023-59700c9754 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-59700c9754` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-59700c9754 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
I've tested the update in Ignition CI and it passes. Thanks for the fix, and the context.
(In reply to Benjamin Gilbert from comment #10) > I've tested the update in Ignition CI and it passes. Thanks for the fix, > and the context. Thank you! And thanks for pointing out the problems with the original change, and apologies for not thinking it through better. -Eric
FEDORA-2023-59700c9754 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.