Bug 2224081
| Summary: | backport https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Colin Walters <walters> | |
| Component: | ostree | Assignee: | Colin Walters <walters> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Aashish Radhakrishnan <aaradhak> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 9.2 | CC: | aaradhak, cshepher, dornelas, esandeen, hhei, manrique, miabbott, peli, sdodson, travier, wking | |
| Target Milestone: | rc | Keywords: | Triaged, ZStream | |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2224102 2224105 2224106 (view as bug list) | Environment: | ||
| Last Closed: | 2023-07-27 13:59:08 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2224102, 2224105, 2224106 | |||
|
Description
Colin Walters
2023-07-19 18:21:56 UTC
OK, I'm working on tooling to better detect this situation (and after that, repair it).
But basically, this will generally only occur with large disks (which for XFS, result in multiple allocation groups which hand out distinct inode ranges as I understand it). That's how it's sailed through all of our CI so far. (Also, we don't actually verify the deployment roots today, which is an ostree design flaw that makes this problem much less visible; that design flaw will be fixed with the ongoing composefs work)
The original reporter gave this handy bit:
> The issue was discovered on PoC machine equipped with a large NVME (3.4TB), but the bug can be easily reproduced using cosa run -m 4000 --qemu-size +3TB', followed by installation of any package using rpm-ostree install`. The
Using e.g. ls -Rli /ostree/repo/objects - any case where > 32 bit inode numbers are at risk.
Initial analysis code: https://github.com/ostreedev/ostree-rs-ext/pull/504 This will also then morph into "repair" code soon. So the problem isn't directly disk size - the problem is triggered by > 32 bit inodes (an inode is a unique file identifier). So it's more about the allocation pattern for file identifiers, as distinct from the amount of storage those files occupy. The default RHEL (and RHCOS) filesystem is XFS. It happens to be today (at least up to RHEL9) that XFS will generally only use >32 bit inodes when a sufficient number of allocation groups are used. And allocation groups are a function of disk size. In short: disk size => allocation groups => inodes. We support other filesystems too in RHEL, such as ext4. Its behavior for inode allocations are different; I haven't yet gathered enough information to say how likely ext4 is to trigger this. In general a pattern we recommend for large disks is to provision a separate /var filesystem: there's an example of this here: https://docs.openshift.com/container-platform/4.13/installing/install_config/installing-customizing.html#installation-special-config-raid_installing-customizing This pattern will work around this bug. Regarding the WA proposed I understand it is in installation phase but how can you apply the change on day 2 to move the /var to another filesystem? Is it necessary to reinstall the node, and in any case, if all the nodes in the cluster have a similar configuration, is it necessary to reinstall them all before an upgrade? We don't support re-partitioning the disks on day 2 via the MCO (https://github.com/openshift/machine-config-operator/blob/master/docs/FAQ.md#q-can-i-use-the-mco-to-re-partition-or-re-install) so this would be a manual process for existing nodes or re-provisioning nodes. As far as I understand, this only impacts installations with a large / (/sysroot) partition, as this is where the ostree content is stored, so installations with a large disk and no separated /var partitions. This is likely to impact all filesystems but is exacerbated on XFS by the way that we grow the filesystem: https://github.com/coreos/fedora-coreos-tracker/issues/1183 My rough thoughts here are that we will: - Build and test a fixed version of ostree, and it will be available to apply as a hotfix. This should happen today. - Ship this updated version across all supported OCP and RHEL versions. However, having customers manually apply this hotfix won't scale very well. As has happened multiple times in the past now, because the machine-config-daemon daemonset is upgraded *before* nodes upgrade, we can automate hotfixes that way. That would look like: - Inject the new ostree code into the MCD binary in openshift/machine-config-operator - Prepare code in the MCD which applies this hotfix before initiating an OS upgrade: - Flush all merged container images (ensuring we re-fetch them from the network and regenerate the merged filesystem tree with the fixed version of ostree) There's a bit more code being worked on in https://github.com/ostreedev/ostree-rs-ext/pull/504 that will be helpful, but I think not required. I'm still trying to better characterize how to reproduce this. I am *very* quickly reproducing it with current Fedora CoreOS stable (which ships ostree-2023.1-2.fc38.x86_64 which does *not* have the fix). Just a quick: ``` $ cosa run --qemu-size '+3T # rpm-ostree rebase ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:stable ``` Shows that the target filesystem tree before rebooting is just heavily corrupted (your results will vary, but spot checking with the new deployment tree in /sysroot/ostree/deploy...) on various binaries will show this. I am working to publish a container image with my much more strict analysis and repair tooling. However...turning to RHEL: I think this issue is also heavily intersecting with https://github.com/coreos/fedora-coreos-config/pull/2320 because with that fix, we end up with this: ``` $ xfs_info /|grep agcount= meta-data=/dev/vda4 isize=512 agcount=20, agsize=268435455 blks ``` Notice the agcount of 20. I think this ends up making the inode space much more sparse (each AG has a much larger range of inodes). However with current RHCOS 4.12, in a "stock configuration" where we don't reprovision the rootfs, we end up with e.g.: ``` [root@cosa-devsh ~]# xfs_info / meta-data=/dev/vda4 isize=512 agcount=3326, agsize=242112 blks ``` Which is is much, much larger AG count (and one the XFS maintainers think is an important bug, that's why we added the reprovisioning on firstboot). But, this AG count seems to have the side effect of making the inodes *much* less likely to collide by default. And this is another reason I think it took us so long to find this. More analysis incoming. Hi Colin, Kindly let me know if the below process is right with FCOS for verification. aaradhak@fedora ~/fcos2 $ cosa run --qemu-image ./fedora-coreos-38.20230514.3.0-qemu.x86_64.qcow2 + podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/fcos2:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa quay.io/coreos-assembler/coreos-assembler:latest run --qemu-image ./fedora-coreos-38.20230514.3.0-qemu.x86_64.qcow2 Fedora CoreOS 38.20230514.3.0 Tracker: https://github.com/coreos/fedora-coreos-tracker Discuss: https://discussion.fedoraproject.org/tag/coreos Last login: Thu Jul 20 18:20:44 2023 [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ rpm-ostree status State: busy AutomaticUpdatesDriver: Zincati DriverState: active; found update on remote: 38.20230625.3.0 Transaction: deploy --lock-finalization --skip-branch-check revision=e841d77aadb875bb801ac845a0d9b8a70b4224bdeb15e7d6c5bff1da932c0301 --disallow-downgrade Initiator: caller :1.31 Deployments: ● fedora:fedora/x86_64/coreos/stable Version: 38.20230514.3.0 (2023-05-31T10:22:06Z) Commit: a8d8c301ec3e4267106208ed194603e2b1979a67458fb8d3542c54b05b5b1766 GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464 [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ alias verify-ostree="sudo podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run" [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ verify-ostree Trying to pull quay.io/cgwalters/ostree-ext-dev:latest... Getting image source signatures Copying blob dbc9fb14b970 done Copying blob ad5077952f52 [=================>--------------------] 32.3MiB / 70.1MiB Copying blob 19d2bba2169e [=================================>----] 20.5MiB / 22.6MiB Broadcast message from Zincati at Thu 2023-07-20 18:21:54 UTC: New update 38.20230625.3.0 is available and has been deployed. If permitted by the update strategy, Zincati will reboot into this update when Copying blob dbc9fb14b970 done Copying blob ad5077952f52 done Copying blob 19d2bba2169e done Copying config 991ff7104a done Writing manifest to image destination Storing signatures Running in container, assuming we can remount /rootfs/sysroot writable OK no derived commits found. [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable error: Updates and deployments are driven by Zincati (zincati.service) See Zincati's documentation at https://github.com/coreos/zincati Use --bypass-driver to bypass Zincati and perform the operation anyways [core@cosa-devsh ~]$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable --bypass-driver Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable (digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132) ostree chunk layers stored: 0 needed: 51 (727.7 MB) Fetching ostree chunk sha256:2d72193934f5 (49.1 MB) Fetched ostree chunk sha256:2d72193934f5 Fetching ostree chunk sha256:1d1851157929 (39.1 MB) Fetched ostree chunk sha256:1d1851157929 Fetching ostree chunk sha256:38fb7f3103e6 (86.3 MB) Fetched ostree chunk sha256:38fb7f3103e6 Fetching ostree chunk sha256:ea9e24ce3684 (56.7 MB) Fetched ostree chunk sha256:ea9e24ce3684 Fetching ostree chunk sha256:92fc970dd1db (22.1 MB) Fetched ostree chunk sha256:92fc970dd1db Fetching ostree chunk sha256:c0d3e4ab2dfb (18.7 MB) Fetched ostree chunk sha256:c0d3e4ab2dfb Fetching ostree chunk sha256:4e3a86b78101 (14.7 MB) Fetched ostree chunk sha256:4e3a86b78101 Fetching ostree chunk sha256:55ed8482122a (30.9 MB) Fetched ostree chunk sha256:55ed8482122a Fetching ostree chunk sha256:e15d15e60129 (10.2 MB) Fetched ostree chunk sha256:e15d15e60129 Fetching ostree chunk sha256:80409ccf031c (8.1 MB) Fetched ostree chunk sha256:80409ccf031c Fetching ostree chunk sha256:77bf2ccbd010 (15.9 MB) Fetched ostree chunk sha256:77bf2ccbd010 Fetching ostree chunk sha256:c4596e0f9d2f (22.8 MB) Fetched ostree chunk sha256:c4596e0f9d2f Fetching ostree chunk sha256:66c3d1e01b20 (8.6 MB) Fetched ostree chunk sha256:66c3d1e01b20 Fetching ostree chunk sha256:dbed5df4d655 (7.2 MB) Fetched ostree chunk sha256:dbed5df4d655 Fetching ostree chunk sha256:8cb97e7cbe0b (18.4 MB) Fetched ostree chunk sha256:8cb97e7cbe0b Fetching ostree chunk sha256:9a111e34f0a7 (5.7 MB) Fetched ostree chunk sha256:9a111e34f0a7 Fetching ostree chunk sha256:b2852d7115ed (4.5 MB) Fetched ostree chunk sha256:b2852d7115ed Fetching ostree chunk sha256:32e3e4336e95 (4.5 MB) Fetched ostree chunk sha256:32e3e4336e95 Fetching ostree chunk sha256:6e7d355821b7 (3.9 MB) Fetched ostree chunk sha256:6e7d355821b7 Fetching ostree chunk sha256:720bcbf5c3f5 (4.2 MB) Fetched ostree chunk sha256:720bcbf5c3f5 Fetching ostree chunk sha256:306acdde4360 (2.5 MB) Fetched ostree chunk sha256:306acdde4360 Fetching ostree chunk sha256:770cb05ccb97 (3.8 MB) Fetched ostree chunk sha256:770cb05ccb97 Fetching ostree chunk sha256:d83d9388b8c8 (3.4 MB) Fetched ostree chunk sha256:d83d9388b8c8 Fetching ostree chunk sha256:2fd8145ed1e8 (984.6 kB) Fetched ostree chunk sha256:2fd8145ed1e8 Fetching ostree chunk sha256:4556015edf3e (10.0 MB) Fetched ostree chunk sha256:4556015edf3e Fetching ostree chunk sha256:faf33bf71717 (3.1 MB) Fetched ostree chunk sha256:faf33bf71717 Fetching ostree chunk sha256:87e02b8ee016 (4.0 MB) Fetched ostree chunk sha256:87e02b8ee016 Fetching ostree chunk sha256:d153f54f6c23 (1.6 MB) Fetched ostree chunk sha256:d153f54f6c23 Fetching ostree chunk sha256:6ae4e4f5687d (2.7 MB) Fetched ostree chunk sha256:6ae4e4f5687d Fetching ostree chunk sha256:1f3ba420fe2a (3.1 MB) Fetched ostree chunk sha256:1f3ba420fe2a Fetching ostree chunk sha256:12e618cc4270 (918.5 kB) Fetched ostree chunk sha256:12e618cc4270 Fetching ostree chunk sha256:85274c02297e (8.0 MB) Fetched ostree chunk sha256:85274c02297e Fetching ostree chunk sha256:6078667503e5 (34.6 MB) Fetched ostree chunk sha256:6078667503e5 Fetching ostree chunk sha256:c6a22bae3945 (2.2 MB) Fetched ostree chunk sha256:c6a22bae3945 Fetching ostree chunk sha256:6b67185e644b (3.1 MB) Fetched ostree chunk sha256:6b67185e644b Fetching ostree chunk sha256:1a038abae32f (2.5 MB) Fetched ostree chunk sha256:1a038abae32f Fetching ostree chunk sha256:4e59715e2ad8 (1.7 MB) Fetched ostree chunk sha256:4e59715e2ad8 Fetching ostree chunk sha256:bee65ef58f1c (11.3 MB) Fetched ostree chunk sha256:bee65ef58f1c Fetching ostree chunk sha256:4bae9fc1a5a8 (31.3 MB) Fetched ostree chunk sha256:4bae9fc1a5a8 Fetching ostree chunk sha256:2581e41cf6dd (3.7 MB) Fetched ostree chunk sha256:2581e41cf6dd Fetching ostree chunk sha256:bdaa253e11c0 (1.2 MB) Fetched ostree chunk sha256:bdaa253e11c0 Fetching ostree chunk sha256:f32c284711ca (1.2 MB) Fetched ostree chunk sha256:f32c284711ca Fetching ostree chunk sha256:8f773e26787d (10.1 MB) Fetched ostree chunk sha256:8f773e26787d Fetching ostree chunk sha256:50dc647e78c9 (5.1 MB) Fetched ostree chunk sha256:50dc647e78c9 Fetching ostree chunk sha256:b10485f2b393 (4.7 MB) Fetched ostree chunk sha256:b10485f2b393 Fetching ostree chunk sha256:22c06f2cd431 (4.1 MB) Fetched ostree chunk sha256:22c06f2cd431 Fetching ostree chunk sha256:9727f6669e28 (2.3 MB) Fetched ostree chunk sha256:9727f6669e28 Fetching ostree chunk sha256:fa253d5e6b8d (22.1 MB) Fetched ostree chunk sha256:fa253d5e6b8d Fetching ostree chunk sha256:4b2fc5e89c68 (109.3 MB) Fetched ostree chunk sha256:4b2fc5e89c68 Fetching ostree chunk sha256:5f9d3dcf5281 (2.3 kB) Fetched ostree chunk sha256:5f9d3dcf5281 Fetching ostree chunk sha256:074b4ac01d1e (1.4 MB) Fetched ostree chunk sha256:074b4ac01d1e Staging deployment... done Upgraded: afterburn 5.4.0-2.fc38 -> 5.4.2-1.fc38 afterburn-dracut 5.4.0-2.fc38 -> 5.4.2-1.fc38 amd-gpu-firmware 20230404-149.fc38 -> 20230515-150.fc38 atheros-firmware 20230404-149.fc38 -> 20230515-150.fc38 bind-libs 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38 bind-license 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38 bind-utils 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38 brcmfmac-firmware 20230404-149.fc38 -> 20230515-150.fc38 btrfs-progs 6.2.2-1.fc38 -> 6.3.2-1.fc38 c-ares 1.19.0-1.fc38 -> 1.19.1-1.fc38 container-selinux 2:2.211.1-1.fc38 -> 2:2.218.0-1.fc38 crun 1.8.4-1.fc38 -> 1.8.5-1.fc38 curl 8.0.1-1.fc38 -> 8.0.1-2.fc38 dbus 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38 dbus-common 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38 dbus-libs 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38 dnsmasq 2.89-2.fc38 -> 2.89-5.fc38 ethtool 2:6.2-1.fc38 -> 2:6.3-1.fc38 fedora-release-common 38-35 -> 38-36 fedora-release-coreos 38-35 -> 38-36 fedora-release-identity-coreos 38-35 -> 38-36 fuse-overlayfs 1.10-3.fc38 -> 1.12-1.fc38 fwupd 1.8.15-1.fc38 -> 1.9.2-1.fc38 git-core 2.40.1-1.fc38 -> 2.41.0-1.fc38 glib2 2.76.2-1.fc38 -> 2.76.3-1.fc38 intel-gpu-firmware 20230404-149.fc38 -> 20230515-150.fc38 iptables-legacy 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-legacy-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-nft 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-services 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-utils 1.8.9-2.fc38 -> 1.8.9-4.fc38 irqbalance 2:1.9.1-2.fc38 -> 2:1.9.2-1.fc38 kernel 6.2.15-300.fc38 -> 6.3.8-200.fc38 kernel-core 6.2.15-300.fc38 -> 6.3.8-200.fc38 kernel-modules 6.2.15-300.fc38 -> 6.3.8-200.fc38 kernel-modules-core 6.2.15-300.fc38 -> 6.3.8-200.fc38 libcurl-minimal 8.0.1-1.fc38 -> 8.0.1-2.fc38 libgcc 13.1.1-1.fc38 -> 13.1.1-4.fc38 libipa_hbac 2.8.2-4.fc38 -> 2.9.0-1.fc38 libmd 1.0.4-3.fc38 -> 1.1.0-1.fc38 libmodulemd 2.14.0-5.fc38 -> 2.15.0-2.fc38 libsmbclient 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38 libsss_certmap 2.8.2-4.fc38 -> 2.9.0-1.fc38 libsss_idmap 2.8.2-4.fc38 -> 2.9.0-1.fc38 libsss_nss_idmap 2.8.2-4.fc38 -> 2.9.0-1.fc38 libsss_sudo 2.8.2-4.fc38 -> 2.9.0-1.fc38 libstdc++ 13.1.1-1.fc38 -> 13.1.1-4.fc38 libwbclient 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38 libxcrypt 4.4.33-7.fc38 -> 4.4.35-1.fc38 linux-firmware 20230404-149.fc38 -> 20230515-150.fc38 linux-firmware-whence 20230404-149.fc38 -> 20230515-150.fc38 microcode_ctl 2:2.1-54.fc38 -> 2:2.1-55.fc38 mt7xxx-firmware 20230404-149.fc38 -> 20230515-150.fc38 nmstate 2.2.10-4.fc38 -> 2.2.12-3.fc38 nvidia-gpu-firmware 20230404-149.fc38 -> 20230515-150.fc38 openssl 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38 openssl-libs 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38 ostree 2023.1-2.fc38 -> 2023.4-1.fc38 ostree-libs 2023.1-2.fc38 -> 2023.4-1.fc38 podman 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38 podman-plugins 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38 publicsuffix-list-dafsa 20230318-1.fc38 -> 20230614-1.fc38 realtek-firmware 20230404-149.fc38 -> 20230515-150.fc38 rpm-ostree 2023.3-1.fc38 -> 2023.4-2.fc38 rpm-ostree-libs 2023.3-1.fc38 -> 2023.4-2.fc38 rpm-sequoia 1.4.0-2.fc38 -> 1.4.0-3.fc38 samba-client-libs 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38 samba-common 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38 samba-common-libs 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38 selinux-policy 38.12-1.fc38 -> 38.17-1.fc38 selinux-policy-targeted 38.12-1.fc38 -> 38.17-1.fc38 sssd-ad 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-client 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-common 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-common-pac 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-ipa 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-krb5 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-krb5-common 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-ldap 2.8.2-4.fc38 -> 2.9.0-1.fc38 sssd-nfs-idmap 2.8.2-4.fc38 -> 2.9.0-1.fc38 sudo 1.9.13-1.p2.fc38 -> 1.9.13-2.p2.fc38 vim-data 2:9.0.1486-1.fc38 -> 2:9.0.1649-1.fc38 vim-minimal 2:9.0.1486-1.fc38 -> 2:9.0.1649-1.fc38 Added: google-compute-engine-guest-configs-udev-20230526.00-3.fc38.noarch ipcalc-1.0.3-1.fc38.x86_64 passt-0^20230509.g96f8d55-1.fc38.x86_64 passt-selinux-0^20230509.g96f8d55-1.fc38.noarch Changes queued for next boot. Run "systemctl reboot" to start a reboot [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 2473723 Free: 1897947 Available: 1897947 Inodes: Total: 4980160 Free: 4921274 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 0 objects with > 32 bit inode numbers and 30322 objects with <= 32 bit inode numbers OK no colliding inodes found [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ rpm-ostree status State: idle AutomaticUpdatesDriver: Zincati DriverState: active; update staged: 38.20230625.3.0; reboot delayed due to active user sessions Deployments: ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132 Version: 38.20230625.3.0 (2023-07-20T18:24:21Z) Diff: 84 upgraded, 4 added ● fedora:fedora/x86_64/coreos/stable Version: 38.20230514.3.0 (2023-05-31T10:22:06Z) Commit: a8d8c301ec3e4267106208ed194603e2b1979a67458fb8d3542c54b05b5b1766 GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464 [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ rpm -qa ostree ostree-2023.1-2.fc38.x86_64 [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ [core@cosa-devsh ~]$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel --bypass-driver Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel (digest: sha256:9b066cddd8ccdd7d5d00d46a2769afc0c594f7e91beb28687552c0b6bba14d3f) ostree chunk layers stored: 16 needed: 35 (553.8 MB) Fetching ostree chunk sha256:ca3ad6937343 (92.2 MB) Fetched ostree chunk sha256:ca3ad6937343 Fetching ostree chunk sha256:8f060c236f10 (56.8 MB) Fetched ostree chunk sha256:8f060c236f10 Fetching ostree chunk sha256:ab61108bdccc (31.0 MB) Fetched ostree chunk sha256:ab61108bdccc Fetching ostree chunk sha256:29d43d7532fc (8.6 MB) Fetched ostree chunk sha256:29d43d7532fc Fetching ostree chunk sha256:9914a26e2503 (10.1 MB) Fetched ostree chunk sha256:9914a26e2503 Fetching ostree chunk sha256:1304bedcfaa3 (15.9 MB) Fetched ostree chunk sha256:1304bedcfaa3 Fetching ostree chunk sha256:28f61efc79e6 (22.8 MB) Fetched ostree chunk sha256:28f61efc79e6 Fetching ostree chunk sha256:98ad05d12c8a (7.2 MB) Fetched ostree chunk sha256:98ad05d12c8a Fetching ostree chunk sha256:10b550fd6e8b (17.6 MB) Fetched ostree chunk sha256:10b550fd6e8b Fetching ostree chunk sha256:b2b8ed02a657 (5.7 MB) Fetched ostree chunk sha256:b2b8ed02a657 Fetching ostree chunk sha256:2d4192f895b5 (4.5 MB) Fetched ostree chunk sha256:2d4192f895b5 Fetching ostree chunk sha256:4c4f0056c9e8 (10.0 MB) Fetched ostree chunk sha256:4c4f0056c9e8 Fetching ostree chunk sha256:645abb5ac7e4 (3.1 MB) Fetched ostree chunk sha256:645abb5ac7e4 Fetching ostree chunk sha256:e4adf6da1658 (4.0 MB) Fetched ostree chunk sha256:e4adf6da1658 Fetching ostree chunk sha256:7f69978960d1 (2.7 MB) Fetched ostree chunk sha256:7f69978960d1 Fetching ostree chunk sha256:1917ba532d95 (917.7 kB) Fetched ostree chunk sha256:1917ba532d95 Fetching ostree chunk sha256:5ed208adc48c (8.0 MB) Fetched ostree chunk sha256:5ed208adc48c Fetching ostree chunk sha256:733a4b9db00a (34.6 MB) Fetched ostree chunk sha256:733a4b9db00a Fetching ostree chunk sha256:112c9f768dd3 (2.3 MB) Fetched ostree chunk sha256:112c9f768dd3 Fetching ostree chunk sha256:a0a20e2d01cd (2.6 MB) Fetched ostree chunk sha256:a0a20e2d01cd Fetching ostree chunk sha256:21e7be9e5b0a (3.1 MB) Fetched ostree chunk sha256:21e7be9e5b0a Fetching ostree chunk sha256:e96687909700 (1.2 MB) Fetched ostree chunk sha256:e96687909700 Fetching ostree chunk sha256:b07c2c7c2526 (11.3 MB) Fetched ostree chunk sha256:b07c2c7c2526 Fetching ostree chunk sha256:0e1c1f114196 (31.4 MB) Fetched ostree chunk sha256:0e1c1f114196 Fetching ostree chunk sha256:762c2cd46bf4 (4.0 MB) Fetched ostree chunk sha256:762c2cd46bf4 Fetching ostree chunk sha256:7dd9eaad5ccb (1.5 MB) Fetched ostree chunk sha256:7dd9eaad5ccb Fetching ostree chunk sha256:4d187e9b978a (660.6 kB) Fetched ostree chunk sha256:4d187e9b978a Fetching ostree chunk sha256:18efcb95f6ba (12.1 MB) Fetched ostree chunk sha256:18efcb95f6ba Fetching ostree chunk sha256:d0a7b2958ebf (5.3 MB) Fetched ostree chunk sha256:d0a7b2958ebf Fetching ostree chunk sha256:b34fecc0d98a (4.5 MB) Fetched ostree chunk sha256:b34fecc0d98a Fetching ostree chunk sha256:1d71a8f5c106 (4.2 MB) Fetched ostree chunk sha256:1d71a8f5c106 Fetching ostree chunk sha256:e353c6b5901b (2.5 MB) Fetched ostree chunk sha256:e353c6b5901b Fetching ostree chunk sha256:02aa7defe84f (20.4 MB) Fetched ostree chunk sha256:02aa7defe84f Fetching ostree chunk sha256:368f77051bee (109.6 MB) Fetched ostree chunk sha256:368f77051bee Fetching ostree chunk sha256:98dd1141e87d (1.4 MB) Fetched ostree chunk sha256:98dd1141e87d Staging deployment... done Upgraded: NetworkManager 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38 NetworkManager-cloud-setup 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38 NetworkManager-libnm 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38 NetworkManager-team 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38 NetworkManager-tui 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38 aardvark-dns 1.6.0-1.fc38 -> 1.7.0-1.fc38 afterburn 5.4.0-2.fc38 -> 5.4.2-1.fc38 afterburn-dracut 5.4.0-2.fc38 -> 5.4.2-1.fc38 amd-gpu-firmware 20230404-149.fc38 -> 20230625-151.fc38 atheros-firmware 20230404-149.fc38 -> 20230625-151.fc38 bind-libs 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38 bind-license 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38 bind-utils 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38 brcmfmac-firmware 20230404-149.fc38 -> 20230625-151.fc38 btrfs-progs 6.2.2-1.fc38 -> 6.3.2-1.fc38 c-ares 1.19.0-1.fc38 -> 1.19.1-1.fc38 container-selinux 2:2.211.1-1.fc38 -> 2:2.219.0-1.fc38 cracklib 2.9.7-31.fc38 -> 2.9.11-1.fc38 crun 1.8.4-1.fc38 -> 1.8.5-1.fc38 curl 8.0.1-1.fc38 -> 8.0.1-2.fc38 dbus 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38 dbus-common 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38 dbus-libs 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38 diffutils 3.9-1.fc38 -> 3.10-1.fc38 dnsmasq 2.89-2.fc38 -> 2.89-5.fc38 elfutils-default-yama-scope 0.189-2.fc38 -> 0.189-3.fc38 elfutils-libelf 0.189-2.fc38 -> 0.189-3.fc38 elfutils-libs 0.189-2.fc38 -> 0.189-3.fc38 ethtool 2:6.2-1.fc38 -> 2:6.4-1.fc38 fedora-release-common 38-35 -> 38-36 fedora-release-coreos 38-35 -> 38-36 fedora-release-identity-coreos 38-35 -> 38-36 fuse-overlayfs 1.10-3.fc38 -> 1.12-1.fc38 fwupd 1.8.15-1.fc38 -> 1.9.2-1.fc38 git-core 2.40.1-1.fc38 -> 2.41.0-1.fc38 glib2 2.76.2-1.fc38 -> 2.76.4-1.fc38 ignition 2.15.0-3.fc38 -> 2.16.2-1.fc38 inih 56-3.fc38 -> 57-1.fc38 intel-gpu-firmware 20230404-149.fc38 -> 20230625-151.fc38 iptables-legacy 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-legacy-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-nft 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-services 1.8.9-2.fc38 -> 1.8.9-4.fc38 iptables-utils 1.8.9-2.fc38 -> 1.8.9-4.fc38 irqbalance 2:1.9.1-2.fc38 -> 2:1.9.2-1.fc38 kernel 6.2.15-300.fc38 -> 6.3.12-200.fc38 kernel-core 6.2.15-300.fc38 -> 6.3.12-200.fc38 kernel-modules 6.2.15-300.fc38 -> 6.3.12-200.fc38 kernel-modules-core 6.2.15-300.fc38 -> 6.3.12-200.fc38 krb5-libs 1.20.1-8.fc38 -> 1.21-2.fc38 libassuan 2.5.5-6.fc38 -> 2.5.6-1.fc38 libcurl-minimal 8.0.1-1.fc38 -> 8.0.1-2.fc38 libgcc 13.1.1-1.fc38 -> 13.1.1-4.fc38 libipa_hbac 2.8.2-4.fc38 -> 2.9.1-1.fc38 libksba 1.6.3-2.fc38 -> 1.6.4-1.fc38 libmd 1.0.4-3.fc38 -> 1.1.0-1.fc38 libmodulemd 2.14.0-5.fc38 -> 2.15.0-2.fc38 libnetfilter_conntrack 1.0.8-7.fc38 -> 1.0.9-1.fc38 libreport-filesystem 2.17.10-1.fc38 -> 2.17.11-1.fc38 libsmbclient 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38 libsss_certmap 2.8.2-4.fc38 -> 2.9.1-1.fc38 libsss_idmap 2.8.2-4.fc38 -> 2.9.1-1.fc38 libsss_nss_idmap 2.8.2-4.fc38 -> 2.9.1-1.fc38 libsss_sudo 2.8.2-4.fc38 -> 2.9.1-1.fc38 libstdc++ 13.1.1-1.fc38 -> 13.1.1-4.fc38 libuv 1:1.44.2-3.fc38 -> 1:1.46.0-1.fc38 libwbclient 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38 libxcrypt 4.4.33-7.fc38 -> 4.4.36-1.fc38 linux-firmware 20230404-149.fc38 -> 20230625-151.fc38 linux-firmware-whence 20230404-149.fc38 -> 20230625-151.fc38 lmdb-libs 0.9.30-1.fc38 -> 0.9.31-1.fc38 microcode_ctl 2:2.1-54.fc38 -> 2:2.1-55.fc38 mt7xxx-firmware 20230404-149.fc38 -> 20230625-151.fc38 netavark 1.6.0-2.fc38 -> 1.7.0-1.fc38 nmstate 2.2.10-4.fc38 -> 2.2.12-3.fc38 nvidia-gpu-firmware 20230404-149.fc38 -> 20230625-151.fc38 openssl 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38 openssl-libs 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38 ostree 2023.1-2.fc38 -> 2023.4-2.fc38 ostree-libs 2023.1-2.fc38 -> 2023.4-2.fc38 podman 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38 podman-plugins 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38 publicsuffix-list-dafsa 20230318-1.fc38 -> 20230614-1.fc38 realtek-firmware 20230404-149.fc38 -> 20230625-151.fc38 rpm-ostree 2023.3-1.fc38 -> 2023.5-1.fc38 rpm-ostree-libs 2023.3-1.fc38 -> 2023.5-1.fc38 rpm-sequoia 1.4.0-2.fc38 -> 1.4.1-1.fc38 samba-client-libs 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38 samba-common 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38 samba-common-libs 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38 selinux-policy 38.12-1.fc38 -> 38.21-1.fc38 selinux-policy-targeted 38.12-1.fc38 -> 38.21-1.fc38 skopeo 1:1.12.0-1.fc38 -> 1:1.13.0-1.fc38 sssd-ad 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-client 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-common 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-common-pac 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-ipa 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-krb5 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-krb5-common 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-ldap 2.8.2-4.fc38 -> 2.9.1-1.fc38 sssd-nfs-idmap 2.8.2-4.fc38 -> 2.9.1-1.fc38 sudo 1.9.13-1.p2.fc38 -> 1.9.13-2.p2.fc38 vim-data 2:9.0.1486-1.fc38 -> 2:9.0.1671-1.fc38 vim-minimal 2:9.0.1486-1.fc38 -> 2:9.0.1671-1.fc38 yajl 2.1.0-20.fc38 -> 2.1.0-21.fc38 Added: google-compute-engine-guest-configs-udev-20230526.00-3.fc38.noarch ipcalc-1.0.3-1.fc38.x86_64 passt-0^20230625.g32660ce-1.fc38.x86_64 passt-selinux-0^20230625.g32660ce-1.fc38.noarch Changes queued for next boot. Run "systemctl reboot" to start a reboot [core@cosa-devsh ~]$ Broadcast message from Zincati at Thu 2023-07-20 18:30:55 UTC: New update 38.20230625.3.0 is available and has been deployed. If permitted by the update strategy, Zincati will reboot into this update when all interactive users have logged out, or in 1 minute, whichever comes earlier. Please log out of all active sessions in order to let the auto-update process continue. [core@cosa-devsh ~]$ rpm -qa ostree ostree-2023.1-2.fc38.x86_64 [core@cosa-devsh ~]$ verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 2473723 Free: 1790941 Available: 1790941 Inodes: Total: 4980160 Free: 4914850 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 0 objects with > 32 bit inode numbers and 36720 objects with <= 32 bit inode numbers OK no colliding inodes found Just for reference: CentOS Stream 9: already has ostree-2023.4-2.el9 which contains the fix for this bug (this update was already shipped earlier). CentOS Stream 8: MR in https://gitlab.com/redhat/centos-stream/rpms/ostree/-/merge_requests/22 Just going to write for the record here that I consider this bug very embarrassing - the integrity and reliability of operating system upgrades is really The Job that we're trying to do here. It's my top priority to fix this.
Further, I can say that we already have some in-flight work upstream in "composefs" which will provide significantly stronger mechanisms for integrity, but also crucially will enable us to share a fully common storage model between the OS updates and containers, which will ensure both work the same way.
But, OK so an immediate thing we're trying to work through here is reproducing this reliably in RHCOS 4.13 (RHEL 9.2) and RHCOS 4.12 (RHEL 8.6). And the reason for this seems to again have to do with how many XFS allocation groups are in use in the disk images we generate there. XFS seems to tend to prefer using lower AG numbers for allocations, which in turn imply most inodes are going to be <= 32 bit which avoids collision. Again, this is one of the compounding factors that let this bug get through our CI so far.
I tried to quickly test with ext4, but what I'm seeing there is that its inode allocation is quite serial; with XFS a simple script to just create and unlink files will *sometimes* get > 32 bit inodes (because they may be allocated in a later AG), but as far as I can tell never with ext4 - unless we actually tried creating billions of files. I think this architectural difference between XFS and ext4 is why XFS can scales better: because of how it internally "subdivides" everything into AGs, whereas ext4 has more filesystem-global metadata.
It's likely we could likely force this situation in XFS to be more likely by actually filling up the disk space, but at terabyte sizes that's going to get expensive to do in CI and testing. There may be a way to try to steer things to "later" AGs, but:
[time passes]
OK, I figured out how to craft a Butane (Ignition) configuration that will force Ignition to regenerate the XFS filesystem at the full new size:
variant: fcos
version: 1.4.0
storage:
disks:
- device: /dev/disk/by-id/coreos-boot-disk
wipe_table: false
partitions:
- number: 4
label: root
# Allocate 2.8T for the root
size_mib: 2936012.8
resize: true
- size_mib: 0
label: foo
filesystems:
- device: /dev/disk/by-partlabel/root
wipe_filesystem: true
format: xfs
label: root
- path: /var/mnt/foo
device: /dev/disk/by-partlabel/foo
# We can select the filesystem we'd like.
format: xfs
# Ask Butane to generate a mount unit for us so that this filesystem
# gets mounted in the real root.
with_mount_unit: true
I've verified this Butane config works with 4.12 and 4.13, and crucially we now see:
```
$ xfs_info / |grep agcount=
meta-data=/dev/vda4 isize=512 agcount=4, agsize=187904768 blks
```
An AG count of just 4 (which is actually better for XFS) means each AG has a much, much greater inode allocation range.
And in this situation, running the reproducer, the analysis tooling now very reliably finds corruption; notice we have almost 50% of inodes that are > 32 bit:
```
Gathering inodes for ostree objects...
Analyzed 23467 objects with > 32 bit inode numbers and 25411 objects with <= 32 bit inode numbers
warning: 443 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
inode clashes: 19
unknown: 0
ok: 572
warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled
```
Hi Colin, I am able to reproduce the collision with the butane config. aaradhak@fedora ~/rhcos/rhcos-4.13 $ cosa run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size '+3T' --qemu-memory 8192 -B reprovision-xfs.bu COREOS_ASSEMBLER_CONTAINER=quay.io/coreos-assembler/coreos-assembler:rhcos-4.13 COREOS_ASSEMBLER_ADD_CERTS=y + podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/rhcos/rhcos-4.13:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa -v=/etc/pki/ca-trust:/etc/pki/ca-trust:ro quay.io/coreos-assembler/coreos-assembler:rhcos-4.13 run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size +3T --qemu-memory 8192 -B reprovision-xfs.bu Red Hat Enterprise Linux CoreOS 413.92.202305021736-0 Part of OpenShift 4.13, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.13/architecture/architecture-rhcos.html --- Last login: Fri Jul 21 15:11:47 2023 [core@cosa-devsh ~]$ xfs_info / |grep agcount= meta-data=/dev/vda4 isize=512 agcount=4, agsize=187904768 blks [core@cosa-devsh ~]$ sudo -i [root@cosa-devsh ~]# alias verify-ostree="podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run" [root@cosa-devsh ~]# verify-ostree Trying to pull quay.io/cgwalters/ostree-ext-dev:latest... Getting image source signatures Copying blob ad5077952f52 done Copying blob dbc9fb14b970 done Copying blob 19d2bba2169e done Copying config 991ff7104a done Writing manifest to image destination Storing signatures Running in container, assuming we can remount /rootfs/sysroot writable OK no derived commits found. [root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable (digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132) ostree chunk layers stored: 0 needed: 51 (727.7 MB) Fetching ostree chunk sha256:2d72193934f5 (49.1 MB) Fetched ostree chunk sha256:2d72193934f5 Fetching ostree chunk sha256:1d1851157929 (39.1 MB) Fetched ostree chunk sha256:1d1851157929 Fetching ostree chunk sha256:38fb7f3103e6 (86.3 MB) Fetched ostree chunk sha256:38fb7f3103e6 Fetching ostree chunk sha256:ea9e24ce3684 (56.7 MB) Fetched ostree chunk sha256:ea9e24ce3684 Fetching ostree chunk sha256:92fc970dd1db (22.1 MB) Fetched ostree chunk sha256:92fc970dd1db Fetching ostree chunk sha256:c0d3e4ab2dfb (18.7 MB) Fetched ostree chunk sha256:c0d3e4ab2dfb Fetching ostree chunk sha256:4e3a86b78101 (14.7 MB) Fetched ostree chunk sha256:4e3a86b78101 Fetching ostree chunk sha256:55ed8482122a (30.9 MB) Fetched ostree chunk sha256:55ed8482122a Fetching ostree chunk sha256:e15d15e60129 (10.2 MB) Fetched ostree chunk sha256:e15d15e60129 Fetching ostree chunk sha256:80409ccf031c (8.1 MB) Fetched ostree chunk sha256:80409ccf031c Fetching ostree chunk sha256:77bf2ccbd010 (15.9 MB) Fetched ostree chunk sha256:77bf2ccbd010 Fetching ostree chunk sha256:c4596e0f9d2f (22.8 MB) Fetched ostree chunk sha256:c4596e0f9d2f Fetching ostree chunk sha256:66c3d1e01b20 (8.6 MB) Fetched ostree chunk sha256:66c3d1e01b20 Fetching ostree chunk sha256:dbed5df4d655 (7.2 MB) Fetched ostree chunk sha256:dbed5df4d655 Fetching ostree chunk sha256:8cb97e7cbe0b (18.4 MB) Fetched ostree chunk sha256:8cb97e7cbe0b Fetching ostree chunk sha256:9a111e34f0a7 (5.7 MB) Fetched ostree chunk sha256:9a111e34f0a7 Fetching ostree chunk sha256:b2852d7115ed (4.5 MB) Fetched ostree chunk sha256:b2852d7115ed Fetching ostree chunk sha256:32e3e4336e95 (4.5 MB) Fetched ostree chunk sha256:32e3e4336e95 Fetching ostree chunk sha256:6e7d355821b7 (3.9 MB) Fetched ostree chunk sha256:6e7d355821b7 Fetching ostree chunk sha256:720bcbf5c3f5 (4.2 MB) Fetched ostree chunk sha256:720bcbf5c3f5 Fetching ostree chunk sha256:306acdde4360 (2.5 MB) Fetched ostree chunk sha256:306acdde4360 Fetching ostree chunk sha256:770cb05ccb97 (3.8 MB) Fetched ostree chunk sha256:770cb05ccb97 Fetching ostree chunk sha256:d83d9388b8c8 (3.4 MB) Fetched ostree chunk sha256:d83d9388b8c8 Fetching ostree chunk sha256:2fd8145ed1e8 (984.6 kB) Fetched ostree chunk sha256:2fd8145ed1e8 Fetching ostree chunk sha256:4556015edf3e (10.0 MB) Fetched ostree chunk sha256:4556015edf3e Fetching ostree chunk sha256:faf33bf71717 (3.1 MB) Fetched ostree chunk sha256:faf33bf71717 Fetching ostree chunk sha256:87e02b8ee016 (4.0 MB) Fetched ostree chunk sha256:87e02b8ee016 Fetching ostree chunk sha256:d153f54f6c23 (1.6 MB) Fetched ostree chunk sha256:d153f54f6c23 Fetching ostree chunk sha256:6ae4e4f5687d (2.7 MB) Fetched ostree chunk sha256:6ae4e4f5687d Fetching ostree chunk sha256:1f3ba420fe2a (3.1 MB) Fetched ostree chunk sha256:1f3ba420fe2a Fetching ostree chunk sha256:12e618cc4270 (918.5 kB) Fetched ostree chunk sha256:12e618cc4270 Fetching ostree chunk sha256:85274c02297e (8.0 MB) Fetched ostree chunk sha256:85274c02297e Fetching ostree chunk sha256:6078667503e5 (34.6 MB) Fetched ostree chunk sha256:6078667503e5 Fetching ostree chunk sha256:c6a22bae3945 (2.2 MB) Fetched ostree chunk sha256:c6a22bae3945 Fetching ostree chunk sha256:6b67185e644b (3.1 MB) Fetched ostree chunk sha256:6b67185e644b Fetching ostree chunk sha256:1a038abae32f (2.5 MB) Fetched ostree chunk sha256:1a038abae32f Fetching ostree chunk sha256:4e59715e2ad8 (1.7 MB) Fetched ostree chunk sha256:4e59715e2ad8 Fetching ostree chunk sha256:bee65ef58f1c (11.3 MB) Fetched ostree chunk sha256:bee65ef58f1c Fetching ostree chunk sha256:4bae9fc1a5a8 (31.3 MB) Fetched ostree chunk sha256:4bae9fc1a5a8 Fetching ostree chunk sha256:2581e41cf6dd (3.7 MB) Fetched ostree chunk sha256:2581e41cf6dd Fetching ostree chunk sha256:bdaa253e11c0 (1.2 MB) Fetched ostree chunk sha256:bdaa253e11c0 Fetching ostree chunk sha256:f32c284711ca (1.2 MB) Fetched ostree chunk sha256:f32c284711ca Fetching ostree chunk sha256:8f773e26787d (10.1 MB) Fetched ostree chunk sha256:8f773e26787d Fetching ostree chunk sha256:50dc647e78c9 (5.1 MB) Fetched ostree chunk sha256:50dc647e78c9 Fetching ostree chunk sha256:b10485f2b393 (4.7 MB) Fetched ostree chunk sha256:b10485f2b393 Fetching ostree chunk sha256:22c06f2cd431 (4.1 MB) Fetched ostree chunk sha256:22c06f2cd431 Fetching ostree chunk sha256:9727f6669e28 (2.3 MB) Fetched ostree chunk sha256:9727f6669e28 Fetching ostree chunk sha256:fa253d5e6b8d (22.1 MB) Fetched ostree chunk sha256:fa253d5e6b8d Fetching ostree chunk sha256:4b2fc5e89c68 (109.3 MB) Fetched ostree chunk sha256:4b2fc5e89c68 Fetching ostree chunk sha256:5f9d3dcf5281 (2.3 kB) Fetched ostree chunk sha256:5f9d3dcf5281 Fetching ostree chunk sha256:074b4ac01d1e (1.4 MB) Fetched ostree chunk sha256:074b4ac01d1e Staging deployment... done Upgraded: NetworkManager 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38 NetworkManager-cloud-setup 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38 NetworkManager-libnm 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38 NetworkManager-team 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38 NetworkManager-tui 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38 WALinuxAgent-udev 2.7.0.6-9.el9 -> 2.9.0.4-1.fc38 acl 2.3.1-3.el9 -> 2.3.1-6.fc38 adcli 0.9.2-1.el9 -> 0.9.2-2.fc38 afterburn 5.4.1-1.el9 -> 5.4.2-1.fc38 afterburn-dracut 5.4.1-1.el9 -> 5.4.2-1.fc38 alternatives 1.20-2.el9 -> 1.24-1.fc38 attr 2.5.1-3.el9 -> 2.5.1-6.fc38 audit-libs 3.0.7-103.el9 -> 3.1.1-1.fc38 authselect 1.2.6-1.el9 -> 1.4.2-2.fc38 authselect-libs 1.2.6-1.el9 -> 1.4.2-2.fc38 avahi-libs 0.8-12.el9 -> 0.8-22.fc38 basesystem 11-13.el9 -> 11-15.fc38 bash 5.1.8-6.el9_1 -> 5.2.15-3.fc38 bash-completion 1:2.11-4.el9 -> 1:2.11-9.fc38 bind-libs 32:9.16.23-11.el9 -> 32:9.18.16-1.fc38 bind-license 32:9.16.23-11.el9 -> 32:9.18.16-1.fc38 bind-utils 32:9.16.23-11.el9 -> 32:9.18.16-1.fc38 bootupd 0.2.7-2.el9 -> 0.2.9-1.fc38 bsdtar 3.5.3-4.el9 -> 3.6.1-4.fc38 bubblewrap 0.4.1-6.el9 -> 0.7.0-1.fc38 bzip2 1.0.8-8.el9 -> 1.0.8-13.fc38 bzip2-libs 1.0.8-8.el9 -> 1.0.8-13.fc38 c-ares 1.17.1-5.el9 -> 1.19.1-1.fc38 ca-certificates 2022.2.54-90.2.el9_0 -> 2023.2.60-2.fc38 chrony 4.3-1.el9 -> 4.3-3.fc38 cifs-utils 7.0-1.el9 -> 7.0-1.fc38 clevis 18-110.el9 -> 19-2.fc38 clevis-dracut 18-110.el9 -> 19-2.fc38 clevis-luks 18-110.el9 -> 19-2.fc38 clevis-systemd 18-110.el9 -> 19-2.fc38 console-login-helper-messages 0.21.3-1.el9 -> 0.21.3-4.fc38 console-login-helper-messages-issuegen 0.21.3-1.el9 -> 0.21.3-4.fc38 console-login-helper-messages-profile 0.21.3-1.el9 -> 0.21.3-4.fc38 containers-common 3:1-34.rhaos4.13.el9 -> 4:1-89.fc38 coreos-installer 0.17.0-1.rhaos4.13.el9 -> 0.17.0-3.fc38 coreos-installer-bootinfra 0.17.0-1.rhaos4.13.el9 -> 0.17.0-3.fc38 coreutils 8.32-34.el9 -> 9.1-12.fc38 coreutils-common 8.32-34.el9 -> 9.1-12.fc38 cracklib 2.9.6-27.el9 -> 2.9.7-31.fc38 criu 3.17-4.el9 -> 3.17.1-5.fc38 criu-libs 3.17-4.el9 -> 3.17.1-5.fc38 crun 1.8.4-1.rhaos4.13.el9 -> 1.8.5-1.fc38 crypto-policies 20221215-1.git9a18988.el9 -> 20230301-1.gita12f7b2.fc38 cryptsetup 2.6.0-2.el9 -> 2.6.1-1.fc38 cryptsetup-libs 2.6.0-2.el9 -> 2.6.1-1.fc38 curl 7.76.1-23.el9 -> 8.0.1-2.fc38 cyrus-sasl-gssapi 2.1.27-21.el9 -> 2.1.28-9.fc38 cyrus-sasl-lib 2.1.27-21.el9 -> 2.1.28-9.fc38 dbus 1:1.12.20-7.el9_1 -> 1:1.14.8-1.fc38 dbus-broker 28-7.el9 -> 33-1.fc38 dbus-common 1:1.12.20-7.el9_1 -> 1:1.14.8-1.fc38 dbus-libs 1:1.12.20-7.el9_1 -> 1:1.14.8-1.fc38 device-mapper-multipath 0.8.7-20.el9 -> 0.9.4-2.fc38 device-mapper-multipath-libs 0.8.7-20.el9 -> 0.9.4-2.fc38 diffutils 3.7-12.el9 -> 3.9-1.fc38 dnsmasq 2.85-6.el9 -> 2.89-5.fc38 dosfstools 4.2-3.el9 -> 4.2-6.fc38 dracut 057-21.git20230214.el9 -> 059-3.fc38 dracut-network 057-21.git20230214.el9 -> 059-3.fc38 dracut-squash 057-21.git20230214.el9 -> 059-3.fc38 e2fsprogs 1.46.5-3.el9 -> 1.46.5-4.fc38 e2fsprogs-libs 1.46.5-3.el9 -> 1.46.5-4.fc38 efibootmgr 16-12.el9 -> 18-3.fc38 efivar-libs 38-3.el9 -> 38-7.fc38 elfutils-default-yama-scope 0.188-3.el9 -> 0.189-2.fc38 elfutils-libelf 0.188-3.el9 -> 0.189-2.fc38 elfutils-libs 0.188-3.el9 -> 0.189-2.fc38 ethtool 2:5.16-1.el9 -> 2:6.3-1.fc38 expat 2.5.0-1.el9 -> 2.5.0-2.fc38 file 5.39-12.el9 -> 5.44-3.fc38 file-libs 5.39-12.el9 -> 5.44-3.fc38 filesystem 3.16-2.el9 -> 3.18-3.fc38 findutils 1:4.8.0-5.el9 -> 1:4.9.0-3.fc38 fstrm 0.6.1-3.el9 -> 0.6.1-6.fc38 fuse 2.9.9-15.el9 -> 2.9.9-16.fc38 fuse-common 3.10.2-5.el9 -> 3.14.1-1.fc38 fuse-libs 2.9.9-15.el9 -> 2.9.9-16.fc38 fuse-overlayfs 1.10-2.el9 -> 1.12-1.fc38 fuse3 3.10.2-5.el9 -> 3.14.1-1.fc38 fuse3-libs 3.10.2-5.el9 -> 3.14.1-1.fc38 fwupd 1.8.10-2.el9 -> 1.9.2-1.fc38 gawk 5.1.0-6.el9 -> 5.1.1-5.fc38 gdbm-libs 1:1.19-4.el9 -> 1:1.23-3.fc38 gdisk 1.0.7-5.el9 -> 1.0.9-5.fc38 gettext-libs 0.21-7.el9 -> 0.21.1-2.fc38 git-core 2.39.1-1.el9 -> 2.41.0-1.fc38 glib2 2.68.4-6.el9 -> 2.76.3-1.fc38 glibc 2.34-60.el9 -> 2.37-4.fc38 glibc-common 2.34-60.el9 -> 2.37-4.fc38 glibc-gconv-extra 2.34-60.el9 -> 2.37-4.fc38 glibc-minimal-langpack 2.34-60.el9 -> 2.37-4.fc38 gmp 1:6.2.0-10.el9 -> 1:6.2.1-4.fc38 gnupg2 2.3.3-2.el9_0 -> 2.4.0-3.fc38 gnutls 3.7.6-20.el9_2 -> 3.8.0-2.fc38 gpgme 1.15.1-6.el9 -> 1.17.1-3.fc38 grep 3.6-5.el9 -> 3.8-3.fc38 grub2-common 1:2.06-61.el9 -> 1:2.06-95.fc38 grub2-efi-x64 1:2.06-61.el9 -> 1:2.06-95.fc38 grub2-pc 1:2.06-61.el9 -> 1:2.06-95.fc38 grub2-pc-modules 1:2.06-61.el9 -> 1:2.06-95.fc38 grub2-tools 1:2.06-61.el9 -> 1:2.06-95.fc38 grub2-tools-minimal 1:2.06-61.el9 -> 1:2.06-95.fc38 gzip 1.12-1.el9 -> 1.12-3.fc38 hostname 3.23-6.el9 -> 3.23-8.fc38 ignition 2.15.0-1.el9 -> 2.15.0-3.fc38 inih 49-6.el9 -> 56-3.fc38 iproute 6.1.0-1.el9 -> 6.1.0-1.fc38 iproute-tc 6.1.0-1.el9 -> 6.1.0-1.fc38 iptables-libs 1.8.8-6.el9_1 -> 1.8.9-4.fc38 iptables-nft 1.8.8-6.el9_1 -> 1.8.9-4.fc38 iputils 20210202-8.el9 -> 20221126-2.fc38 irqbalance 2:1.9.0-3.el9 -> 2:1.9.2-1.fc38 iscsi-initiator-utils 6.2.1.4-3.git2a8f9d8.el9 -> 6.2.1.4-10.git2a8f9d8.fc38 iscsi-initiator-utils-iscsiuio 6.2.1.4-3.git2a8f9d8.el9 -> 6.2.1.4-10.git2a8f9d8.fc38 isns-utils-libs 0.101-4.el9 -> 0.101-6.fc38 jose 11-3.el9 -> 11-7.fc38 jq 1.6-14.el9 -> 1.6-15.fc38 json-c 0.14-11.el9 -> 0.16-4.fc38 json-glib 1.6.6-1.el9 -> 1.6.6-4.fc38 kbd 2.4.0-8.el9 -> 2.5.1-5.fc38 kbd-misc 2.4.0-8.el9 -> 2.5.1-5.fc38 kernel 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38 kernel-core 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38 kernel-modules 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38 kernel-modules-core 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38 kexec-tools 2.0.25-13.el9_2 -> 2.0.26-3.fc38 kmod 28-7.el9 -> 30-4.fc38 kmod-libs 28-7.el9 -> 30-4.fc38 kpartx 0.8.7-20.el9 -> 0.9.4-2.fc38 krb5-libs 1.20.1-8.el9 -> 1.20.1-8.fc38 less 590-1.el9_0 -> 633-1.fc38 libacl 2.3.1-3.el9 -> 2.3.1-6.fc38 libaio 0.3.111-13.el9 -> 0.3.111-15.fc38 libarchive 3.5.3-4.el9 -> 3.6.1-4.fc38 libassuan 2.5.5-3.el9 -> 2.5.5-6.fc38 libattr 2.5.1-3.el9 -> 2.5.1-6.fc38 libbasicobjects 0.1.1-53.el9 -> 0.1.1-53.fc38 libblkid 2.37.4-10.el9 -> 2.38.1-4.fc38 libbpf 2:1.0.0-2.el9 -> 2:1.1.0-2.fc38 libcap-ng 0.8.2-7.el9 -> 0.8.3-5.fc38 libcbor 0.7.0-5.el9 -> 0.7.0-9.fc38 libcollection 0.7.0-53.el9 -> 0.7.0-53.fc38 libcom_err 1.46.5-3.el9 -> 1.46.5-4.fc38 libdaemon 0.14-23.el9 -> 0.14-25.fc38 libdb 5.3.28-53.el9 -> 5.3.28-55.fc38 libdhash 0.5.0-53.el9 -> 0.5.0-53.fc38 libedit 3.1-37.20210216cvs.el9 -> 3.1-45.20221030cvs.fc38 libevent 2.1.12-6.el9 -> 2.1.12-8.fc38 libfdisk 2.37.4-10.el9 -> 2.38.1-4.fc38 libffi 3.4.2-7.el9 -> 3.4.4-2.fc38 libfido2 1.6.0-7.el9 -> 1.12.0-3.fc38 libgcab1 1.4-6.el9 -> 1.5-2.fc38 libgcc 11.3.1-4.3.el9 -> 13.1.1-4.fc38 libgcrypt 1.10.0-10.el9_1 -> 1.10.2-1.fc38 libgpg-error 1.42-5.el9 -> 1.47-1.fc38 libgudev 237-1.el9 -> 237-4.fc38 libgusb 0.3.8-1.el9 -> 0.4.5-1.fc38 libibverbs 44.0-2.el9 -> 44.0-3.fc38 libicu 67.1-9.el9 -> 72.1-2.fc38 libidn2 2.3.0-7.el9 -> 2.3.4-2.fc38 libini_config 1.3.1-53.el9 -> 1.3.1-53.fc38 libipa_hbac 2.8.2-2.el9 -> 2.9.0-1.fc38 libjcat 0.1.6-3.el9 -> 0.1.13-1.fc38 libjose 11-3.el9 -> 11-7.fc38 libkcapi 1.3.1-3.el9 -> 1.4.0-5.fc38 libkcapi-hmaccalc 1.3.1-3.el9 -> 1.4.0-5.fc38 libksba 1.5.1-6.el9_1 -> 1.6.3-2.fc38 libldb 2.6.1-1.el9 -> 2.7.2-1.fc38 libluksmeta 9-12.el9 -> 9-15.fc38 libmaxminddb 1.5.2-3.el9 -> 1.7.1-2.fc38 libmnl 1.0.4-15.el9 -> 1.0.5-2.fc38 libmodulemd 2.13.0-2.el9 -> 2.15.0-2.fc38 libmount 2.37.4-10.el9 -> 2.38.1-4.fc38 libndp 1.8-4.el9 -> 1.8-5.fc38 libnet 1.2-6.el9 -> 1.2-7.fc38 libnfnetlink 1.0.1-21.el9 -> 1.0.1-23.fc38 libnfsidmap 1:2.5.4-18.el9 -> 1:2.6.3-0.fc38 libnftnl 1.2.2-1.el9 -> 1.2.4-2.fc38 libnghttp2 1.43.0-5.el9 -> 1.52.0-1.fc38 libnl3 3.7.0-1.el9 -> 3.7.0-3.fc38 libnl3-cli 3.7.0-1.el9 -> 3.7.0-3.fc38 libnvme 1.2-2.el9 -> 1.4-2.fc38 libpath_utils 0.2.1-53.el9 -> 0.2.1-53.fc38 libpcap 14:1.10.0-4.el9 -> 14:1.10.4-1.fc38 libpkgconf 1.7.3-10.el9 -> 1.8.0-6.fc38 libpsl 0.21.1-5.el9 -> 0.21.2-2.fc38 libpwquality 1.4.4-8.el9 -> 1.4.5-3.fc38 libref_array 0.1.5-53.el9 -> 0.1.5-53.fc38 librepo 1.14.5-1.el9 -> 1.15.1-2.fc38 libreport-filesystem 2.15.2-6.el9 -> 2.17.10-1.fc38 libseccomp 2.5.2-2.el9 -> 2.5.3-4.fc38 libselinux 3.5-1.el9 -> 3.5-1.fc38 libselinux-utils 3.5-1.el9 -> 3.5-1.fc38 libsemanage 3.5-1.el9 -> 3.5-2.fc38 libsepol 3.5-1.el9 -> 3.5-1.fc38 libsigsegv 2.13-4.el9 -> 2.14-4.fc38 libslirp 4.4.0-7.el9 -> 4.7.0-3.fc38 libsmartcols 2.37.4-10.el9 -> 2.38.1-4.fc38 libsmbclient 4.17.5-102.el9 -> 2:4.18.3-3.fc38 libsolv 0.7.22-4.el9 -> 0.7.22-4.fc38 libss 1.46.5-3.el9 -> 1.46.5-4.fc38 libsss_certmap 2.8.2-2.el9 -> 2.9.0-1.fc38 libsss_idmap 2.8.2-2.el9 -> 2.9.0-1.fc38 libsss_nss_idmap 2.8.2-2.el9 -> 2.9.0-1.fc38 libsss_sudo 2.8.2-2.el9 -> 2.9.0-1.fc38 libstdc++ 11.3.1-4.3.el9 -> 13.1.1-4.fc38 libtalloc 2.3.4-1.el9 -> 2.4.0-2.fc38 libtasn1 4.16.0-8.el9_1 -> 4.19.0-2.fc38 libtdb 1.4.7-1.el9 -> 1.4.8-1.fc38 libtevent 0.13.0-1.el9 -> 0.14.1-1.fc38 libtirpc 1.3.3-1.el9 -> 1.3.3-1.rc1.fc38 libtool-ltdl 2.4.6-45.el9 -> 2.4.7-6.fc38 libunistring 0.9.10-15.el9 -> 1.1-3.fc38 libuser 0.63-12.el9 -> 0.64-2.fc38 libutempter 1.2.1-6.el9 -> 1.2.1-8.fc38 libuuid 2.37.4-10.el9 -> 2.38.1-4.fc38 libuv 1:1.42.0-1.el9 -> 1:1.44.2-3.fc38 libverto 0.3.2-3.el9 -> 0.3.2-5.fc38 libwbclient 4.17.5-102.el9 -> 2:4.18.3-3.fc38 libxcrypt 4.4.18-3.el9 -> 4.4.35-1.fc38 libxml2 2.9.13-3.el9_1 -> 2.10.4-1.fc38 libxmlb 0.3.10-1.el9 -> 0.3.11-1.fc38 libyaml 0.2.5-7.el9 -> 0.2.5-9.fc38 libzstd 1.5.1-2.el9 -> 1.5.5-1.fc38 linux-firmware 20230310-133.el9_2 -> 20230515-150.fc38 linux-firmware-whence 20230310-133.el9_2 -> 20230515-150.fc38 lmdb-libs 0.9.29-3.el9 -> 0.9.30-1.fc38 logrotate 3.18.0-8.el9 -> 3.21.0-2.fc38 lsof 4.94.0-3.el9 -> 4.96.3-3.fc38 lua-libs 5.4.4-3.el9 -> 5.4.4-9.fc38 luksmeta 9-12.el9 -> 9-15.fc38 lz4-libs 1.9.3-5.el9 -> 1.9.4-2.fc38 lzo 2.10-7.el9 -> 2.10-8.fc38 mokutil 2:0.6.0-4.el9 -> 2:0.6.0-6.fc38 mpfr 4.1.0-7.el9 -> 4.1.1-3.fc38 nano 5.6.1-5.el9 -> 7.2-2.fc38 ncurses 6.2-8.20210508.el9 -> 6.4-3.20230114.fc38 ncurses-base 6.2-8.20210508.el9 -> 6.4-3.20230114.fc38 ncurses-libs 6.2-8.20210508.el9 -> 6.4-3.20230114.fc38 net-tools 2.0-0.62.20160912git.el9 -> 2.0-0.65.20160912git.fc38 nettle 3.8-3.el9_0 -> 3.8-3.fc38 newt 0.52.21-11.el9 -> 0.52.23-2.fc38 nftables 1:1.0.4-10.el9_1 -> 1:1.0.5-2.fc38 npth 1.6-8.el9 -> 1.6-12.fc38 nss-altfiles 2.18.1-20.el9 -> 2.18.1-22.fc38 numactl-libs 2.0.14-9.el9 -> 2.0.16-2.fc38 nvme-cli 2.2.1-2.el9 -> 2.4-2.fc38 oniguruma 6.9.6-1.el9.5 -> 6.9.8-2.D20220919gitb041f6d.fc38.1 openldap 2.6.2-3.el9 -> 2.6.4-1.fc38 openssh 8.7p1-28.el9 -> 9.0p1-15.fc38 openssh-clients 8.7p1-28.el9 -> 9.0p1-15.fc38 openssh-server 8.7p1-28.el9 -> 9.0p1-15.fc38 openssl 1:3.0.7-6.el9_2 -> 1:3.0.9-1.fc38 openssl-libs 1:3.0.7-6.el9_2 -> 1:3.0.9-1.fc38 os-prober 1.77-10.el9 -> 1.81-3.fc38 ostree 2022.6-3.el9 -> 2023.4-1.fc38 ostree-libs 2022.6-3.el9 -> 2023.4-1.fc38 p11-kit 0.24.1-2.el9 -> 0.24.1-6.fc38 p11-kit-trust 0.24.1-2.el9 -> 0.24.1-6.fc38 pam 1.5.1-14.el9 -> 1.5.2-16.fc38 passwd 0.80-12.el9 -> 0.80-14.fc38 pcre2 10.40-2.el9 -> 10.42-1.fc38.1 pcre2-syntax 10.40-2.el9 -> 10.42-1.fc38.1 pigz 2.5-4.el9 -> 2.7-3.fc38 pkgconf 1.7.3-10.el9 -> 1.8.0-6.fc38 pkgconf-m4 1.7.3-10.el9 -> 1.8.0-6.fc38 pkgconf-pkg-config 1.7.3-10.el9 -> 1.8.0-6.fc38 podman 3:4.4.1-3.rhaos4.13.el9 -> 5:4.5.1-1.fc38 policycoreutils 3.5-1.el9 -> 3.5-1.fc38 polkit 0.117-11.el9 -> 122-3.fc38.1 polkit-libs 0.117-11.el9 -> 122-3.fc38.1 polkit-pkla-compat 0.1-21.el9 -> 0.1-23.fc38 popt 1.18-8.el9 -> 1.19-2.fc38 protobuf-c 1.3.3-12.el9 -> 1.4.1-4.fc38 psmisc 23.4-3.el9 -> 23.6-2.fc38 publicsuffix-list-dafsa 20210518-3.el9 -> 20230614-1.fc38 readline 8.1-4.el9 -> 8.2-3.fc38 rpm 4.16.1.3-22.el9 -> 4.18.1-3.fc38 rpm-libs 4.16.1.3-22.el9 -> 4.18.1-3.fc38 rpm-ostree 2022.19-3.el9 -> 2023.4-2.fc38 rpm-ostree-libs 2022.19-3.el9 -> 2023.4-2.fc38 rpm-plugin-selinux 4.16.1.3-22.el9 -> 4.18.1-3.fc38 rsync 3.2.3-19.el9 -> 3.2.7-2.fc38 samba-client-libs 4.17.5-102.el9 -> 2:4.18.3-3.fc38 samba-common 4.17.5-102.el9 -> 2:4.18.3-3.fc38 samba-common-libs 4.17.5-102.el9 -> 2:4.18.3-3.fc38 sed 4.8-9.el9 -> 4.8-12.fc38 selinux-policy 38.1.11-2.el9_2 -> 38.17-1.fc38 selinux-policy-targeted 38.1.11-2.el9_2 -> 38.17-1.fc38 setup 2.13.7-9.el9 -> 2.14.3-2.fc38 shadow-utils 2:4.9-6.el9 -> 2:4.13-6.fc38 shadow-utils-subid 2:4.9-6.el9 -> 2:4.13-6.fc38 shared-mime-info 2.1-5.el9 -> 2.2-3.fc38 shim-x64 15.6-1.el9 -> 15.6-2 slang 2.3.2-11.el9 -> 2.3.3-3.fc38 slirp4netns 1.2.0-3.el9 -> 1.2.0-12.fc38 snappy 1.1.8-8.el9 -> 1.1.9-7.fc38 socat 1.7.4.1-5.el9 -> 1.7.4.4-2.fc38 sqlite-libs 3.34.1-6.el9_1 -> 3.40.1-2.fc38 squashfs-tools 4.4-8.git1.el9 -> 4.5.1-3.fc38 ssh-key-dir 0.1.4-1.el9 -> 0.1.4-3.fc38 sssd-ad 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-client 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-common 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-common-pac 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-ipa 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-krb5 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-krb5-common 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-ldap 2.8.2-2.el9 -> 2.9.0-1.fc38 sssd-nfs-idmap 2.8.2-2.el9 -> 2.9.0-1.fc38 sudo 1.9.5p2-9.el9 -> 1.9.13-2.p2.fc38 systemd 252-14.el9.rhaos4.13 -> 253.4-1.fc38 systemd-libs 252-14.el9.rhaos4.13 -> 253.4-1.fc38 systemd-pam 252-14.el9.rhaos4.13 -> 253.4-1.fc38 systemd-udev 252-14.el9.rhaos4.13 -> 253.4-1.fc38 tar 2:1.34-6.el9_1 -> 2:1.34-8.fc38 tpm2-tools 5.2-2.el9_1 -> 5.5-3.fc38 tpm2-tss 3.0.3-8.el9 -> 4.0.1-3.fc38 tzdata 2022g-2.el9 -> 2023c-1.fc38 userspace-rcu 0.12.1-6.el9 -> 0.13.2-2.fc38 util-linux 2.37.4-10.el9 -> 2.38.1-4.fc38 util-linux-core 2.37.4-10.el9 -> 2.38.1-4.fc38 vim-minimal 2:8.2.2637-20.el9_1 -> 2:9.0.1649-1.fc38 which 2.21-28.el9 -> 2.21-39.fc38 xfsprogs 5.14.2-1.el9 -> 6.1.0-3.fc38 xz 5.2.5-8.el9_0 -> 5.4.1-1.fc38 xz-libs 5.2.5-8.el9_0 -> 5.4.1-1.fc38 zlib 1.2.11-39.el9 -> 1.2.13-3.fc38 zstd 1.5.1-2.el9 -> 1.5.5-1.fc38 Downgraded: aardvark-dns 2:1.5.0-2.el9 -> 1.6.0-1.fc38 cloud-utils-growpart 0.33-1.el9 -> 0.31-12.fc38 conmon 3:2.1.7-1.rhaos4.13.el9 -> 2:2.1.7-2.fc38 container-selinux 3:2.208.0-2.rhaos4.13.el9 -> 2:2.218.0-1.fc38 containernetworking-plugins 1:1.2.0-1.el9 -> 1.1.1-16.fc38 cpio 2.13-16.el9 -> 2.13-14.fc38 device-mapper 9:1.02.187-7.el9 -> 1.02.189-2.fc38 device-mapper-event 9:1.02.187-7.el9 -> 1.02.189-2.fc38 device-mapper-event-libs 9:1.02.187-7.el9 -> 1.02.189-2.fc38 device-mapper-libs 9:1.02.187-7.el9 -> 1.02.189-2.fc38 device-mapper-persistent-data 0.9.0-13.el9 -> 0.9.0-10.fc38 efi-filesystem 6-2.el9_0 -> 5-7.fc38 jansson 2.14-1.el9 -> 2.13.1-6.fc38 keyutils 1.6.3-1.el9 -> 1.6.1-6.fc38 keyutils-libs 1.6.3-1.el9 -> 1.6.1-6.fc38 libcap 2.48-8.el9 -> 2.48-6.fc38 libeconf 0.4.1-2.el9 -> 0.4.0-5.fc38 libnetfilter_conntrack 1.0.9-1.el9 -> 1.0.8-7.fc38 libteam 1.31-16.el9_1 -> 1.31-7.fc38 lvm2 9:2.03.17-7.el9 -> 2.03.18-2.fc38 lvm2-libs 9:2.03.17-7.el9 -> 2.03.18-2.fc38 mdadm 4.2-8.el9 -> 4.2-4.fc38 microcode_ctl 4:20220809-2.el9 -> 2:2.1-55.fc38 netavark 2:1.5.0-2.el9 -> 1.6.0-2.fc38 procps-ng 3.3.17-11.el9 -> 3.3.17-9.fc38 rpcbind 1.2.6-5.el9 -> 1.2.6-4.rc2.fc38 runc 4:1.1.6-3.rhaos4.13.el9 -> 2:1.1.7-1.fc38 sg3_utils 1.47-9.el9 -> 1.46-5.fc38 sg3_utils-libs 1.47-9.el9 -> 1.46-5.fc38 skopeo 2:1.11.2-0.1.el9 -> 1:1.12.0-1.fc38 stalld 1.17.1-1.el9 -> 1.16-3.fc38 teamd 1.31-16.el9_1 -> 1.31-7.fc38 toolbox 0.1.2-1.rhaos4.13.el9 -> 0.0.99.4-1.fc38 yajl 2.1.0-21.el9 -> 2.1.0-20.fc38 Removed: NetworkManager-ovs-1:1.42.2-1.el9.x86_64 audit-3.0.7-103.el9.x86_64 checkpolicy-3.5-1.el9.x86_64 conmon-rs-0.5.1-5.rhaos4.13.git.el9.x86_64 conntrack-tools-1.4.7-2.el9.x86_64 cracklib-dicts-2.9.6-27.el9.x86_64 cri-o-1.26.3-3.rhaos4.13.git641290e.el9.x86_64 cri-tools-1.26.0-1.el9.x86_64 crypto-policies-scripts-20221215-1.git9a18988.el9.noarch dmidecode-1:3.3-7.el9.x86_64 firewalld-filesystem-1.2.1-1.el9.noarch flashrom-1.2-10.el9.x86_64 fwupd-plugin-flashrom-1.8.10-2.el9.x86_64 gawk-all-langpacks-5.1.0-6.el9.x86_64 gettext-0.21-7.el9.x86_64 glusterfs-6.0-57.4.el9.x86_64 glusterfs-client-xlators-6.0-57.4.el9.x86_64 glusterfs-fuse-6.0-57.4.el9.x86_64 glusterfs-libs-6.0-57.4.el9.x86_64 groff-base-1.22.4-10.el9.x86_64 gssproxy-0.8.4-4.el9.x86_64 hwdata-0.348-9.7.el9.noarch info-6.7-15.el9.x86_64 initscripts-service-10.11.5-1.el9.noarch kernel-modules-extra-5.14.0-284.13.1.el9_2.x86_64 libbrotli-1.0.9-6.el9.x86_64 libcurl-7.76.1-23.el9.x86_64 libdrm-2.4.114-1.el9.x86_64 libev-4.33-5.el9.x86_64 libgomp-11.3.1-4.3.el9.x86_64 libmicrohttpd-1:0.9.72-4.el9.x86_64 libmspack-0.10.1-0.7.alpha.el9.x86_64 libnetfilter_cthelper-1.0.0-22.el9.x86_64 libnetfilter_cttimeout-1.0.0-19.el9.x86_64 libnetfilter_queue-1.0.5-1.el9.x86_64 libpciaccess-0.16-6.el9.x86_64 librhsm-0.0.3-7.el9.x86_64 libsmbios-2.4.3-4.el9.x86_64 libssh-0.10.4-8.el9.x86_64 libssh-config-0.10.4-8.el9.noarch libusbx-1.0.26-1.el9.x86_64 libverto-libev-0.3.2-3.el9.x86_64 libxcrypt-compat-4.4.18-3.el9.x86_64 libxslt-1.1.34-9.el9.x86_64 nfs-utils-1:2.5.4-18.el9.x86_64 nmap-ncat-3:7.91-12.el9.x86_64 open-vm-tools-12.1.5-1.el9.x86_64 openldap-compat-2.6.2-3.el9.x86_64 openshift-clients-4.13.0-202303241616.p0.g92b1a3d.assembly.stream.el9.x86_64 openshift-hyperkube-4.13.0-202304211155.p0.gb404935.assembly.stream.el9.x86_64 openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch openvswitch3.1-3.1.0-14.el9fdp.x86_64 ostree-grub2-2022.6-3.el9.x86_64 pciutils-3.7.0-5.el9.x86_64 pciutils-libs-3.7.0-5.el9.x86_64 pcre-8.44-3.el9.3.x86_64 perl-AutoLoader-5.74-480.el9.noarch perl-B-1.80-480.el9.x86_64 perl-Carp-1.50-460.el9.noarch perl-Class-Struct-0.66-480.el9.noarch perl-Data-Dumper-2.174-462.el9.x86_64 perl-Digest-1.19-4.el9.noarch perl-Digest-MD5-2.58-4.el9.x86_64 perl-Encode-4:3.08-462.el9.x86_64 perl-Errno-1.30-480.el9.x86_64 perl-Exporter-5.74-461.el9.noarch perl-Fcntl-1.13-480.el9.x86_64 perl-File-Basename-2.85-480.el9.noarch perl-File-Path-2.18-4.el9.noarch perl-File-Temp-1:0.231.100-4.el9.noarch perl-File-stat-1.09-480.el9.noarch perl-FileHandle-2.03-480.el9.noarch perl-Getopt-Long-1:2.52-4.el9.noarch perl-Getopt-Std-1.12-480.el9.noarch perl-HTTP-Tiny-0.076-460.el9.noarch perl-IO-1.43-480.el9.x86_64 perl-IO-Socket-IP-0.41-5.el9.noarch perl-IO-Socket-SSL-2.073-1.el9.noarch perl-IPC-Open3-1.21-480.el9.noarch perl-MIME-Base64-3.16-4.el9.x86_64 perl-Mozilla-CA-20200520-6.el9.noarch perl-NDBM_File-1.15-480.el9.x86_64 perl-Net-SSLeay-1.92-2.el9.x86_64 perl-POSIX-1.94-480.el9.x86_64 perl-PathTools-3.78-461.el9.x86_64 perl-Pod-Escapes-1:1.07-460.el9.noarch perl-Pod-Perldoc-3.28.01-461.el9.noarch perl-Pod-Simple-1:3.42-4.el9.noarch perl-Pod-Usage-4:2.01-4.el9.noarch perl-Scalar-List-Utils-4:1.56-461.el9.x86_64 perl-SelectSaver-1.02-480.el9.noarch perl-Socket-4:2.031-4.el9.x86_64 perl-Storable-1:3.21-460.el9.x86_64 perl-Symbol-1.08-480.el9.noarch perl-Term-ANSIColor-5.01-461.el9.noarch perl-Term-Cap-1.17-460.el9.noarch perl-Text-ParseWords-3.30-460.el9.noarch perl-Text-Tabs+Wrap-2013.0523-460.el9.noarch perl-Time-Local-2:1.300-7.el9.noarch perl-URI-5.09-3.el9.noarch perl-base-2.27-480.el9.noarch perl-constant-1.33-461.el9.noarch perl-if-0.60.800-480.el9.noarch perl-interpreter-4:5.32.1-480.el9.x86_64 perl-libnet-3.13-4.el9.noarch perl-libs-4:5.32.1-480.el9.x86_64 perl-mro-1.23-480.el9.x86_64 perl-overload-1.31-480.el9.noarch perl-overloading-0.02-480.el9.noarch perl-parent-1:0.238-460.el9.noarch perl-podlators-1:4.14-460.el9.noarch perl-subs-1.03-480.el9.noarch perl-vars-1.05-480.el9.noarch policycoreutils-python-utils-3.5-1.el9.noarch python-unversioned-command-3.9.16-1.el9.noarch python3-3.9.16-1.el9.x86_64 python3-audit-3.0.7-103.el9.x86_64 python3-distro-1.5.0-7.el9.noarch python3-libs-3.9.16-1.el9.x86_64 python3-libselinux-3.5-1.el9.x86_64 python3-libsemanage-3.5-1.el9.x86_64 python3-pip-wheel-21.2.3-6.el9.noarch python3-policycoreutils-3.5-1.el9.noarch python3-pyyaml-5.4.1-6.el9.x86_64 python3-setools-4.4.1-1.el9.x86_64 python3-setuptools-53.0.0-12.el9.noarch python3-setuptools-wheel-53.0.0-12.el9.noarch qemu-guest-agent-17:7.2.0-14.el9_2.x86_64 quota-1:4.06-6.el9.x86_64 quota-nls-1:4.06-6.el9.noarch rdma-core-44.0-2.el9.x86_64 redhat-release-9.2-0.13.el9.x86_64 redhat-release-eula-9.2-0.13.el9.x86_64 sssd-2.8.2-2.el9.x86_64 sssd-proxy-2.8.2-2.el9.x86_64 strace-5.18-2.el9.x86_64 subscription-manager-rhsm-certificates-20220623-1.el9.noarch systemd-journal-remote-252-14.el9.rhaos4.13.x86_64 systemd-rpm-macros-252-14.el9.rhaos4.13.noarch tmux-3.2a-4.el9.x86_64 unbound-libs-1.16.2-3.el9.x86_64 xmlsec1-1.2.29-9.el9.x86_64 xmlsec1-openssl-1.2.29-9.el9.x86_64 Added: amd-gpu-firmware-20230515-150.fc38.noarch atheros-firmware-20230515-150.fc38.noarch brcmfmac-firmware-20230515-150.fc38.noarch btrfs-progs-6.3.2-1.fc38.x86_64 catatonit-0.1.7-14.fc38.x86_64 console-login-helper-messages-motdgen-0.21.3-4.fc38.noarch containerd-1.6.19-1.fc38.x86_64 containers-common-extra-4:1-89.fc38.noarch duktape-2.7.0-2.fc38.x86_64 fedora-gpg-keys-38-1.noarch fedora-release-common-38-36.noarch fedora-release-coreos-38-36.noarch fedora-release-identity-coreos-38-36.noarch fedora-repos-38-1.noarch fedora-repos-archive-38-1.noarch fedora-repos-modular-38-1.noarch fedora-repos-ostree-38-1.noarch flatpak-session-helper-1.15.4-1.fc38.x86_64 fuse-sshfs-3.7.3-3.fc38.x86_64 gettext-envsubst-0.21.1-2.fc38.x86_64 gettext-runtime-0.21.1-2.fc38.x86_64 google-compute-engine-guest-configs-udev-20230526.00-3.fc38.noarch intel-gpu-firmware-20230515-150.fc38.noarch ipcalc-1.0.3-1.fc38.x86_64 iptables-legacy-1.8.9-4.fc38.x86_64 iptables-legacy-libs-1.8.9-4.fc38.x86_64 iptables-services-1.8.9-4.fc38.noarch iptables-utils-1.8.9-4.fc38.x86_64 jemalloc-5.3.0-3.fc38.x86_64 kbd-legacy-2.5.1-5.fc38.noarch libargon2-20190702-2.fc38.x86_64 libbsd-0.11.7-4.fc38.x86_64 libcurl-minimal-8.0.1-2.fc38.x86_64 libmd-1.1.0-1.fc38.x86_64 libnsl2-2.0.0-5.fc38.x86_64 libunistring1.0-1.0-1.fc38.x86_64 libusb1-1.0.26-2.fc38.x86_64 linux-atm-libs-2.5.1-34.fc38.x86_64 moby-engine-20.10.23-1.fc38.x86_64 mt7xxx-firmware-20230515-150.fc38.noarch nano-default-editor-7.2-2.fc38.noarch nfs-utils-coreos-1:2.6.3-0.fc38.x86_64 nmstate-2.2.12-3.fc38.x86_64 nvidia-gpu-firmware-20230515-150.fc38.noarch pam-libs-1.5.2-16.fc38.x86_64 passt-0^20230509.g96f8d55-1.fc38.x86_64 passt-selinux-0^20230509.g96f8d55-1.fc38.noarch podman-plugins-5:4.5.1-1.fc38.x86_64 realtek-firmware-20230515-150.fc38.noarch rpm-sequoia-1.4.0-3.fc38.x86_64 systemd-container-253.4-1.fc38.x86_64 systemd-resolved-253.4-1.fc38.x86_64 vim-data-2:9.0.1649-1.fc38.noarch wireguard-tools-1.0.20210914-4.fc38.x86_64 xxhash-libs-0.8.1-4.fc38.x86_64 zchunk-libs-1.3.1-1.fc38.x86_64 zincati-0.0.25-4.fc38.x86_64 zram-generator-1.1.2-4.fc38.x86_64 Changes queued for next boot. Run "systemctl reboot" to start a reboot [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745051837 Available: 745051837 Inodes: Total: 300647616 Free: 300579053 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21682 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f Image has 0 derived layers warning: Found corrupted merge commit inode clashes: 884 unknown: 0 ok: 3472 warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable error: Found potential corruption, dry-run mode enabled Pre-verification continued from comment 23: [root@cosa-devsh ~]# rpm -qa ostree ostree-2022.6-3.el9.x86_64 [root@cosa-devsh ~]# [root@cosa-devsh ~]# [root@cosa-devsh ~]# [root@cosa-devsh ~]# rpm-ostree usroverlay Development mode enabled. A writable overlayfs is now mounted on /usr. All changes there will be discarded on reboot. [root@cosa-devsh ~]# curl -L -k --fail --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/ostree/2023.1/4.el9_2/x86_64/ostree-{,libs-,grub2-}2023.1-4.el9_2.x86_64.rpm % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 242k 100 242k 0 0 488k 0 --:--:-- --:--:-- --:--:-- 488k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 441k 100 441k 0 0 1128k 0 --:--:-- --:--:-- --:--:-- 1125k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10057 100 10057 0 0 185k 0 --:--:-- --:--:-- --:--:-- 185k [root@cosa-devsh ~]# rpm -Uvh ostree-* Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ostree-libs-2023.1-4.el9_2 ################################# [ 17%] 2:ostree-2023.1-4.el9_2 ################################# [ 33%] 3:ostree-grub2-2023.1-4.el9_2 ################################# [ 50%] Cleaning up / removing... 4:ostree-grub2-2022.6-3.el9 ################################# [ 67%] 5:ostree-2022.6-3.el9 ################################# [ 83%] 6:ostree-libs-2022.6-3.el9 ################################# [100%] /usr/lib/tmpfiles.d/tmp.conf:12: Duplicate line for path "/var/tmp", ignoring. /usr/lib/tmpfiles.d/var.conf:14: Duplicate line for path "/var/log", ignoring. /usr/lib/tmpfiles.d/var.conf:19: Duplicate line for path "/var/cache", ignoring. /usr/lib/tmpfiles.d/var.conf:21: Duplicate line for path "/var/lib", ignoring. /usr/lib/tmpfiles.d/var.conf:23: Duplicate line for path "/var/spool", ignoring. "/home" already exists and is not a directory. "/srv" already exists and is not a directory. "/root" already exists and is not a directory. [root@cosa-devsh ~]# systemctl restart rpm-ostreed [root@cosa-devsh ~]# rpm -qa ostree ostree-2023.1-4.el9_2.x86_64 [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745046135 Available: 745046135 Inodes: Total: 300647616 Free: 300574187 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21779 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f Image has 0 derived layers warning: Found corrupted merge commit inode clashes: 884 unknown: 0 ok: 3472 warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable error: Found potential corruption, dry-run mode enabled [root@cosa-devsh ~]# rpm-ostree status State: idle Deployments: ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132 Version: 38.20230625.3.0 (2023-07-21T15:15:37Z) Diff: 334 upgraded, 34 downgraded, 143 removed, 58 added ● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360 Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z) Unlocked: development [root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo error: Pruning docker://quay.io/fedora/fedora-coreos:stable: unlink(ostree/container/image/docker_3A__2F__2F_quay_2E_io/fedora/fedora-coreos_3A_stable): Read-only file system [root@cosa-devsh ~]# mount -o remount,rw /sysroot [root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo Removed images: 1 layers: 0 [root@cosa-devsh ~]# rpm-ostree status State: idle Deployments: ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132 Version: 38.20230625.3.0 (2023-07-21T15:15:37Z) Diff: 334 upgraded, 34 downgraded, 143 removed, 58 added ● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360 Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z) Unlocked: development [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745046136 Available: 745046136 Inodes: Total: 300647616 Free: 300574188 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21779 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 0 ostree-container images OK no corrupted images found [root@cosa-devsh ~]# [root@cosa-devsh ~]# [root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel (digest: sha256:da542dbe516f4cbf6498f9a64b4702e1d11885b800675265b55ab36028fb3d8e) ''' ''' Changes queued for next boot. Run "systemctl reboot" to start a reboot [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745046052 Available: 745046052 Inodes: Total: 300647616 Free: 300578976 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 22673 objects with > 32 bit inode numbers and 17806 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_726728f89dea4a2f5aafe923b606d9bc9783a63475a698dbc9edc525e5dd9700 Image has 0 derived layers OK image docker://quay.io/fedora/fedora-coreos:testing-devel (verified=11 OK no corrupted images found OK, our initial analysis code has landed in CentOS Stream 9 builds. This is *checking only* - it won't attempt to hotfix anything. We're just trying to gather data. This will also aid us in figuring out how many and exactly which files may have been swapped which is going to be useful for analysis. We also want to check that we're actually hitting the bug we think we are! I'm quite confident in this, but it's *possible* there's something else going wrong. This will help us know, without changing anything right now - be sure to use the --dry-run flag! Here's instructions to test it out: First...we need a container image that has rpm-ostree; and I looked around and it's actually most convenient to use the RHEL CoreOS 9 image *as a* container (a new power since we landed https://docs.openshift.com/container-platform/4.13/post_installation_configuration/coreos-layering.html#coreos-layering ) Per that doc, we need to find the reference to the image: ``` $ oc adm release info --image-for=rhel-coreos quay.io/openshift-release-dev/ocp-release:4.13.3-x86_64 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6973e9353f29e678cad79fe768c22cfd6697d8aa30d2aeaa78cceea989925ded ``` This is for x86_64. We're going to spawn a privileged pod on a node using that base image: ``` $ oc debug --image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6973e9353f29e678cad79fe768c22cfd6697d8aa30d2aeaa78cceea989925ded node/ci-ln-6fkcxpt-72292-m58pv-worker-a-9gq6x ``` Replace the node name with the node we want to analyze. Inside the pod shell, we're going to pull the updated rpm-ostree and ostree from CentOS Stream 9: ``` $ rpm -Uvh https://kojihub.stream.centos.org/kojifiles/packages/rpm-ostree/2023.5/2.el9/x86_64/rpm-ostree-{,libs-}2023.5-2.el9.x86_64.rpm https://kojihub.stream.centos.org/kojifiles/packages/ostree/2023.4/2.el9/x86_64/ostree-{,libs-,grub2-}2023.4-2.el9.x86_64.rpm ``` (Ignore the errors about /bin/sh, they're an unrelated benign selinux bug that is harmless) Finally, we can run the analysis tooling in "check only" mode: ``` $ ostree provisional-repair repair --sysroot /host/sysroot --dry-run Running in container, assuming we can remount /host/sysroot writable Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... ostree inode check: 64bit inodes: 0 32 bit inodes: 28922 collisions: 0 OK no colliding inodes found $ ``` Test with latest 4.13 and update ostree/rpm-ostree, let me know if there is something that I missed. $ oc adm release info --image-for=rhel-coreos quay.io/openshift-release-dev/ocp-release:4.13.6-x86_64 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8436f6414523d81cc41a3dab8a31c54f464e146928675180f304a478993dab26 $ oc debug --image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8436f6414523d81cc41a3dab8a31c54f464e146928675180f304a478993dab26 node/ip-10-0-213-53.us-east-2.compute.internal Temporary namespace openshift-debug-h7nhz is created for debugging node... Starting pod/ip-10-0-213-53us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.213.53 If you don't see a command prompt, try pressing enter. sh-5.1# rpm -Uvh https://kojihub.stream.centos.org/kojifiles/packages/rpm-ostree/2023.5/2.el9/x86_64/rpm-ostree-{,libs-}2023.5-2.el9.x86_64.rpm https://kojihu.stream.centos.org/kojifiles/packages/ostree/2023.4/2.el9/x86_64/ostree-{,libs-,grub2-}2023.4-2.el9.x86_64.rpm ... sh-5.1# ostree provisional-repair repair --sysroot /host/sysroot --dry-run Running in container, assuming we can remount /host/sysroot writable Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... ostree inode check: 64bit inodes: 0 32 bit inodes: 26201 collisions: 0 OK no colliding inodes found OK no corruption found At a procedural level, this bug is for RHEL 9.3. Which was actually fixed by https://gitlab.com/redhat/centos-stream/rpms/ostree/-/merge_requests/20 a month ago; we've long had the core bug fixed for 9.3. There's nothing more to do logically for this bug, but let's keep it open until we've at least got queued errata for 9.2 and 8.6. After that I'll likely close, and the remaining zstream tracking bugs become the canonical place to track shipping. I would like to continue using this bug as a centralized information hub. I spent some time looking into exactly what XFS setups are at risk. First, https://access.redhat.com/solutions/2884651 and https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf are useful resources. It looks like provisioning RHCOS 4.13 or below to a ~1TB disk will result in a setup like: agcount=1109, agsize=242112 will only have 31 bit inodes, and is hence safe. If we provision XFS *correctly* (i.e. with a low AG count) which requires a workaround Butane/Ignition (linked above), then even going to 2T is safely at 32 bits. 3T and above though will require 33 bits which rounds up to 64. Based on analysis so far, I am comfortable saying that it's safe to use 1TB. For OCP 4.12, the fixed ostree is part of https://amd64.ocp.releases.ci.openshift.org/releasestream/4.12.0-0.nightly/release/4.12.0-0.nightly-2023-07-27-023653 For OCP 4.13, the fixed ostree is part of https://amd64.ocp.releases.ci.openshift.org/releasestream/4.13.0-0.nightly/release/4.13.0-0.nightly-2023-07-26-041700 At this point again, at at technical level we already have the fix queued in current C9S and RHEL 9.3. The clones of this bug will track the zstream releases. |