OK, I'm working on tooling to better detect this situation (and after that, repair it).

But basically, this will generally only occur with large disks (which for XFS, result in multiple allocation groups which hand out distinct inode ranges as I understand it).  That's how it's sailed through all of our CI so far.  (Also, we don't actually verify the deployment roots today, which is an ostree design flaw that makes this problem much less visible; that design flaw will be fixed with the ongoing composefs work)

The original reporter gave this handy bit:

> The issue was discovered on PoC machine equipped with a large NVME (3.4TB), but the bug can be easily reproduced using cosa run -m 4000 --qemu-size +3TB', followed by installation of any package using rpm-ostree install`. The 

Using e.g. ls -Rli /ostree/repo/objects - any case where > 32 bit inode numbers are at risk.

Initial analysis code: https://github.com/ostreedev/ostree-rs-ext/pull/504

This will also then morph into "repair" code soon.

So the problem isn't directly disk size - the problem is triggered by > 32 bit inodes (an inode is a unique file identifier).  So it's more about the allocation pattern for file identifiers, as distinct from the amount of storage those files occupy.

The default RHEL (and RHCOS) filesystem is XFS.  It happens to be today (at least up to RHEL9) that XFS will generally only use >32 bit inodes when a sufficient number of allocation groups are used.  And allocation groups are a function of disk size.  In short: disk size => allocation groups => inodes.  We support other filesystems too in RHEL, such as ext4.  Its behavior for inode allocations are different; I haven't yet gathered enough information to say how likely ext4 is to trigger this.

In general a pattern we recommend for large disks is to provision a separate /var filesystem: there's an example of this here: https://docs.openshift.com/container-platform/4.13/installing/install_config/installing-customizing.html#installation-special-config-raid_installing-customizing

This pattern will work around this bug.

Regarding the WA proposed I understand it is in installation phase but how can you apply the change on day 2 to move the /var to another filesystem?

Is it necessary to reinstall the node, and in any case, if all the nodes in the cluster have a similar configuration, is it necessary to reinstall them all before an upgrade?

We don't support re-partitioning the disks on day 2 via the MCO (https://github.com/openshift/machine-config-operator/blob/master/docs/FAQ.md#q-can-i-use-the-mco-to-re-partition-or-re-install) so this would be a manual process for existing nodes or re-provisioning nodes.

As far as I understand, this only impacts installations with a large / (/sysroot) partition, as this is where the ostree content is stored, so installations with a large disk and no separated /var partitions.

This is likely to impact all filesystems but is exacerbated on XFS by the way that we grow the filesystem: https://github.com/coreos/fedora-coreos-tracker/issues/1183

My rough thoughts here are that we will:

- Build and test a fixed version of ostree, and it will be available to apply as a hotfix.  This should happen today.
- Ship this updated version across all supported OCP and RHEL versions.

However, having customers manually apply this hotfix won't scale very well.  As has happened multiple times in the past now, because the machine-config-daemon daemonset is upgraded *before* nodes upgrade, we can automate hotfixes that way.  That would look like:

- Inject the new ostree code into the MCD binary in openshift/machine-config-operator
- Prepare code in the MCD which applies this hotfix before initiating an OS upgrade:
- Flush all merged container images (ensuring we re-fetch them from the network and regenerate the merged filesystem tree with the fixed version of ostree)

There's a bit more code being worked on in https://github.com/ostreedev/ostree-rs-ext/pull/504 that will be helpful, but I think not required.

I'm still trying to better characterize how to reproduce this.  I am *very* quickly reproducing it with current Fedora CoreOS stable (which ships ostree-2023.1-2.fc38.x86_64 which does *not* have the fix).

Just a quick:
```
$ cosa run --qemu-size '+3T
# rpm-ostree rebase ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:stable
```

Shows that the target filesystem tree before rebooting is just heavily corrupted (your results will vary, but spot checking with the new deployment tree in /sysroot/ostree/deploy...) on various binaries will show this.  I am working to publish a container image with my much more strict analysis and repair tooling.


However...turning to RHEL: I think this issue is also heavily intersecting with https://github.com/coreos/fedora-coreos-config/pull/2320 because with that fix, we end up with this:

```
$ xfs_info /|grep agcount=
meta-data=/dev/vda4              isize=512    agcount=20, agsize=268435455 blks
```

Notice the agcount of 20.  I think this ends up making the inode space much more sparse (each AG has a much larger range of inodes).

However with current RHCOS 4.12, in a "stock configuration" where we don't reprovision the rootfs, we end up with e.g.:

```
[root@cosa-devsh ~]# xfs_info /
meta-data=/dev/vda4              isize=512    agcount=3326, agsize=242112 blks
```

Which is is much, much larger AG count (and one the XFS maintainers think is an important bug, that's why we added the reprovisioning on firstboot).

But, this AG count seems to have the side effect of making the inodes *much* less likely to collide by default.  And this is another reason I think it took us so long to find this.  More analysis incoming.

Hi Colin,

Kindly let me know if the below process is right with FCOS for verification. 

aaradhak@fedora ~/fcos2 $ cosa run --qemu-image ./fedora-coreos-38.20230514.3.0-qemu.x86_64.qcow2
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/fcos2:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa quay.io/coreos-assembler/coreos-assembler:latest run --qemu-image ./fedora-coreos-38.20230514.3.0-qemu.x86_64.qcow2
Fedora CoreOS 38.20230514.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Thu Jul 20 18:20:44 2023
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ rpm-ostree status
State: busy
AutomaticUpdatesDriver: Zincati
  DriverState: active; found update on remote: 38.20230625.3.0
Transaction: deploy --lock-finalization --skip-branch-check revision=e841d77aadb875bb801ac845a0d9b8a70b4224bdeb15e7d6c5bff1da932c0301 --disallow-downgrade 
  Initiator: caller :1.31
Deployments:
● fedora:fedora/x86_64/coreos/stable
                  Version: 38.20230514.3.0 (2023-05-31T10:22:06Z)
                   Commit: a8d8c301ec3e4267106208ed194603e2b1979a67458fb8d3542c54b05b5b1766
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ alias verify-ostree="sudo podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run"
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ verify-ostree 
Trying to pull quay.io/cgwalters/ostree-ext-dev:latest...
Getting image source signatures
Copying blob dbc9fb14b970 done  
Copying blob ad5077952f52 [=================>--------------------] 32.3MiB / 70.1MiB
Copying blob 19d2bba2169e [=================================>----] 20.5MiB / 22.6MiB

Broadcast message from Zincati at Thu 2023-07-20 18:21:54 UTC:
New update 38.20230625.3.0 is available and has been deployed.
If permitted by the update strategy, Zincati will reboot into this update when
Copying blob dbc9fb14b970 done  
Copying blob ad5077952f52 done  
Copying blob 19d2bba2169e done  
Copying config 991ff7104a done  
Writing manifest to image destination
Storing signatures
Running in container, assuming we can remount /rootfs/sysroot writable
OK no derived commits found.
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
error: Updates and deployments are driven by Zincati (zincati.service)
See Zincati's documentation at https://github.com/coreos/zincati
Use --bypass-driver to bypass Zincati and perform the operation anyways
[core@cosa-devsh ~]$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable --bypass-driver
Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable
Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable (digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132)
ostree chunk layers stored: 0 needed: 51 (727.7 MB)
Fetching ostree chunk sha256:2d72193934f5 (49.1 MB)
Fetched ostree chunk sha256:2d72193934f5
Fetching ostree chunk sha256:1d1851157929 (39.1 MB)
Fetched ostree chunk sha256:1d1851157929
Fetching ostree chunk sha256:38fb7f3103e6 (86.3 MB)
Fetched ostree chunk sha256:38fb7f3103e6
Fetching ostree chunk sha256:ea9e24ce3684 (56.7 MB)
Fetched ostree chunk sha256:ea9e24ce3684
Fetching ostree chunk sha256:92fc970dd1db (22.1 MB)
Fetched ostree chunk sha256:92fc970dd1db
Fetching ostree chunk sha256:c0d3e4ab2dfb (18.7 MB)
Fetched ostree chunk sha256:c0d3e4ab2dfb
Fetching ostree chunk sha256:4e3a86b78101 (14.7 MB)
Fetched ostree chunk sha256:4e3a86b78101
Fetching ostree chunk sha256:55ed8482122a (30.9 MB)
Fetched ostree chunk sha256:55ed8482122a
Fetching ostree chunk sha256:e15d15e60129 (10.2 MB)
Fetched ostree chunk sha256:e15d15e60129
Fetching ostree chunk sha256:80409ccf031c (8.1 MB)
Fetched ostree chunk sha256:80409ccf031c
Fetching ostree chunk sha256:77bf2ccbd010 (15.9 MB)
Fetched ostree chunk sha256:77bf2ccbd010
Fetching ostree chunk sha256:c4596e0f9d2f (22.8 MB)
Fetched ostree chunk sha256:c4596e0f9d2f
Fetching ostree chunk sha256:66c3d1e01b20 (8.6 MB)
Fetched ostree chunk sha256:66c3d1e01b20
Fetching ostree chunk sha256:dbed5df4d655 (7.2 MB)
Fetched ostree chunk sha256:dbed5df4d655
Fetching ostree chunk sha256:8cb97e7cbe0b (18.4 MB)
Fetched ostree chunk sha256:8cb97e7cbe0b
Fetching ostree chunk sha256:9a111e34f0a7 (5.7 MB)
Fetched ostree chunk sha256:9a111e34f0a7
Fetching ostree chunk sha256:b2852d7115ed (4.5 MB)
Fetched ostree chunk sha256:b2852d7115ed
Fetching ostree chunk sha256:32e3e4336e95 (4.5 MB)
Fetched ostree chunk sha256:32e3e4336e95
Fetching ostree chunk sha256:6e7d355821b7 (3.9 MB)
Fetched ostree chunk sha256:6e7d355821b7
Fetching ostree chunk sha256:720bcbf5c3f5 (4.2 MB)
Fetched ostree chunk sha256:720bcbf5c3f5
Fetching ostree chunk sha256:306acdde4360 (2.5 MB)
Fetched ostree chunk sha256:306acdde4360
Fetching ostree chunk sha256:770cb05ccb97 (3.8 MB)
Fetched ostree chunk sha256:770cb05ccb97
Fetching ostree chunk sha256:d83d9388b8c8 (3.4 MB)
Fetched ostree chunk sha256:d83d9388b8c8
Fetching ostree chunk sha256:2fd8145ed1e8 (984.6 kB)
Fetched ostree chunk sha256:2fd8145ed1e8
Fetching ostree chunk sha256:4556015edf3e (10.0 MB)
Fetched ostree chunk sha256:4556015edf3e
Fetching ostree chunk sha256:faf33bf71717 (3.1 MB)
Fetched ostree chunk sha256:faf33bf71717
Fetching ostree chunk sha256:87e02b8ee016 (4.0 MB)
Fetched ostree chunk sha256:87e02b8ee016
Fetching ostree chunk sha256:d153f54f6c23 (1.6 MB)
Fetched ostree chunk sha256:d153f54f6c23
Fetching ostree chunk sha256:6ae4e4f5687d (2.7 MB)
Fetched ostree chunk sha256:6ae4e4f5687d
Fetching ostree chunk sha256:1f3ba420fe2a (3.1 MB)
Fetched ostree chunk sha256:1f3ba420fe2a
Fetching ostree chunk sha256:12e618cc4270 (918.5 kB)
Fetched ostree chunk sha256:12e618cc4270
Fetching ostree chunk sha256:85274c02297e (8.0 MB)
Fetched ostree chunk sha256:85274c02297e
Fetching ostree chunk sha256:6078667503e5 (34.6 MB)
Fetched ostree chunk sha256:6078667503e5
Fetching ostree chunk sha256:c6a22bae3945 (2.2 MB)
Fetched ostree chunk sha256:c6a22bae3945
Fetching ostree chunk sha256:6b67185e644b (3.1 MB)
Fetched ostree chunk sha256:6b67185e644b
Fetching ostree chunk sha256:1a038abae32f (2.5 MB)
Fetched ostree chunk sha256:1a038abae32f
Fetching ostree chunk sha256:4e59715e2ad8 (1.7 MB)
Fetched ostree chunk sha256:4e59715e2ad8
Fetching ostree chunk sha256:bee65ef58f1c (11.3 MB)
Fetched ostree chunk sha256:bee65ef58f1c
Fetching ostree chunk sha256:4bae9fc1a5a8 (31.3 MB)
Fetched ostree chunk sha256:4bae9fc1a5a8
Fetching ostree chunk sha256:2581e41cf6dd (3.7 MB)
Fetched ostree chunk sha256:2581e41cf6dd
Fetching ostree chunk sha256:bdaa253e11c0 (1.2 MB)
Fetched ostree chunk sha256:bdaa253e11c0
Fetching ostree chunk sha256:f32c284711ca (1.2 MB)
Fetched ostree chunk sha256:f32c284711ca
Fetching ostree chunk sha256:8f773e26787d (10.1 MB)
Fetched ostree chunk sha256:8f773e26787d
Fetching ostree chunk sha256:50dc647e78c9 (5.1 MB)
Fetched ostree chunk sha256:50dc647e78c9
Fetching ostree chunk sha256:b10485f2b393 (4.7 MB)
Fetched ostree chunk sha256:b10485f2b393
Fetching ostree chunk sha256:22c06f2cd431 (4.1 MB)
Fetched ostree chunk sha256:22c06f2cd431
Fetching ostree chunk sha256:9727f6669e28 (2.3 MB)
Fetched ostree chunk sha256:9727f6669e28
Fetching ostree chunk sha256:fa253d5e6b8d (22.1 MB)
Fetched ostree chunk sha256:fa253d5e6b8d
Fetching ostree chunk sha256:4b2fc5e89c68 (109.3 MB)
Fetched ostree chunk sha256:4b2fc5e89c68
Fetching ostree chunk sha256:5f9d3dcf5281 (2.3 kB)
Fetched ostree chunk sha256:5f9d3dcf5281
Fetching ostree chunk sha256:074b4ac01d1e (1.4 MB)
Fetched ostree chunk sha256:074b4ac01d1e
Staging deployment... done
Upgraded:
  afterburn 5.4.0-2.fc38 -> 5.4.2-1.fc38
  afterburn-dracut 5.4.0-2.fc38 -> 5.4.2-1.fc38
  amd-gpu-firmware 20230404-149.fc38 -> 20230515-150.fc38
  atheros-firmware 20230404-149.fc38 -> 20230515-150.fc38
  bind-libs 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38
  bind-license 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38
  bind-utils 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38
  brcmfmac-firmware 20230404-149.fc38 -> 20230515-150.fc38
  btrfs-progs 6.2.2-1.fc38 -> 6.3.2-1.fc38
  c-ares 1.19.0-1.fc38 -> 1.19.1-1.fc38
  container-selinux 2:2.211.1-1.fc38 -> 2:2.218.0-1.fc38
  crun 1.8.4-1.fc38 -> 1.8.5-1.fc38
  curl 8.0.1-1.fc38 -> 8.0.1-2.fc38
  dbus 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38
  dbus-common 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38
  dbus-libs 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38
  dnsmasq 2.89-2.fc38 -> 2.89-5.fc38
  ethtool 2:6.2-1.fc38 -> 2:6.3-1.fc38
  fedora-release-common 38-35 -> 38-36
  fedora-release-coreos 38-35 -> 38-36
  fedora-release-identity-coreos 38-35 -> 38-36
  fuse-overlayfs 1.10-3.fc38 -> 1.12-1.fc38
  fwupd 1.8.15-1.fc38 -> 1.9.2-1.fc38
  git-core 2.40.1-1.fc38 -> 2.41.0-1.fc38
  glib2 2.76.2-1.fc38 -> 2.76.3-1.fc38
  intel-gpu-firmware 20230404-149.fc38 -> 20230515-150.fc38
  iptables-legacy 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-legacy-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-nft 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-services 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-utils 1.8.9-2.fc38 -> 1.8.9-4.fc38
  irqbalance 2:1.9.1-2.fc38 -> 2:1.9.2-1.fc38
  kernel 6.2.15-300.fc38 -> 6.3.8-200.fc38
  kernel-core 6.2.15-300.fc38 -> 6.3.8-200.fc38
  kernel-modules 6.2.15-300.fc38 -> 6.3.8-200.fc38
  kernel-modules-core 6.2.15-300.fc38 -> 6.3.8-200.fc38
  libcurl-minimal 8.0.1-1.fc38 -> 8.0.1-2.fc38
  libgcc 13.1.1-1.fc38 -> 13.1.1-4.fc38
  libipa_hbac 2.8.2-4.fc38 -> 2.9.0-1.fc38
  libmd 1.0.4-3.fc38 -> 1.1.0-1.fc38
  libmodulemd 2.14.0-5.fc38 -> 2.15.0-2.fc38
  libsmbclient 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38
  libsss_certmap 2.8.2-4.fc38 -> 2.9.0-1.fc38
  libsss_idmap 2.8.2-4.fc38 -> 2.9.0-1.fc38
  libsss_nss_idmap 2.8.2-4.fc38 -> 2.9.0-1.fc38
  libsss_sudo 2.8.2-4.fc38 -> 2.9.0-1.fc38
  libstdc++ 13.1.1-1.fc38 -> 13.1.1-4.fc38
  libwbclient 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38
  libxcrypt 4.4.33-7.fc38 -> 4.4.35-1.fc38
  linux-firmware 20230404-149.fc38 -> 20230515-150.fc38
  linux-firmware-whence 20230404-149.fc38 -> 20230515-150.fc38
  microcode_ctl 2:2.1-54.fc38 -> 2:2.1-55.fc38
  mt7xxx-firmware 20230404-149.fc38 -> 20230515-150.fc38
  nmstate 2.2.10-4.fc38 -> 2.2.12-3.fc38
  nvidia-gpu-firmware 20230404-149.fc38 -> 20230515-150.fc38
  openssl 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38
  openssl-libs 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38
  ostree 2023.1-2.fc38 -> 2023.4-1.fc38
  ostree-libs 2023.1-2.fc38 -> 2023.4-1.fc38
  podman 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38
  podman-plugins 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38
  publicsuffix-list-dafsa 20230318-1.fc38 -> 20230614-1.fc38
  realtek-firmware 20230404-149.fc38 -> 20230515-150.fc38
  rpm-ostree 2023.3-1.fc38 -> 2023.4-2.fc38
  rpm-ostree-libs 2023.3-1.fc38 -> 2023.4-2.fc38
  rpm-sequoia 1.4.0-2.fc38 -> 1.4.0-3.fc38
  samba-client-libs 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38
  samba-common 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38
  samba-common-libs 2:4.18.2-0.fc38 -> 2:4.18.3-3.fc38
  selinux-policy 38.12-1.fc38 -> 38.17-1.fc38
  selinux-policy-targeted 38.12-1.fc38 -> 38.17-1.fc38
  sssd-ad 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-client 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-common 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-common-pac 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-ipa 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-krb5 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-krb5-common 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-ldap 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sssd-nfs-idmap 2.8.2-4.fc38 -> 2.9.0-1.fc38
  sudo 1.9.13-1.p2.fc38 -> 1.9.13-2.p2.fc38
  vim-data 2:9.0.1486-1.fc38 -> 2:9.0.1649-1.fc38
  vim-minimal 2:9.0.1486-1.fc38 -> 2:9.0.1649-1.fc38
Added:
  google-compute-engine-guest-configs-udev-20230526.00-3.fc38.noarch
  ipcalc-1.0.3-1.fc38.x86_64
  passt-0^20230509.g96f8d55-1.fc38.x86_64
  passt-selinux-0^20230509.g96f8d55-1.fc38.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 2473723    Free: 1897947    Available: 1897947
Inodes: Total: 4980160    Free: 4921274
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 0 objects with > 32 bit inode numbers and 30322 objects with <= 32 bit inode numbers
OK no colliding inodes found
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; update staged: 38.20230625.3.0; reboot delayed due to active user sessions
Deployments:
  ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
                   Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132
                  Version: 38.20230625.3.0 (2023-07-20T18:24:21Z)
                     Diff: 84 upgraded, 4 added

● fedora:fedora/x86_64/coreos/stable
                  Version: 38.20230514.3.0 (2023-05-31T10:22:06Z)
                   Commit: a8d8c301ec3e4267106208ed194603e2b1979a67458fb8d3542c54b05b5b1766
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ rpm -qa ostree
ostree-2023.1-2.fc38.x86_64
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel --bypass-driver
Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel
Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel (digest: sha256:9b066cddd8ccdd7d5d00d46a2769afc0c594f7e91beb28687552c0b6bba14d3f)
ostree chunk layers stored: 16 needed: 35 (553.8 MB)
Fetching ostree chunk sha256:ca3ad6937343 (92.2 MB)
Fetched ostree chunk sha256:ca3ad6937343
Fetching ostree chunk sha256:8f060c236f10 (56.8 MB)
Fetched ostree chunk sha256:8f060c236f10
Fetching ostree chunk sha256:ab61108bdccc (31.0 MB)
Fetched ostree chunk sha256:ab61108bdccc
Fetching ostree chunk sha256:29d43d7532fc (8.6 MB)
Fetched ostree chunk sha256:29d43d7532fc
Fetching ostree chunk sha256:9914a26e2503 (10.1 MB)
Fetched ostree chunk sha256:9914a26e2503
Fetching ostree chunk sha256:1304bedcfaa3 (15.9 MB)
Fetched ostree chunk sha256:1304bedcfaa3
Fetching ostree chunk sha256:28f61efc79e6 (22.8 MB)
Fetched ostree chunk sha256:28f61efc79e6
Fetching ostree chunk sha256:98ad05d12c8a (7.2 MB)
Fetched ostree chunk sha256:98ad05d12c8a
Fetching ostree chunk sha256:10b550fd6e8b (17.6 MB)
Fetched ostree chunk sha256:10b550fd6e8b
Fetching ostree chunk sha256:b2b8ed02a657 (5.7 MB)
Fetched ostree chunk sha256:b2b8ed02a657
Fetching ostree chunk sha256:2d4192f895b5 (4.5 MB)
Fetched ostree chunk sha256:2d4192f895b5
Fetching ostree chunk sha256:4c4f0056c9e8 (10.0 MB)
Fetched ostree chunk sha256:4c4f0056c9e8
Fetching ostree chunk sha256:645abb5ac7e4 (3.1 MB)
Fetched ostree chunk sha256:645abb5ac7e4
Fetching ostree chunk sha256:e4adf6da1658 (4.0 MB)
Fetched ostree chunk sha256:e4adf6da1658
Fetching ostree chunk sha256:7f69978960d1 (2.7 MB)
Fetched ostree chunk sha256:7f69978960d1
Fetching ostree chunk sha256:1917ba532d95 (917.7 kB)
Fetched ostree chunk sha256:1917ba532d95
Fetching ostree chunk sha256:5ed208adc48c (8.0 MB)
Fetched ostree chunk sha256:5ed208adc48c
Fetching ostree chunk sha256:733a4b9db00a (34.6 MB)
Fetched ostree chunk sha256:733a4b9db00a
Fetching ostree chunk sha256:112c9f768dd3 (2.3 MB)
Fetched ostree chunk sha256:112c9f768dd3
Fetching ostree chunk sha256:a0a20e2d01cd (2.6 MB)
Fetched ostree chunk sha256:a0a20e2d01cd
Fetching ostree chunk sha256:21e7be9e5b0a (3.1 MB)
Fetched ostree chunk sha256:21e7be9e5b0a
Fetching ostree chunk sha256:e96687909700 (1.2 MB)
Fetched ostree chunk sha256:e96687909700
Fetching ostree chunk sha256:b07c2c7c2526 (11.3 MB)
Fetched ostree chunk sha256:b07c2c7c2526
Fetching ostree chunk sha256:0e1c1f114196 (31.4 MB)
Fetched ostree chunk sha256:0e1c1f114196
Fetching ostree chunk sha256:762c2cd46bf4 (4.0 MB)
Fetched ostree chunk sha256:762c2cd46bf4
Fetching ostree chunk sha256:7dd9eaad5ccb (1.5 MB)
Fetched ostree chunk sha256:7dd9eaad5ccb
Fetching ostree chunk sha256:4d187e9b978a (660.6 kB)
Fetched ostree chunk sha256:4d187e9b978a
Fetching ostree chunk sha256:18efcb95f6ba (12.1 MB)
Fetched ostree chunk sha256:18efcb95f6ba
Fetching ostree chunk sha256:d0a7b2958ebf (5.3 MB)
Fetched ostree chunk sha256:d0a7b2958ebf
Fetching ostree chunk sha256:b34fecc0d98a (4.5 MB)
Fetched ostree chunk sha256:b34fecc0d98a
Fetching ostree chunk sha256:1d71a8f5c106 (4.2 MB)
Fetched ostree chunk sha256:1d71a8f5c106
Fetching ostree chunk sha256:e353c6b5901b (2.5 MB)
Fetched ostree chunk sha256:e353c6b5901b
Fetching ostree chunk sha256:02aa7defe84f (20.4 MB)
Fetched ostree chunk sha256:02aa7defe84f
Fetching ostree chunk sha256:368f77051bee (109.6 MB)
Fetched ostree chunk sha256:368f77051bee
Fetching ostree chunk sha256:98dd1141e87d (1.4 MB)
Fetched ostree chunk sha256:98dd1141e87d
Staging deployment... done
Upgraded:
  NetworkManager 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
  NetworkManager-cloud-setup 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
  NetworkManager-libnm 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
  NetworkManager-team 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
  NetworkManager-tui 1:1.42.6-1.fc38 -> 1:1.42.8-1.fc38
  aardvark-dns 1.6.0-1.fc38 -> 1.7.0-1.fc38
  afterburn 5.4.0-2.fc38 -> 5.4.2-1.fc38
  afterburn-dracut 5.4.0-2.fc38 -> 5.4.2-1.fc38
  amd-gpu-firmware 20230404-149.fc38 -> 20230625-151.fc38
  atheros-firmware 20230404-149.fc38 -> 20230625-151.fc38
  bind-libs 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38
  bind-license 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38
  bind-utils 32:9.18.14-1.fc38 -> 32:9.18.16-1.fc38
  brcmfmac-firmware 20230404-149.fc38 -> 20230625-151.fc38
  btrfs-progs 6.2.2-1.fc38 -> 6.3.2-1.fc38
  c-ares 1.19.0-1.fc38 -> 1.19.1-1.fc38
  container-selinux 2:2.211.1-1.fc38 -> 2:2.219.0-1.fc38
  cracklib 2.9.7-31.fc38 -> 2.9.11-1.fc38
  crun 1.8.4-1.fc38 -> 1.8.5-1.fc38
  curl 8.0.1-1.fc38 -> 8.0.1-2.fc38
  dbus 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38
  dbus-common 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38
  dbus-libs 1:1.14.6-1.fc38 -> 1:1.14.8-1.fc38
  diffutils 3.9-1.fc38 -> 3.10-1.fc38
  dnsmasq 2.89-2.fc38 -> 2.89-5.fc38
  elfutils-default-yama-scope 0.189-2.fc38 -> 0.189-3.fc38
  elfutils-libelf 0.189-2.fc38 -> 0.189-3.fc38
  elfutils-libs 0.189-2.fc38 -> 0.189-3.fc38
  ethtool 2:6.2-1.fc38 -> 2:6.4-1.fc38
  fedora-release-common 38-35 -> 38-36
  fedora-release-coreos 38-35 -> 38-36
  fedora-release-identity-coreos 38-35 -> 38-36
  fuse-overlayfs 1.10-3.fc38 -> 1.12-1.fc38
  fwupd 1.8.15-1.fc38 -> 1.9.2-1.fc38
  git-core 2.40.1-1.fc38 -> 2.41.0-1.fc38
  glib2 2.76.2-1.fc38 -> 2.76.4-1.fc38
  ignition 2.15.0-3.fc38 -> 2.16.2-1.fc38
  inih 56-3.fc38 -> 57-1.fc38
  intel-gpu-firmware 20230404-149.fc38 -> 20230625-151.fc38
  iptables-legacy 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-legacy-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-libs 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-nft 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-services 1.8.9-2.fc38 -> 1.8.9-4.fc38
  iptables-utils 1.8.9-2.fc38 -> 1.8.9-4.fc38
  irqbalance 2:1.9.1-2.fc38 -> 2:1.9.2-1.fc38
  kernel 6.2.15-300.fc38 -> 6.3.12-200.fc38
  kernel-core 6.2.15-300.fc38 -> 6.3.12-200.fc38
  kernel-modules 6.2.15-300.fc38 -> 6.3.12-200.fc38
  kernel-modules-core 6.2.15-300.fc38 -> 6.3.12-200.fc38
  krb5-libs 1.20.1-8.fc38 -> 1.21-2.fc38
  libassuan 2.5.5-6.fc38 -> 2.5.6-1.fc38
  libcurl-minimal 8.0.1-1.fc38 -> 8.0.1-2.fc38
  libgcc 13.1.1-1.fc38 -> 13.1.1-4.fc38
  libipa_hbac 2.8.2-4.fc38 -> 2.9.1-1.fc38
  libksba 1.6.3-2.fc38 -> 1.6.4-1.fc38
  libmd 1.0.4-3.fc38 -> 1.1.0-1.fc38
  libmodulemd 2.14.0-5.fc38 -> 2.15.0-2.fc38
  libnetfilter_conntrack 1.0.8-7.fc38 -> 1.0.9-1.fc38
  libreport-filesystem 2.17.10-1.fc38 -> 2.17.11-1.fc38
  libsmbclient 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38
  libsss_certmap 2.8.2-4.fc38 -> 2.9.1-1.fc38
  libsss_idmap 2.8.2-4.fc38 -> 2.9.1-1.fc38
  libsss_nss_idmap 2.8.2-4.fc38 -> 2.9.1-1.fc38
  libsss_sudo 2.8.2-4.fc38 -> 2.9.1-1.fc38
  libstdc++ 13.1.1-1.fc38 -> 13.1.1-4.fc38
  libuv 1:1.44.2-3.fc38 -> 1:1.46.0-1.fc38
  libwbclient 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38
  libxcrypt 4.4.33-7.fc38 -> 4.4.36-1.fc38
  linux-firmware 20230404-149.fc38 -> 20230625-151.fc38
  linux-firmware-whence 20230404-149.fc38 -> 20230625-151.fc38
  lmdb-libs 0.9.30-1.fc38 -> 0.9.31-1.fc38
  microcode_ctl 2:2.1-54.fc38 -> 2:2.1-55.fc38
  mt7xxx-firmware 20230404-149.fc38 -> 20230625-151.fc38
  netavark 1.6.0-2.fc38 -> 1.7.0-1.fc38
  nmstate 2.2.10-4.fc38 -> 2.2.12-3.fc38
  nvidia-gpu-firmware 20230404-149.fc38 -> 20230625-151.fc38
  openssl 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38
  openssl-libs 1:3.0.8-2.fc38 -> 1:3.0.9-1.fc38
  ostree 2023.1-2.fc38 -> 2023.4-2.fc38
  ostree-libs 2023.1-2.fc38 -> 2023.4-2.fc38
  podman 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38
  podman-plugins 5:4.5.0-1.fc38 -> 5:4.5.1-1.fc38
  publicsuffix-list-dafsa 20230318-1.fc38 -> 20230614-1.fc38
  realtek-firmware 20230404-149.fc38 -> 20230625-151.fc38
  rpm-ostree 2023.3-1.fc38 -> 2023.5-1.fc38
  rpm-ostree-libs 2023.3-1.fc38 -> 2023.5-1.fc38
  rpm-sequoia 1.4.0-2.fc38 -> 1.4.1-1.fc38
  samba-client-libs 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38
  samba-common 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38
  samba-common-libs 2:4.18.2-0.fc38 -> 2:4.18.4-0.fc38
  selinux-policy 38.12-1.fc38 -> 38.21-1.fc38
  selinux-policy-targeted 38.12-1.fc38 -> 38.21-1.fc38
  skopeo 1:1.12.0-1.fc38 -> 1:1.13.0-1.fc38
  sssd-ad 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-client 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-common 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-common-pac 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-ipa 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-krb5 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-krb5-common 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-ldap 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sssd-nfs-idmap 2.8.2-4.fc38 -> 2.9.1-1.fc38
  sudo 1.9.13-1.p2.fc38 -> 1.9.13-2.p2.fc38
  vim-data 2:9.0.1486-1.fc38 -> 2:9.0.1671-1.fc38
  vim-minimal 2:9.0.1486-1.fc38 -> 2:9.0.1671-1.fc38
  yajl 2.1.0-20.fc38 -> 2.1.0-21.fc38
Added:
  google-compute-engine-guest-configs-udev-20230526.00-3.fc38.noarch
  ipcalc-1.0.3-1.fc38.x86_64
  passt-0^20230625.g32660ce-1.fc38.x86_64
  passt-selinux-0^20230625.g32660ce-1.fc38.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[core@cosa-devsh ~]$ 
Broadcast message from Zincati at Thu 2023-07-20 18:30:55 UTC:
New update 38.20230625.3.0 is available and has been deployed.
If permitted by the update strategy, Zincati will reboot into this update when
all interactive users have logged out, or in 1 minute, whichever comes
earlier. Please log out of all active sessions in order to let the auto-update
process continue.

[core@cosa-devsh ~]$ rpm -qa ostree
ostree-2023.1-2.fc38.x86_64
[core@cosa-devsh ~]$ verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 2473723    Free: 1790941    Available: 1790941
Inodes: Total: 4980160    Free: 4914850
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 0 objects with > 32 bit inode numbers and 36720 objects with <= 32 bit inode numbers
OK no colliding inodes found

Just for reference:

CentOS Stream 9: already has ostree-2023.4-2.el9 which contains the fix for this bug (this update was already shipped earlier).
CentOS Stream 8: MR in https://gitlab.com/redhat/centos-stream/rpms/ostree/-/merge_requests/22

Just going to write for the record here that I consider this bug very embarrassing - the integrity and reliability of operating system upgrades is really The Job that we're trying to do here.  It's my top priority to fix this.

Further, I can say that we already have some in-flight work upstream in "composefs" which will provide significantly stronger mechanisms for integrity, but also crucially will enable us to share a fully common storage model between the OS updates and containers, which will ensure both work the same way.

But, OK so an immediate thing we're trying to work through here is reproducing this reliably in RHCOS 4.13 (RHEL 9.2) and RHCOS 4.12 (RHEL 8.6).  And the reason for this seems to again have to do with how many XFS allocation groups are in use in the disk images we generate there.  XFS seems to tend to prefer using lower AG numbers for allocations, which in turn imply most inodes are going to be <= 32 bit which avoids collision.  Again, this is one of the compounding factors that let this bug get through our CI so far.

I tried to quickly test with ext4, but what I'm seeing there is that its inode allocation is quite serial; with XFS a simple script to just create and unlink files will *sometimes* get > 32 bit inodes (because they may be allocated in a later AG), but as far as I can tell never with ext4 - unless we actually tried creating billions of files.  I think this architectural difference between XFS and ext4 is why XFS can scales better: because of how it internally "subdivides" everything into AGs, whereas ext4 has more filesystem-global metadata.

It's likely we could likely force this situation in XFS to be more likely by actually filling up the disk space, but at terabyte sizes that's going to get expensive to do in CI and testing.  There may be a way to try to steer things to "later" AGs, but:

[time passes]

OK, I figured out how to craft a Butane (Ignition) configuration that will force Ignition to regenerate the XFS filesystem at the full new size:

variant: fcos
version: 1.4.0
storage:
  disks:
  - device: /dev/disk/by-id/coreos-boot-disk
    wipe_table: false
    partitions:
    - number: 4
      label: root
      # Allocate 2.8T for the root
      size_mib: 2936012.8
      resize: true
    - size_mib: 0
      label: foo
  filesystems:
    - device: /dev/disk/by-partlabel/root
      wipe_filesystem: true
      format: xfs
      label: root
    - path: /var/mnt/foo
      device: /dev/disk/by-partlabel/foo
      # We can select the filesystem we'd like.
      format: xfs
      # Ask Butane to generate a mount unit for us so that this filesystem
      # gets mounted in the real root.
      with_mount_unit: true

I've verified this Butane config works with 4.12 and 4.13, and crucially we now see:

```
$ xfs_info / |grep agcount=
meta-data=/dev/vda4              isize=512    agcount=4, agsize=187904768 blks
```

An AG count of just 4 (which is actually better for XFS) means each AG has a much, much greater inode allocation range.

And in this situation, running the reproducer, the analysis tooling now very reliably finds corruption; notice we have almost 50% of inodes that are > 32 bit:

```
Gathering inodes for ostree objects...
Analyzed 23467 objects with > 32 bit inode numbers and 25411 objects with <= 32 bit inode numbers
warning: 443 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
  inode clashes: 19
  unknown:       0
  ok:            572

warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled
```

Hi Colin,
I am able to reproduce the collision with the butane config.

aaradhak@fedora ~/rhcos/rhcos-4.13 $ cosa run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size '+3T' --qemu-memory 8192 -B reprovision-xfs.bu
COREOS_ASSEMBLER_CONTAINER=quay.io/coreos-assembler/coreos-assembler:rhcos-4.13
COREOS_ASSEMBLER_ADD_CERTS=y
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/rhcos/rhcos-4.13:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa -v=/etc/pki/ca-trust:/etc/pki/ca-trust:ro quay.io/coreos-assembler/coreos-assembler:rhcos-4.13 run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size +3T --qemu-memory 8192 -B reprovision-xfs.bu
Red Hat Enterprise Linux CoreOS 413.92.202305021736-0
  Part of OpenShift 4.13, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.13/architecture/architecture-rhcos.html

---
Last login: Fri Jul 21 15:11:47 2023
[core@cosa-devsh ~]$ xfs_info / |grep agcount=
meta-data=/dev/vda4              isize=512    agcount=4, agsize=187904768 blks
[core@cosa-devsh ~]$ sudo -i
[root@cosa-devsh ~]# alias verify-ostree="podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run"
[root@cosa-devsh ~]# verify-ostree 
Trying to pull quay.io/cgwalters/ostree-ext-dev:latest...
Getting image source signatures
Copying blob ad5077952f52 done  
Copying blob dbc9fb14b970 done  
Copying blob 19d2bba2169e done  
Copying config 991ff7104a done  
Writing manifest to image destination
Storing signatures
Running in container, assuming we can remount /rootfs/sysroot writable
OK no derived commits found.
[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable
Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable (digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132)
ostree chunk layers stored: 0 needed: 51 (727.7 MB)
Fetching ostree chunk sha256:2d72193934f5 (49.1 MB)
Fetched ostree chunk sha256:2d72193934f5
Fetching ostree chunk sha256:1d1851157929 (39.1 MB)
Fetched ostree chunk sha256:1d1851157929
Fetching ostree chunk sha256:38fb7f3103e6 (86.3 MB)
Fetched ostree chunk sha256:38fb7f3103e6
Fetching ostree chunk sha256:ea9e24ce3684 (56.7 MB)
Fetched ostree chunk sha256:ea9e24ce3684
Fetching ostree chunk sha256:92fc970dd1db (22.1 MB)
Fetched ostree chunk sha256:92fc970dd1db
Fetching ostree chunk sha256:c0d3e4ab2dfb (18.7 MB)
Fetched ostree chunk sha256:c0d3e4ab2dfb
Fetching ostree chunk sha256:4e3a86b78101 (14.7 MB)
Fetched ostree chunk sha256:4e3a86b78101
Fetching ostree chunk sha256:55ed8482122a (30.9 MB)
Fetched ostree chunk sha256:55ed8482122a
Fetching ostree chunk sha256:e15d15e60129 (10.2 MB)
Fetched ostree chunk sha256:e15d15e60129
Fetching ostree chunk sha256:80409ccf031c (8.1 MB)
Fetched ostree chunk sha256:80409ccf031c
Fetching ostree chunk sha256:77bf2ccbd010 (15.9 MB)
Fetched ostree chunk sha256:77bf2ccbd010
Fetching ostree chunk sha256:c4596e0f9d2f (22.8 MB)
Fetched ostree chunk sha256:c4596e0f9d2f
Fetching ostree chunk sha256:66c3d1e01b20 (8.6 MB)
Fetched ostree chunk sha256:66c3d1e01b20
Fetching ostree chunk sha256:dbed5df4d655 (7.2 MB)
Fetched ostree chunk sha256:dbed5df4d655
Fetching ostree chunk sha256:8cb97e7cbe0b (18.4 MB)
Fetched ostree chunk sha256:8cb97e7cbe0b
Fetching ostree chunk sha256:9a111e34f0a7 (5.7 MB)
Fetched ostree chunk sha256:9a111e34f0a7
Fetching ostree chunk sha256:b2852d7115ed (4.5 MB)
Fetched ostree chunk sha256:b2852d7115ed
Fetching ostree chunk sha256:32e3e4336e95 (4.5 MB)
Fetched ostree chunk sha256:32e3e4336e95
Fetching ostree chunk sha256:6e7d355821b7 (3.9 MB)
Fetched ostree chunk sha256:6e7d355821b7
Fetching ostree chunk sha256:720bcbf5c3f5 (4.2 MB)
Fetched ostree chunk sha256:720bcbf5c3f5
Fetching ostree chunk sha256:306acdde4360 (2.5 MB)
Fetched ostree chunk sha256:306acdde4360
Fetching ostree chunk sha256:770cb05ccb97 (3.8 MB)
Fetched ostree chunk sha256:770cb05ccb97
Fetching ostree chunk sha256:d83d9388b8c8 (3.4 MB)
Fetched ostree chunk sha256:d83d9388b8c8
Fetching ostree chunk sha256:2fd8145ed1e8 (984.6 kB)
Fetched ostree chunk sha256:2fd8145ed1e8
Fetching ostree chunk sha256:4556015edf3e (10.0 MB)
Fetched ostree chunk sha256:4556015edf3e
Fetching ostree chunk sha256:faf33bf71717 (3.1 MB)
Fetched ostree chunk sha256:faf33bf71717
Fetching ostree chunk sha256:87e02b8ee016 (4.0 MB)
Fetched ostree chunk sha256:87e02b8ee016
Fetching ostree chunk sha256:d153f54f6c23 (1.6 MB)
Fetched ostree chunk sha256:d153f54f6c23
Fetching ostree chunk sha256:6ae4e4f5687d (2.7 MB)
Fetched ostree chunk sha256:6ae4e4f5687d
Fetching ostree chunk sha256:1f3ba420fe2a (3.1 MB)
Fetched ostree chunk sha256:1f3ba420fe2a
Fetching ostree chunk sha256:12e618cc4270 (918.5 kB)
Fetched ostree chunk sha256:12e618cc4270
Fetching ostree chunk sha256:85274c02297e (8.0 MB)
Fetched ostree chunk sha256:85274c02297e
Fetching ostree chunk sha256:6078667503e5 (34.6 MB)
Fetched ostree chunk sha256:6078667503e5
Fetching ostree chunk sha256:c6a22bae3945 (2.2 MB)
Fetched ostree chunk sha256:c6a22bae3945
Fetching ostree chunk sha256:6b67185e644b (3.1 MB)
Fetched ostree chunk sha256:6b67185e644b
Fetching ostree chunk sha256:1a038abae32f (2.5 MB)
Fetched ostree chunk sha256:1a038abae32f
Fetching ostree chunk sha256:4e59715e2ad8 (1.7 MB)
Fetched ostree chunk sha256:4e59715e2ad8
Fetching ostree chunk sha256:bee65ef58f1c (11.3 MB)
Fetched ostree chunk sha256:bee65ef58f1c
Fetching ostree chunk sha256:4bae9fc1a5a8 (31.3 MB)
Fetched ostree chunk sha256:4bae9fc1a5a8
Fetching ostree chunk sha256:2581e41cf6dd (3.7 MB)
Fetched ostree chunk sha256:2581e41cf6dd
Fetching ostree chunk sha256:bdaa253e11c0 (1.2 MB)
Fetched ostree chunk sha256:bdaa253e11c0
Fetching ostree chunk sha256:f32c284711ca (1.2 MB)
Fetched ostree chunk sha256:f32c284711ca
Fetching ostree chunk sha256:8f773e26787d (10.1 MB)
Fetched ostree chunk sha256:8f773e26787d
Fetching ostree chunk sha256:50dc647e78c9 (5.1 MB)
Fetched ostree chunk sha256:50dc647e78c9
Fetching ostree chunk sha256:b10485f2b393 (4.7 MB)
Fetched ostree chunk sha256:b10485f2b393
Fetching ostree chunk sha256:22c06f2cd431 (4.1 MB)
Fetched ostree chunk sha256:22c06f2cd431
Fetching ostree chunk sha256:9727f6669e28 (2.3 MB)
Fetched ostree chunk sha256:9727f6669e28
Fetching ostree chunk sha256:fa253d5e6b8d (22.1 MB)
Fetched ostree chunk sha256:fa253d5e6b8d
Fetching ostree chunk sha256:4b2fc5e89c68 (109.3 MB)
Fetched ostree chunk sha256:4b2fc5e89c68
Fetching ostree chunk sha256:5f9d3dcf5281 (2.3 kB)
Fetched ostree chunk sha256:5f9d3dcf5281
Fetching ostree chunk sha256:074b4ac01d1e (1.4 MB)
Fetched ostree chunk sha256:074b4ac01d1e
Staging deployment... done
Upgraded:
  NetworkManager 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38
  NetworkManager-cloud-setup 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38
  NetworkManager-libnm 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38
  NetworkManager-team 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38
  NetworkManager-tui 1:1.42.2-1.el9 -> 1:1.42.6-1.fc38
  WALinuxAgent-udev 2.7.0.6-9.el9 -> 2.9.0.4-1.fc38
  acl 2.3.1-3.el9 -> 2.3.1-6.fc38
  adcli 0.9.2-1.el9 -> 0.9.2-2.fc38
  afterburn 5.4.1-1.el9 -> 5.4.2-1.fc38
  afterburn-dracut 5.4.1-1.el9 -> 5.4.2-1.fc38
  alternatives 1.20-2.el9 -> 1.24-1.fc38
  attr 2.5.1-3.el9 -> 2.5.1-6.fc38
  audit-libs 3.0.7-103.el9 -> 3.1.1-1.fc38
  authselect 1.2.6-1.el9 -> 1.4.2-2.fc38
  authselect-libs 1.2.6-1.el9 -> 1.4.2-2.fc38
  avahi-libs 0.8-12.el9 -> 0.8-22.fc38
  basesystem 11-13.el9 -> 11-15.fc38
  bash 5.1.8-6.el9_1 -> 5.2.15-3.fc38
  bash-completion 1:2.11-4.el9 -> 1:2.11-9.fc38
  bind-libs 32:9.16.23-11.el9 -> 32:9.18.16-1.fc38
  bind-license 32:9.16.23-11.el9 -> 32:9.18.16-1.fc38
  bind-utils 32:9.16.23-11.el9 -> 32:9.18.16-1.fc38
  bootupd 0.2.7-2.el9 -> 0.2.9-1.fc38
  bsdtar 3.5.3-4.el9 -> 3.6.1-4.fc38
  bubblewrap 0.4.1-6.el9 -> 0.7.0-1.fc38
  bzip2 1.0.8-8.el9 -> 1.0.8-13.fc38
  bzip2-libs 1.0.8-8.el9 -> 1.0.8-13.fc38
  c-ares 1.17.1-5.el9 -> 1.19.1-1.fc38
  ca-certificates 2022.2.54-90.2.el9_0 -> 2023.2.60-2.fc38
  chrony 4.3-1.el9 -> 4.3-3.fc38
  cifs-utils 7.0-1.el9 -> 7.0-1.fc38
  clevis 18-110.el9 -> 19-2.fc38
  clevis-dracut 18-110.el9 -> 19-2.fc38
  clevis-luks 18-110.el9 -> 19-2.fc38
  clevis-systemd 18-110.el9 -> 19-2.fc38
  console-login-helper-messages 0.21.3-1.el9 -> 0.21.3-4.fc38
  console-login-helper-messages-issuegen 0.21.3-1.el9 -> 0.21.3-4.fc38
  console-login-helper-messages-profile 0.21.3-1.el9 -> 0.21.3-4.fc38
  containers-common 3:1-34.rhaos4.13.el9 -> 4:1-89.fc38
  coreos-installer 0.17.0-1.rhaos4.13.el9 -> 0.17.0-3.fc38
  coreos-installer-bootinfra 0.17.0-1.rhaos4.13.el9 -> 0.17.0-3.fc38
  coreutils 8.32-34.el9 -> 9.1-12.fc38
  coreutils-common 8.32-34.el9 -> 9.1-12.fc38
  cracklib 2.9.6-27.el9 -> 2.9.7-31.fc38
  criu 3.17-4.el9 -> 3.17.1-5.fc38
  criu-libs 3.17-4.el9 -> 3.17.1-5.fc38
  crun 1.8.4-1.rhaos4.13.el9 -> 1.8.5-1.fc38
  crypto-policies 20221215-1.git9a18988.el9 -> 20230301-1.gita12f7b2.fc38
  cryptsetup 2.6.0-2.el9 -> 2.6.1-1.fc38
  cryptsetup-libs 2.6.0-2.el9 -> 2.6.1-1.fc38
  curl 7.76.1-23.el9 -> 8.0.1-2.fc38
  cyrus-sasl-gssapi 2.1.27-21.el9 -> 2.1.28-9.fc38
  cyrus-sasl-lib 2.1.27-21.el9 -> 2.1.28-9.fc38
  dbus 1:1.12.20-7.el9_1 -> 1:1.14.8-1.fc38
  dbus-broker 28-7.el9 -> 33-1.fc38
  dbus-common 1:1.12.20-7.el9_1 -> 1:1.14.8-1.fc38
  dbus-libs 1:1.12.20-7.el9_1 -> 1:1.14.8-1.fc38
  device-mapper-multipath 0.8.7-20.el9 -> 0.9.4-2.fc38
  device-mapper-multipath-libs 0.8.7-20.el9 -> 0.9.4-2.fc38
  diffutils 3.7-12.el9 -> 3.9-1.fc38
  dnsmasq 2.85-6.el9 -> 2.89-5.fc38
  dosfstools 4.2-3.el9 -> 4.2-6.fc38
  dracut 057-21.git20230214.el9 -> 059-3.fc38
  dracut-network 057-21.git20230214.el9 -> 059-3.fc38
  dracut-squash 057-21.git20230214.el9 -> 059-3.fc38
  e2fsprogs 1.46.5-3.el9 -> 1.46.5-4.fc38
  e2fsprogs-libs 1.46.5-3.el9 -> 1.46.5-4.fc38
  efibootmgr 16-12.el9 -> 18-3.fc38
  efivar-libs 38-3.el9 -> 38-7.fc38
  elfutils-default-yama-scope 0.188-3.el9 -> 0.189-2.fc38
  elfutils-libelf 0.188-3.el9 -> 0.189-2.fc38
  elfutils-libs 0.188-3.el9 -> 0.189-2.fc38
  ethtool 2:5.16-1.el9 -> 2:6.3-1.fc38
  expat 2.5.0-1.el9 -> 2.5.0-2.fc38
  file 5.39-12.el9 -> 5.44-3.fc38
  file-libs 5.39-12.el9 -> 5.44-3.fc38
  filesystem 3.16-2.el9 -> 3.18-3.fc38
  findutils 1:4.8.0-5.el9 -> 1:4.9.0-3.fc38
  fstrm 0.6.1-3.el9 -> 0.6.1-6.fc38
  fuse 2.9.9-15.el9 -> 2.9.9-16.fc38
  fuse-common 3.10.2-5.el9 -> 3.14.1-1.fc38
  fuse-libs 2.9.9-15.el9 -> 2.9.9-16.fc38
  fuse-overlayfs 1.10-2.el9 -> 1.12-1.fc38
  fuse3 3.10.2-5.el9 -> 3.14.1-1.fc38
  fuse3-libs 3.10.2-5.el9 -> 3.14.1-1.fc38
  fwupd 1.8.10-2.el9 -> 1.9.2-1.fc38
  gawk 5.1.0-6.el9 -> 5.1.1-5.fc38
  gdbm-libs 1:1.19-4.el9 -> 1:1.23-3.fc38
  gdisk 1.0.7-5.el9 -> 1.0.9-5.fc38
  gettext-libs 0.21-7.el9 -> 0.21.1-2.fc38
  git-core 2.39.1-1.el9 -> 2.41.0-1.fc38
  glib2 2.68.4-6.el9 -> 2.76.3-1.fc38
  glibc 2.34-60.el9 -> 2.37-4.fc38
  glibc-common 2.34-60.el9 -> 2.37-4.fc38
  glibc-gconv-extra 2.34-60.el9 -> 2.37-4.fc38
  glibc-minimal-langpack 2.34-60.el9 -> 2.37-4.fc38
  gmp 1:6.2.0-10.el9 -> 1:6.2.1-4.fc38
  gnupg2 2.3.3-2.el9_0 -> 2.4.0-3.fc38
  gnutls 3.7.6-20.el9_2 -> 3.8.0-2.fc38
  gpgme 1.15.1-6.el9 -> 1.17.1-3.fc38
  grep 3.6-5.el9 -> 3.8-3.fc38
  grub2-common 1:2.06-61.el9 -> 1:2.06-95.fc38
  grub2-efi-x64 1:2.06-61.el9 -> 1:2.06-95.fc38
  grub2-pc 1:2.06-61.el9 -> 1:2.06-95.fc38
  grub2-pc-modules 1:2.06-61.el9 -> 1:2.06-95.fc38
  grub2-tools 1:2.06-61.el9 -> 1:2.06-95.fc38
  grub2-tools-minimal 1:2.06-61.el9 -> 1:2.06-95.fc38
  gzip 1.12-1.el9 -> 1.12-3.fc38
  hostname 3.23-6.el9 -> 3.23-8.fc38
  ignition 2.15.0-1.el9 -> 2.15.0-3.fc38
  inih 49-6.el9 -> 56-3.fc38
  iproute 6.1.0-1.el9 -> 6.1.0-1.fc38
  iproute-tc 6.1.0-1.el9 -> 6.1.0-1.fc38
  iptables-libs 1.8.8-6.el9_1 -> 1.8.9-4.fc38
  iptables-nft 1.8.8-6.el9_1 -> 1.8.9-4.fc38
  iputils 20210202-8.el9 -> 20221126-2.fc38
  irqbalance 2:1.9.0-3.el9 -> 2:1.9.2-1.fc38
  iscsi-initiator-utils 6.2.1.4-3.git2a8f9d8.el9 -> 6.2.1.4-10.git2a8f9d8.fc38
  iscsi-initiator-utils-iscsiuio 6.2.1.4-3.git2a8f9d8.el9 -> 6.2.1.4-10.git2a8f9d8.fc38
  isns-utils-libs 0.101-4.el9 -> 0.101-6.fc38
  jose 11-3.el9 -> 11-7.fc38
  jq 1.6-14.el9 -> 1.6-15.fc38
  json-c 0.14-11.el9 -> 0.16-4.fc38
  json-glib 1.6.6-1.el9 -> 1.6.6-4.fc38
  kbd 2.4.0-8.el9 -> 2.5.1-5.fc38
  kbd-misc 2.4.0-8.el9 -> 2.5.1-5.fc38
  kernel 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38
  kernel-core 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38
  kernel-modules 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38
  kernel-modules-core 5.14.0-284.13.1.el9_2 -> 6.3.8-200.fc38
  kexec-tools 2.0.25-13.el9_2 -> 2.0.26-3.fc38
  kmod 28-7.el9 -> 30-4.fc38
  kmod-libs 28-7.el9 -> 30-4.fc38
  kpartx 0.8.7-20.el9 -> 0.9.4-2.fc38
  krb5-libs 1.20.1-8.el9 -> 1.20.1-8.fc38
  less 590-1.el9_0 -> 633-1.fc38
  libacl 2.3.1-3.el9 -> 2.3.1-6.fc38
  libaio 0.3.111-13.el9 -> 0.3.111-15.fc38
  libarchive 3.5.3-4.el9 -> 3.6.1-4.fc38
  libassuan 2.5.5-3.el9 -> 2.5.5-6.fc38
  libattr 2.5.1-3.el9 -> 2.5.1-6.fc38
  libbasicobjects 0.1.1-53.el9 -> 0.1.1-53.fc38
  libblkid 2.37.4-10.el9 -> 2.38.1-4.fc38
  libbpf 2:1.0.0-2.el9 -> 2:1.1.0-2.fc38
  libcap-ng 0.8.2-7.el9 -> 0.8.3-5.fc38
  libcbor 0.7.0-5.el9 -> 0.7.0-9.fc38
  libcollection 0.7.0-53.el9 -> 0.7.0-53.fc38
  libcom_err 1.46.5-3.el9 -> 1.46.5-4.fc38
  libdaemon 0.14-23.el9 -> 0.14-25.fc38
  libdb 5.3.28-53.el9 -> 5.3.28-55.fc38
  libdhash 0.5.0-53.el9 -> 0.5.0-53.fc38
  libedit 3.1-37.20210216cvs.el9 -> 3.1-45.20221030cvs.fc38
  libevent 2.1.12-6.el9 -> 2.1.12-8.fc38
  libfdisk 2.37.4-10.el9 -> 2.38.1-4.fc38
  libffi 3.4.2-7.el9 -> 3.4.4-2.fc38
  libfido2 1.6.0-7.el9 -> 1.12.0-3.fc38
  libgcab1 1.4-6.el9 -> 1.5-2.fc38
  libgcc 11.3.1-4.3.el9 -> 13.1.1-4.fc38
  libgcrypt 1.10.0-10.el9_1 -> 1.10.2-1.fc38
  libgpg-error 1.42-5.el9 -> 1.47-1.fc38
  libgudev 237-1.el9 -> 237-4.fc38
  libgusb 0.3.8-1.el9 -> 0.4.5-1.fc38
  libibverbs 44.0-2.el9 -> 44.0-3.fc38
  libicu 67.1-9.el9 -> 72.1-2.fc38
  libidn2 2.3.0-7.el9 -> 2.3.4-2.fc38
  libini_config 1.3.1-53.el9 -> 1.3.1-53.fc38
  libipa_hbac 2.8.2-2.el9 -> 2.9.0-1.fc38
  libjcat 0.1.6-3.el9 -> 0.1.13-1.fc38
  libjose 11-3.el9 -> 11-7.fc38
  libkcapi 1.3.1-3.el9 -> 1.4.0-5.fc38
  libkcapi-hmaccalc 1.3.1-3.el9 -> 1.4.0-5.fc38
  libksba 1.5.1-6.el9_1 -> 1.6.3-2.fc38
  libldb 2.6.1-1.el9 -> 2.7.2-1.fc38
  libluksmeta 9-12.el9 -> 9-15.fc38
  libmaxminddb 1.5.2-3.el9 -> 1.7.1-2.fc38
  libmnl 1.0.4-15.el9 -> 1.0.5-2.fc38
  libmodulemd 2.13.0-2.el9 -> 2.15.0-2.fc38
  libmount 2.37.4-10.el9 -> 2.38.1-4.fc38
  libndp 1.8-4.el9 -> 1.8-5.fc38
  libnet 1.2-6.el9 -> 1.2-7.fc38
  libnfnetlink 1.0.1-21.el9 -> 1.0.1-23.fc38
  libnfsidmap 1:2.5.4-18.el9 -> 1:2.6.3-0.fc38
  libnftnl 1.2.2-1.el9 -> 1.2.4-2.fc38
  libnghttp2 1.43.0-5.el9 -> 1.52.0-1.fc38
  libnl3 3.7.0-1.el9 -> 3.7.0-3.fc38
  libnl3-cli 3.7.0-1.el9 -> 3.7.0-3.fc38
  libnvme 1.2-2.el9 -> 1.4-2.fc38
  libpath_utils 0.2.1-53.el9 -> 0.2.1-53.fc38
  libpcap 14:1.10.0-4.el9 -> 14:1.10.4-1.fc38
  libpkgconf 1.7.3-10.el9 -> 1.8.0-6.fc38
  libpsl 0.21.1-5.el9 -> 0.21.2-2.fc38
  libpwquality 1.4.4-8.el9 -> 1.4.5-3.fc38
  libref_array 0.1.5-53.el9 -> 0.1.5-53.fc38
  librepo 1.14.5-1.el9 -> 1.15.1-2.fc38
  libreport-filesystem 2.15.2-6.el9 -> 2.17.10-1.fc38
  libseccomp 2.5.2-2.el9 -> 2.5.3-4.fc38
  libselinux 3.5-1.el9 -> 3.5-1.fc38
  libselinux-utils 3.5-1.el9 -> 3.5-1.fc38
  libsemanage 3.5-1.el9 -> 3.5-2.fc38
  libsepol 3.5-1.el9 -> 3.5-1.fc38
  libsigsegv 2.13-4.el9 -> 2.14-4.fc38
  libslirp 4.4.0-7.el9 -> 4.7.0-3.fc38
  libsmartcols 2.37.4-10.el9 -> 2.38.1-4.fc38
  libsmbclient 4.17.5-102.el9 -> 2:4.18.3-3.fc38
  libsolv 0.7.22-4.el9 -> 0.7.22-4.fc38
  libss 1.46.5-3.el9 -> 1.46.5-4.fc38
  libsss_certmap 2.8.2-2.el9 -> 2.9.0-1.fc38
  libsss_idmap 2.8.2-2.el9 -> 2.9.0-1.fc38
  libsss_nss_idmap 2.8.2-2.el9 -> 2.9.0-1.fc38
  libsss_sudo 2.8.2-2.el9 -> 2.9.0-1.fc38
  libstdc++ 11.3.1-4.3.el9 -> 13.1.1-4.fc38
  libtalloc 2.3.4-1.el9 -> 2.4.0-2.fc38
  libtasn1 4.16.0-8.el9_1 -> 4.19.0-2.fc38
  libtdb 1.4.7-1.el9 -> 1.4.8-1.fc38
  libtevent 0.13.0-1.el9 -> 0.14.1-1.fc38
  libtirpc 1.3.3-1.el9 -> 1.3.3-1.rc1.fc38
  libtool-ltdl 2.4.6-45.el9 -> 2.4.7-6.fc38
  libunistring 0.9.10-15.el9 -> 1.1-3.fc38
  libuser 0.63-12.el9 -> 0.64-2.fc38
  libutempter 1.2.1-6.el9 -> 1.2.1-8.fc38
  libuuid 2.37.4-10.el9 -> 2.38.1-4.fc38
  libuv 1:1.42.0-1.el9 -> 1:1.44.2-3.fc38
  libverto 0.3.2-3.el9 -> 0.3.2-5.fc38
  libwbclient 4.17.5-102.el9 -> 2:4.18.3-3.fc38
  libxcrypt 4.4.18-3.el9 -> 4.4.35-1.fc38
  libxml2 2.9.13-3.el9_1 -> 2.10.4-1.fc38
  libxmlb 0.3.10-1.el9 -> 0.3.11-1.fc38
  libyaml 0.2.5-7.el9 -> 0.2.5-9.fc38
  libzstd 1.5.1-2.el9 -> 1.5.5-1.fc38
  linux-firmware 20230310-133.el9_2 -> 20230515-150.fc38
  linux-firmware-whence 20230310-133.el9_2 -> 20230515-150.fc38
  lmdb-libs 0.9.29-3.el9 -> 0.9.30-1.fc38
  logrotate 3.18.0-8.el9 -> 3.21.0-2.fc38
  lsof 4.94.0-3.el9 -> 4.96.3-3.fc38
  lua-libs 5.4.4-3.el9 -> 5.4.4-9.fc38
  luksmeta 9-12.el9 -> 9-15.fc38
  lz4-libs 1.9.3-5.el9 -> 1.9.4-2.fc38
  lzo 2.10-7.el9 -> 2.10-8.fc38
  mokutil 2:0.6.0-4.el9 -> 2:0.6.0-6.fc38
  mpfr 4.1.0-7.el9 -> 4.1.1-3.fc38
  nano 5.6.1-5.el9 -> 7.2-2.fc38
  ncurses 6.2-8.20210508.el9 -> 6.4-3.20230114.fc38
  ncurses-base 6.2-8.20210508.el9 -> 6.4-3.20230114.fc38
  ncurses-libs 6.2-8.20210508.el9 -> 6.4-3.20230114.fc38
  net-tools 2.0-0.62.20160912git.el9 -> 2.0-0.65.20160912git.fc38
  nettle 3.8-3.el9_0 -> 3.8-3.fc38
  newt 0.52.21-11.el9 -> 0.52.23-2.fc38
  nftables 1:1.0.4-10.el9_1 -> 1:1.0.5-2.fc38
  npth 1.6-8.el9 -> 1.6-12.fc38
  nss-altfiles 2.18.1-20.el9 -> 2.18.1-22.fc38
  numactl-libs 2.0.14-9.el9 -> 2.0.16-2.fc38
  nvme-cli 2.2.1-2.el9 -> 2.4-2.fc38
  oniguruma 6.9.6-1.el9.5 -> 6.9.8-2.D20220919gitb041f6d.fc38.1
  openldap 2.6.2-3.el9 -> 2.6.4-1.fc38
  openssh 8.7p1-28.el9 -> 9.0p1-15.fc38
  openssh-clients 8.7p1-28.el9 -> 9.0p1-15.fc38
  openssh-server 8.7p1-28.el9 -> 9.0p1-15.fc38
  openssl 1:3.0.7-6.el9_2 -> 1:3.0.9-1.fc38
  openssl-libs 1:3.0.7-6.el9_2 -> 1:3.0.9-1.fc38
  os-prober 1.77-10.el9 -> 1.81-3.fc38
  ostree 2022.6-3.el9 -> 2023.4-1.fc38
  ostree-libs 2022.6-3.el9 -> 2023.4-1.fc38
  p11-kit 0.24.1-2.el9 -> 0.24.1-6.fc38
  p11-kit-trust 0.24.1-2.el9 -> 0.24.1-6.fc38
  pam 1.5.1-14.el9 -> 1.5.2-16.fc38
  passwd 0.80-12.el9 -> 0.80-14.fc38
  pcre2 10.40-2.el9 -> 10.42-1.fc38.1
  pcre2-syntax 10.40-2.el9 -> 10.42-1.fc38.1
  pigz 2.5-4.el9 -> 2.7-3.fc38
  pkgconf 1.7.3-10.el9 -> 1.8.0-6.fc38
  pkgconf-m4 1.7.3-10.el9 -> 1.8.0-6.fc38
  pkgconf-pkg-config 1.7.3-10.el9 -> 1.8.0-6.fc38
  podman 3:4.4.1-3.rhaos4.13.el9 -> 5:4.5.1-1.fc38
  policycoreutils 3.5-1.el9 -> 3.5-1.fc38
  polkit 0.117-11.el9 -> 122-3.fc38.1
  polkit-libs 0.117-11.el9 -> 122-3.fc38.1
  polkit-pkla-compat 0.1-21.el9 -> 0.1-23.fc38
  popt 1.18-8.el9 -> 1.19-2.fc38
  protobuf-c 1.3.3-12.el9 -> 1.4.1-4.fc38
  psmisc 23.4-3.el9 -> 23.6-2.fc38
  publicsuffix-list-dafsa 20210518-3.el9 -> 20230614-1.fc38
  readline 8.1-4.el9 -> 8.2-3.fc38
  rpm 4.16.1.3-22.el9 -> 4.18.1-3.fc38
  rpm-libs 4.16.1.3-22.el9 -> 4.18.1-3.fc38
  rpm-ostree 2022.19-3.el9 -> 2023.4-2.fc38
  rpm-ostree-libs 2022.19-3.el9 -> 2023.4-2.fc38
  rpm-plugin-selinux 4.16.1.3-22.el9 -> 4.18.1-3.fc38
  rsync 3.2.3-19.el9 -> 3.2.7-2.fc38
  samba-client-libs 4.17.5-102.el9 -> 2:4.18.3-3.fc38
  samba-common 4.17.5-102.el9 -> 2:4.18.3-3.fc38
  samba-common-libs 4.17.5-102.el9 -> 2:4.18.3-3.fc38
  sed 4.8-9.el9 -> 4.8-12.fc38
  selinux-policy 38.1.11-2.el9_2 -> 38.17-1.fc38
  selinux-policy-targeted 38.1.11-2.el9_2 -> 38.17-1.fc38
  setup 2.13.7-9.el9 -> 2.14.3-2.fc38
  shadow-utils 2:4.9-6.el9 -> 2:4.13-6.fc38
  shadow-utils-subid 2:4.9-6.el9 -> 2:4.13-6.fc38
  shared-mime-info 2.1-5.el9 -> 2.2-3.fc38
  shim-x64 15.6-1.el9 -> 15.6-2
  slang 2.3.2-11.el9 -> 2.3.3-3.fc38
  slirp4netns 1.2.0-3.el9 -> 1.2.0-12.fc38
  snappy 1.1.8-8.el9 -> 1.1.9-7.fc38
  socat 1.7.4.1-5.el9 -> 1.7.4.4-2.fc38
  sqlite-libs 3.34.1-6.el9_1 -> 3.40.1-2.fc38
  squashfs-tools 4.4-8.git1.el9 -> 4.5.1-3.fc38
  ssh-key-dir 0.1.4-1.el9 -> 0.1.4-3.fc38
  sssd-ad 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-client 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-common 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-common-pac 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-ipa 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-krb5 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-krb5-common 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-ldap 2.8.2-2.el9 -> 2.9.0-1.fc38
  sssd-nfs-idmap 2.8.2-2.el9 -> 2.9.0-1.fc38
  sudo 1.9.5p2-9.el9 -> 1.9.13-2.p2.fc38
  systemd 252-14.el9.rhaos4.13 -> 253.4-1.fc38
  systemd-libs 252-14.el9.rhaos4.13 -> 253.4-1.fc38
  systemd-pam 252-14.el9.rhaos4.13 -> 253.4-1.fc38
  systemd-udev 252-14.el9.rhaos4.13 -> 253.4-1.fc38
  tar 2:1.34-6.el9_1 -> 2:1.34-8.fc38
  tpm2-tools 5.2-2.el9_1 -> 5.5-3.fc38
  tpm2-tss 3.0.3-8.el9 -> 4.0.1-3.fc38
  tzdata 2022g-2.el9 -> 2023c-1.fc38
  userspace-rcu 0.12.1-6.el9 -> 0.13.2-2.fc38
  util-linux 2.37.4-10.el9 -> 2.38.1-4.fc38
  util-linux-core 2.37.4-10.el9 -> 2.38.1-4.fc38
  vim-minimal 2:8.2.2637-20.el9_1 -> 2:9.0.1649-1.fc38
  which 2.21-28.el9 -> 2.21-39.fc38
  xfsprogs 5.14.2-1.el9 -> 6.1.0-3.fc38
  xz 5.2.5-8.el9_0 -> 5.4.1-1.fc38
  xz-libs 5.2.5-8.el9_0 -> 5.4.1-1.fc38
  zlib 1.2.11-39.el9 -> 1.2.13-3.fc38
  zstd 1.5.1-2.el9 -> 1.5.5-1.fc38
Downgraded:
  aardvark-dns 2:1.5.0-2.el9 -> 1.6.0-1.fc38
  cloud-utils-growpart 0.33-1.el9 -> 0.31-12.fc38
  conmon 3:2.1.7-1.rhaos4.13.el9 -> 2:2.1.7-2.fc38
  container-selinux 3:2.208.0-2.rhaos4.13.el9 -> 2:2.218.0-1.fc38
  containernetworking-plugins 1:1.2.0-1.el9 -> 1.1.1-16.fc38
  cpio 2.13-16.el9 -> 2.13-14.fc38
  device-mapper 9:1.02.187-7.el9 -> 1.02.189-2.fc38
  device-mapper-event 9:1.02.187-7.el9 -> 1.02.189-2.fc38
  device-mapper-event-libs 9:1.02.187-7.el9 -> 1.02.189-2.fc38
  device-mapper-libs 9:1.02.187-7.el9 -> 1.02.189-2.fc38
  device-mapper-persistent-data 0.9.0-13.el9 -> 0.9.0-10.fc38
  efi-filesystem 6-2.el9_0 -> 5-7.fc38
  jansson 2.14-1.el9 -> 2.13.1-6.fc38
  keyutils 1.6.3-1.el9 -> 1.6.1-6.fc38
  keyutils-libs 1.6.3-1.el9 -> 1.6.1-6.fc38
  libcap 2.48-8.el9 -> 2.48-6.fc38
  libeconf 0.4.1-2.el9 -> 0.4.0-5.fc38
  libnetfilter_conntrack 1.0.9-1.el9 -> 1.0.8-7.fc38
  libteam 1.31-16.el9_1 -> 1.31-7.fc38
  lvm2 9:2.03.17-7.el9 -> 2.03.18-2.fc38
  lvm2-libs 9:2.03.17-7.el9 -> 2.03.18-2.fc38
  mdadm 4.2-8.el9 -> 4.2-4.fc38
  microcode_ctl 4:20220809-2.el9 -> 2:2.1-55.fc38
  netavark 2:1.5.0-2.el9 -> 1.6.0-2.fc38
  procps-ng 3.3.17-11.el9 -> 3.3.17-9.fc38
  rpcbind 1.2.6-5.el9 -> 1.2.6-4.rc2.fc38
  runc 4:1.1.6-3.rhaos4.13.el9 -> 2:1.1.7-1.fc38
  sg3_utils 1.47-9.el9 -> 1.46-5.fc38
  sg3_utils-libs 1.47-9.el9 -> 1.46-5.fc38
  skopeo 2:1.11.2-0.1.el9 -> 1:1.12.0-1.fc38
  stalld 1.17.1-1.el9 -> 1.16-3.fc38
  teamd 1.31-16.el9_1 -> 1.31-7.fc38
  toolbox 0.1.2-1.rhaos4.13.el9 -> 0.0.99.4-1.fc38
  yajl 2.1.0-21.el9 -> 2.1.0-20.fc38
Removed:
  NetworkManager-ovs-1:1.42.2-1.el9.x86_64
  audit-3.0.7-103.el9.x86_64
  checkpolicy-3.5-1.el9.x86_64
  conmon-rs-0.5.1-5.rhaos4.13.git.el9.x86_64
  conntrack-tools-1.4.7-2.el9.x86_64
  cracklib-dicts-2.9.6-27.el9.x86_64
  cri-o-1.26.3-3.rhaos4.13.git641290e.el9.x86_64
  cri-tools-1.26.0-1.el9.x86_64
  crypto-policies-scripts-20221215-1.git9a18988.el9.noarch
  dmidecode-1:3.3-7.el9.x86_64
  firewalld-filesystem-1.2.1-1.el9.noarch
  flashrom-1.2-10.el9.x86_64
  fwupd-plugin-flashrom-1.8.10-2.el9.x86_64
  gawk-all-langpacks-5.1.0-6.el9.x86_64
  gettext-0.21-7.el9.x86_64
  glusterfs-6.0-57.4.el9.x86_64
  glusterfs-client-xlators-6.0-57.4.el9.x86_64
  glusterfs-fuse-6.0-57.4.el9.x86_64
  glusterfs-libs-6.0-57.4.el9.x86_64
  groff-base-1.22.4-10.el9.x86_64
  gssproxy-0.8.4-4.el9.x86_64
  hwdata-0.348-9.7.el9.noarch
  info-6.7-15.el9.x86_64
  initscripts-service-10.11.5-1.el9.noarch
  kernel-modules-extra-5.14.0-284.13.1.el9_2.x86_64
  libbrotli-1.0.9-6.el9.x86_64
  libcurl-7.76.1-23.el9.x86_64
  libdrm-2.4.114-1.el9.x86_64
  libev-4.33-5.el9.x86_64
  libgomp-11.3.1-4.3.el9.x86_64
  libmicrohttpd-1:0.9.72-4.el9.x86_64
  libmspack-0.10.1-0.7.alpha.el9.x86_64
  libnetfilter_cthelper-1.0.0-22.el9.x86_64
  libnetfilter_cttimeout-1.0.0-19.el9.x86_64
  libnetfilter_queue-1.0.5-1.el9.x86_64
  libpciaccess-0.16-6.el9.x86_64
  librhsm-0.0.3-7.el9.x86_64
  libsmbios-2.4.3-4.el9.x86_64
  libssh-0.10.4-8.el9.x86_64
  libssh-config-0.10.4-8.el9.noarch
  libusbx-1.0.26-1.el9.x86_64
  libverto-libev-0.3.2-3.el9.x86_64
  libxcrypt-compat-4.4.18-3.el9.x86_64
  libxslt-1.1.34-9.el9.x86_64
  nfs-utils-1:2.5.4-18.el9.x86_64
  nmap-ncat-3:7.91-12.el9.x86_64
  open-vm-tools-12.1.5-1.el9.x86_64
  openldap-compat-2.6.2-3.el9.x86_64
  openshift-clients-4.13.0-202303241616.p0.g92b1a3d.assembly.stream.el9.x86_64
  openshift-hyperkube-4.13.0-202304211155.p0.gb404935.assembly.stream.el9.x86_64
  openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
  openvswitch3.1-3.1.0-14.el9fdp.x86_64
  ostree-grub2-2022.6-3.el9.x86_64
  pciutils-3.7.0-5.el9.x86_64
  pciutils-libs-3.7.0-5.el9.x86_64
  pcre-8.44-3.el9.3.x86_64
  perl-AutoLoader-5.74-480.el9.noarch
  perl-B-1.80-480.el9.x86_64
  perl-Carp-1.50-460.el9.noarch
  perl-Class-Struct-0.66-480.el9.noarch
  perl-Data-Dumper-2.174-462.el9.x86_64
  perl-Digest-1.19-4.el9.noarch
  perl-Digest-MD5-2.58-4.el9.x86_64
  perl-Encode-4:3.08-462.el9.x86_64
  perl-Errno-1.30-480.el9.x86_64
  perl-Exporter-5.74-461.el9.noarch
  perl-Fcntl-1.13-480.el9.x86_64
  perl-File-Basename-2.85-480.el9.noarch
  perl-File-Path-2.18-4.el9.noarch
  perl-File-Temp-1:0.231.100-4.el9.noarch
  perl-File-stat-1.09-480.el9.noarch
  perl-FileHandle-2.03-480.el9.noarch
  perl-Getopt-Long-1:2.52-4.el9.noarch
  perl-Getopt-Std-1.12-480.el9.noarch
  perl-HTTP-Tiny-0.076-460.el9.noarch
  perl-IO-1.43-480.el9.x86_64
  perl-IO-Socket-IP-0.41-5.el9.noarch
  perl-IO-Socket-SSL-2.073-1.el9.noarch
  perl-IPC-Open3-1.21-480.el9.noarch
  perl-MIME-Base64-3.16-4.el9.x86_64
  perl-Mozilla-CA-20200520-6.el9.noarch
  perl-NDBM_File-1.15-480.el9.x86_64
  perl-Net-SSLeay-1.92-2.el9.x86_64
  perl-POSIX-1.94-480.el9.x86_64
  perl-PathTools-3.78-461.el9.x86_64
  perl-Pod-Escapes-1:1.07-460.el9.noarch
  perl-Pod-Perldoc-3.28.01-461.el9.noarch
  perl-Pod-Simple-1:3.42-4.el9.noarch
  perl-Pod-Usage-4:2.01-4.el9.noarch
  perl-Scalar-List-Utils-4:1.56-461.el9.x86_64
  perl-SelectSaver-1.02-480.el9.noarch
  perl-Socket-4:2.031-4.el9.x86_64
  perl-Storable-1:3.21-460.el9.x86_64
  perl-Symbol-1.08-480.el9.noarch
  perl-Term-ANSIColor-5.01-461.el9.noarch
  perl-Term-Cap-1.17-460.el9.noarch
  perl-Text-ParseWords-3.30-460.el9.noarch
  perl-Text-Tabs+Wrap-2013.0523-460.el9.noarch
  perl-Time-Local-2:1.300-7.el9.noarch
  perl-URI-5.09-3.el9.noarch
  perl-base-2.27-480.el9.noarch
  perl-constant-1.33-461.el9.noarch
  perl-if-0.60.800-480.el9.noarch
  perl-interpreter-4:5.32.1-480.el9.x86_64
  perl-libnet-3.13-4.el9.noarch
  perl-libs-4:5.32.1-480.el9.x86_64
  perl-mro-1.23-480.el9.x86_64
  perl-overload-1.31-480.el9.noarch
  perl-overloading-0.02-480.el9.noarch
  perl-parent-1:0.238-460.el9.noarch
  perl-podlators-1:4.14-460.el9.noarch
  perl-subs-1.03-480.el9.noarch
  perl-vars-1.05-480.el9.noarch
  policycoreutils-python-utils-3.5-1.el9.noarch
  python-unversioned-command-3.9.16-1.el9.noarch
  python3-3.9.16-1.el9.x86_64
  python3-audit-3.0.7-103.el9.x86_64
  python3-distro-1.5.0-7.el9.noarch
  python3-libs-3.9.16-1.el9.x86_64
  python3-libselinux-3.5-1.el9.x86_64
  python3-libsemanage-3.5-1.el9.x86_64
  python3-pip-wheel-21.2.3-6.el9.noarch
  python3-policycoreutils-3.5-1.el9.noarch
  python3-pyyaml-5.4.1-6.el9.x86_64
  python3-setools-4.4.1-1.el9.x86_64
  python3-setuptools-53.0.0-12.el9.noarch
  python3-setuptools-wheel-53.0.0-12.el9.noarch
  qemu-guest-agent-17:7.2.0-14.el9_2.x86_64
  quota-1:4.06-6.el9.x86_64
  quota-nls-1:4.06-6.el9.noarch
  rdma-core-44.0-2.el9.x86_64
  redhat-release-9.2-0.13.el9.x86_64
  redhat-release-eula-9.2-0.13.el9.x86_64
  sssd-2.8.2-2.el9.x86_64
  sssd-proxy-2.8.2-2.el9.x86_64
  strace-5.18-2.el9.x86_64
  subscription-manager-rhsm-certificates-20220623-1.el9.noarch
  systemd-journal-remote-252-14.el9.rhaos4.13.x86_64
  systemd-rpm-macros-252-14.el9.rhaos4.13.noarch
  tmux-3.2a-4.el9.x86_64
  unbound-libs-1.16.2-3.el9.x86_64
  xmlsec1-1.2.29-9.el9.x86_64
  xmlsec1-openssl-1.2.29-9.el9.x86_64
Added:
  amd-gpu-firmware-20230515-150.fc38.noarch
  atheros-firmware-20230515-150.fc38.noarch
  brcmfmac-firmware-20230515-150.fc38.noarch
  btrfs-progs-6.3.2-1.fc38.x86_64
  catatonit-0.1.7-14.fc38.x86_64
  console-login-helper-messages-motdgen-0.21.3-4.fc38.noarch
  containerd-1.6.19-1.fc38.x86_64
  containers-common-extra-4:1-89.fc38.noarch
  duktape-2.7.0-2.fc38.x86_64
  fedora-gpg-keys-38-1.noarch
  fedora-release-common-38-36.noarch
  fedora-release-coreos-38-36.noarch
  fedora-release-identity-coreos-38-36.noarch
  fedora-repos-38-1.noarch
  fedora-repos-archive-38-1.noarch
  fedora-repos-modular-38-1.noarch
  fedora-repos-ostree-38-1.noarch
  flatpak-session-helper-1.15.4-1.fc38.x86_64
  fuse-sshfs-3.7.3-3.fc38.x86_64
  gettext-envsubst-0.21.1-2.fc38.x86_64
  gettext-runtime-0.21.1-2.fc38.x86_64
  google-compute-engine-guest-configs-udev-20230526.00-3.fc38.noarch
  intel-gpu-firmware-20230515-150.fc38.noarch
  ipcalc-1.0.3-1.fc38.x86_64
  iptables-legacy-1.8.9-4.fc38.x86_64
  iptables-legacy-libs-1.8.9-4.fc38.x86_64
  iptables-services-1.8.9-4.fc38.noarch
  iptables-utils-1.8.9-4.fc38.x86_64
  jemalloc-5.3.0-3.fc38.x86_64
  kbd-legacy-2.5.1-5.fc38.noarch
  libargon2-20190702-2.fc38.x86_64
  libbsd-0.11.7-4.fc38.x86_64
  libcurl-minimal-8.0.1-2.fc38.x86_64
  libmd-1.1.0-1.fc38.x86_64
  libnsl2-2.0.0-5.fc38.x86_64
  libunistring1.0-1.0-1.fc38.x86_64
  libusb1-1.0.26-2.fc38.x86_64
  linux-atm-libs-2.5.1-34.fc38.x86_64
  moby-engine-20.10.23-1.fc38.x86_64
  mt7xxx-firmware-20230515-150.fc38.noarch
  nano-default-editor-7.2-2.fc38.noarch
  nfs-utils-coreos-1:2.6.3-0.fc38.x86_64
  nmstate-2.2.12-3.fc38.x86_64
  nvidia-gpu-firmware-20230515-150.fc38.noarch
  pam-libs-1.5.2-16.fc38.x86_64
  passt-0^20230509.g96f8d55-1.fc38.x86_64
  passt-selinux-0^20230509.g96f8d55-1.fc38.noarch
  podman-plugins-5:4.5.1-1.fc38.x86_64
  realtek-firmware-20230515-150.fc38.noarch
  rpm-sequoia-1.4.0-3.fc38.x86_64
  systemd-container-253.4-1.fc38.x86_64
  systemd-resolved-253.4-1.fc38.x86_64
  vim-data-2:9.0.1649-1.fc38.noarch
  wireguard-tools-1.0.20210914-4.fc38.x86_64
  xxhash-libs-0.8.1-4.fc38.x86_64
  zchunk-libs-1.3.1-1.fc38.x86_64
  zincati-0.0.25-4.fc38.x86_64
  zram-generator-1.1.2-4.fc38.x86_64
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 745051837  Available: 745051837
Inodes: Total: 300647616  Free: 300579053
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 21682 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers
warning: 1717 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
  inode clashes: 884
  unknown:       0
  ok:            3472

warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled

Pre-verification continued from comment 23:

[root@cosa-devsh ~]# rpm -qa ostree
ostree-2022.6-3.el9.x86_64
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree usroverlay
Development mode enabled.  A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.

[root@cosa-devsh ~]# curl -L -k --fail --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/ostree/2023.1/4.el9_2/x86_64/ostree-{,libs-,grub2-}2023.1-4.el9_2.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  242k  100  242k    0     0   488k      0 --:--:-- --:--:-- --:--:--  488k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  441k  100  441k    0     0  1128k      0 --:--:-- --:--:-- --:--:-- 1125k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10057  100 10057    0     0   185k      0 --:--:-- --:--:-- --:--:--  185k

[root@cosa-devsh ~]# rpm -Uvh ostree-*
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:ostree-libs-2023.1-4.el9_2       ################################# [ 17%]
   2:ostree-2023.1-4.el9_2            ################################# [ 33%]
   3:ostree-grub2-2023.1-4.el9_2      ################################# [ 50%]
Cleaning up / removing...
   4:ostree-grub2-2022.6-3.el9        ################################# [ 67%]
   5:ostree-2022.6-3.el9              ################################# [ 83%]
   6:ostree-libs-2022.6-3.el9         ################################# [100%]
/usr/lib/tmpfiles.d/tmp.conf:12: Duplicate line for path "/var/tmp", ignoring.
/usr/lib/tmpfiles.d/var.conf:14: Duplicate line for path "/var/log", ignoring.
/usr/lib/tmpfiles.d/var.conf:19: Duplicate line for path "/var/cache", ignoring.
/usr/lib/tmpfiles.d/var.conf:21: Duplicate line for path "/var/lib", ignoring.
/usr/lib/tmpfiles.d/var.conf:23: Duplicate line for path "/var/spool", ignoring.
"/home" already exists and is not a directory.
"/srv" already exists and is not a directory.
"/root" already exists and is not a directory.
[root@cosa-devsh ~]# systemctl restart rpm-ostreed
[root@cosa-devsh ~]# rpm -qa ostree
ostree-2023.1-4.el9_2.x86_64
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 745046135  Available: 745046135
Inodes: Total: 300647616  Free: 300574187
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 21779 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers
warning: 1717 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
  inode clashes: 884
  unknown:       0
  ok:            3472

warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
                   Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132
                  Version: 38.20230625.3.0 (2023-07-21T15:15:37Z)
                     Diff: 334 upgraded, 34 downgraded, 143 removed, 58 added

● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360
                  Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z)
                 Unlocked: development
[root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo
error: Pruning docker://quay.io/fedora/fedora-coreos:stable: unlink(ostree/container/image/docker_3A__2F__2F_quay_2E_io/fedora/fedora-coreos_3A_stable): Read-only file system
[root@cosa-devsh ~]# mount -o remount,rw /sysroot
[root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo
Removed images: 1 layers: 0
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
                   Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132
                  Version: 38.20230625.3.0 (2023-07-21T15:15:37Z)
                     Diff: 334 upgraded, 34 downgraded, 143 removed, 58 added

● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360
                  Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z)
                 Unlocked: development
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 745046136  Available: 745046136
Inodes: Total: 300647616  Free: 300574188
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 21779 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers
warning: 1717 potentially colliding inodes found
Verifying 0 ostree-container images
OK no corrupted images found
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel
Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel
Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel (digest: sha256:da542dbe516f4cbf6498f9a64b4702e1d11885b800675265b55ab36028fb3d8e)
'''
'''
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 745046052  Available: 745046052
Inodes: Total: 300647616  Free: 300578976
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 22673 objects with > 32 bit inode numbers and 17806 objects with <= 32 bit inode numbers
warning: 1717 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_726728f89dea4a2f5aafe923b606d9bc9783a63475a698dbc9edc525e5dd9700
Image has 0 derived layers
OK image docker://quay.io/fedora/fedora-coreos:testing-devel (verified=11

OK no corrupted images found

OK, our initial analysis code has landed in CentOS Stream 9 builds.  This is *checking only* - it won't attempt to hotfix anything.  We're just trying to gather data.  This will also aid us in figuring out how many and exactly which files may have been swapped which is going to be useful for analysis.  We also want to check that we're actually hitting the bug we think we are!  I'm quite confident in this, but it's *possible* there's something else going wrong.  This will help us know, without changing anything right now - be sure to use the --dry-run flag!
 
Here's instructions to test it out:

First...we need a container image that has rpm-ostree; and I looked around and it's actually most convenient to use the RHEL CoreOS 9 image *as a* container (a new power since we landed https://docs.openshift.com/container-platform/4.13/post_installation_configuration/coreos-layering.html#coreos-layering )

Per that doc, we need to find the reference to the image:

```
$ oc adm release info --image-for=rhel-coreos quay.io/openshift-release-dev/ocp-release:4.13.3-x86_64
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6973e9353f29e678cad79fe768c22cfd6697d8aa30d2aeaa78cceea989925ded
```

This is for x86_64.  We're going to spawn a privileged pod on a node using that base image:

```
$ oc debug --image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6973e9353f29e678cad79fe768c22cfd6697d8aa30d2aeaa78cceea989925ded node/ci-ln-6fkcxpt-72292-m58pv-worker-a-9gq6x
```
Replace the node name with the node we want to analyze.

Inside the pod shell, we're going to pull the updated rpm-ostree and ostree from CentOS Stream 9:

```
$ rpm -Uvh https://kojihub.stream.centos.org/kojifiles/packages/rpm-ostree/2023.5/2.el9/x86_64/rpm-ostree-{,libs-}2023.5-2.el9.x86_64.rpm https://kojihub.stream.centos.org/kojifiles/packages/ostree/2023.4/2.el9/x86_64/ostree-{,libs-,grub2-}2023.4-2.el9.x86_64.rpm
```

(Ignore the errors about /bin/sh, they're an unrelated benign selinux bug that is harmless)

Finally, we can run the analysis tooling in "check only" mode:

```
$ ostree provisional-repair repair --sysroot /host/sysroot --dry-run
Running in container, assuming we can remount /host/sysroot writable
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
ostree inode check:
  64bit inodes: 0
  32 bit inodes: 28922
  collisions: 0

OK no colliding inodes found
$
```

Test with latest 4.13 and update ostree/rpm-ostree, let me know if there is something that I missed.

$ oc adm release info --image-for=rhel-coreos quay.io/openshift-release-dev/ocp-release:4.13.6-x86_64
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8436f6414523d81cc41a3dab8a31c54f464e146928675180f304a478993dab26

$ oc debug --image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8436f6414523d81cc41a3dab8a31c54f464e146928675180f304a478993dab26 node/ip-10-0-213-53.us-east-2.compute.internal
Temporary namespace openshift-debug-h7nhz is created for debugging node...
Starting pod/ip-10-0-213-53us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.213.53
If you don't see a command prompt, try pressing enter.

sh-5.1# rpm -Uvh https://kojihub.stream.centos.org/kojifiles/packages/rpm-ostree/2023.5/2.el9/x86_64/rpm-ostree-{,libs-}2023.5-2.el9.x86_64.rpm https://kojihu.stream.centos.org/kojifiles/packages/ostree/2023.4/2.el9/x86_64/ostree-{,libs-,grub2-}2023.4-2.el9.x86_64.rpm
...

sh-5.1# ostree provisional-repair repair --sysroot /host/sysroot --dry-run
Running in container, assuming we can remount /host/sysroot writable
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
ostree inode check:
  64bit inodes: 0
  32 bit inodes: 26201
  collisions: 0

OK no colliding inodes found
OK no corruption found

At a procedural level, this bug is for RHEL 9.3.   Which was actually fixed by
https://gitlab.com/redhat/centos-stream/rpms/ostree/-/merge_requests/20
a month ago; we've long had the core bug fixed for 9.3.

There's nothing more to do logically for this bug, but let's keep it open until we've at least got queued errata for 9.2 and 8.6.  After that I'll likely close, and the remaining zstream tracking bugs become the canonical place to track shipping.

I would like to continue using this bug as a centralized information hub.

I spent some time looking into exactly what XFS setups are at risk.  

First, https://access.redhat.com/solutions/2884651 and https://mirrors.edge.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf are useful resources.

It looks like provisioning RHCOS 4.13 or below to a ~1TB disk will result in a setup like:
agcount=1109, agsize=242112
will only have 31 bit inodes, and is hence safe.

If we provision XFS *correctly* (i.e. with a low AG count) which requires a workaround Butane/Ignition (linked above), then
even going to 2T is safely at 32 bits.

3T and above though will require 33 bits which rounds up to 64.

Based on analysis so far, I am comfortable saying that it's safe to use 1TB.

For OCP 4.12, the fixed ostree is part of https://amd64.ocp.releases.ci.openshift.org/releasestream/4.12.0-0.nightly/release/4.12.0-0.nightly-2023-07-27-023653
For OCP 4.13, the fixed ostree is part of https://amd64.ocp.releases.ci.openshift.org/releasestream/4.13.0-0.nightly/release/4.13.0-0.nightly-2023-07-26-041700

At this point again, at at technical level we already have the fix queued in current C9S and RHEL 9.3.  The clones of this bug will track the zstream releases.