OK, I'm working on tooling to better detect this situation (and after that, repair it). But basically, this will generally only occur with large disks (which for XFS, result in multiple allocation groups which hand out distinct inode ranges as I understand it). That's how it's sailed through all of our CI so far. (Also, we don't actually verify the deployment roots today, which is an ostree design flaw that makes this problem much less visible; that design flaw will be fixed with the ongoing composefs work) The original reporter gave this handy bit: > The issue was discovered on PoC machine equipped with a large NVME (3.4TB), but the bug can be easily reproduced using cosa run -m 4000 --qemu-size +3TB', followed by installation of any package using rpm-ostree install`. The Using e.g. ls -Rli /ostree/repo/objects - any case where > 32 bit inode numbers are at risk.
Hi Colin, I am able to reproduce the collision with the butane config. aaradhak@fedora ~/rhcos/rhcos-4.13 $ cosa run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size '+3T' --qemu-memory 8192 -B reprovision-xfs.bu COREOS_ASSEMBLER_CONTAINER=quay.io/coreos-assembler/coreos-assembler:rhcos-4.13 COREOS_ASSEMBLER_ADD_CERTS=y + podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/rhcos/rhcos-4.13:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa -v=/etc/pki/ca-trust:/etc/pki/ca-trust:ro quay.io/coreos-assembler/coreos-assembler:rhcos-4.13 run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size +3T --qemu-memory 8192 -B reprovision-xfs.bu Red Hat Enterprise Linux CoreOS 413.92.202305021736-0 Part of OpenShift 4.13, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.13/architecture/architecture-rhcos.html --- Last login: Fri Jul 21 15:11:47 2023 [core@cosa-devsh ~]$ xfs_info / |grep agcount= meta-data=/dev/vda4 isize=512 agcount=4, agsize=187904768 blks [core@cosa-devsh ~]$ sudo -i [root@cosa-devsh ~]# alias verify-ostree="podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run" [root@cosa-devsh ~]# verify-ostree Trying to pull quay.io/cgwalters/ostree-ext-dev:latest... Getting image source signatures Copying blob ad5077952f52 done Copying blob dbc9fb14b970 done Copying blob 19d2bba2169e done Copying config 991ff7104a done Writing manifest to image destination Storing signatures Running in container, assuming we can remount /rootfs/sysroot writable OK no derived commits found. [root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:stable (digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132) ''' ''' [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745051837 Available: 745051837 Inodes: Total: 300647616 Free: 300579053 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21682 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f Image has 0 derived layers warning: Found corrupted merge commit inode clashes: 884 unknown: 0 ok: 3472 warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable error: Found potential corruption, dry-run mode enabled Pre-verification continued from comment 23: [root@cosa-devsh ~]# rpm -qa ostree ostree-2022.6-3.el9.x86_64 [root@cosa-devsh ~]# [root@cosa-devsh ~]# [root@cosa-devsh ~]# [root@cosa-devsh ~]# rpm-ostree usroverlay Development mode enabled. A writable overlayfs is now mounted on /usr. All changes there will be discarded on reboot. [root@cosa-devsh ~]# curl -L -k --fail --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/ostree/2023.1/4.el9_2/x86_64/ostree-{,libs-,grub2-}2023.1-4.el9_2.x86_64.rpm % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 242k 100 242k 0 0 488k 0 --:--:-- --:--:-- --:--:-- 488k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 441k 100 441k 0 0 1128k 0 --:--:-- --:--:-- --:--:-- 1125k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10057 100 10057 0 0 185k 0 --:--:-- --:--:-- --:--:-- 185k [root@cosa-devsh ~]# rpm -Uvh ostree-* Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ostree-libs-2023.1-4.el9_2 ################################# [ 17%] 2:ostree-2023.1-4.el9_2 ################################# [ 33%] 3:ostree-grub2-2023.1-4.el9_2 ################################# [ 50%] Cleaning up / removing... 4:ostree-grub2-2022.6-3.el9 ################################# [ 67%] 5:ostree-2022.6-3.el9 ################################# [ 83%] 6:ostree-libs-2022.6-3.el9 ################################# [100%] /usr/lib/tmpfiles.d/tmp.conf:12: Duplicate line for path "/var/tmp", ignoring. /usr/lib/tmpfiles.d/var.conf:14: Duplicate line for path "/var/log", ignoring. /usr/lib/tmpfiles.d/var.conf:19: Duplicate line for path "/var/cache", ignoring. /usr/lib/tmpfiles.d/var.conf:21: Duplicate line for path "/var/lib", ignoring. /usr/lib/tmpfiles.d/var.conf:23: Duplicate line for path "/var/spool", ignoring. "/home" already exists and is not a directory. "/srv" already exists and is not a directory. "/root" already exists and is not a directory. [root@cosa-devsh ~]# systemctl restart rpm-ostreed [root@cosa-devsh ~]# rpm -qa ostree ostree-2023.1-4.el9_2.x86_64 [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745046135 Available: 745046135 Inodes: Total: 300647616 Free: 300574187 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21779 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f Image has 0 derived layers warning: Found corrupted merge commit inode clashes: 884 unknown: 0 ok: 3472 warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable error: Found potential corruption, dry-run mode enabled [root@cosa-devsh ~]# rpm-ostree status State: idle Deployments: ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132 Version: 38.20230625.3.0 (2023-07-21T15:15:37Z) Diff: 334 upgraded, 34 downgraded, 143 removed, 58 added ● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360 Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z) Unlocked: development [root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo error: Pruning docker://quay.io/fedora/fedora-coreos:stable: unlink(ostree/container/image/docker_3A__2F__2F_quay_2E_io/fedora/fedora-coreos_3A_stable): Read-only file system [root@cosa-devsh ~]# mount -o remount,rw /sysroot [root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo Removed images: 1 layers: 0 [root@cosa-devsh ~]# rpm-ostree status State: idle Deployments: ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132 Version: 38.20230625.3.0 (2023-07-21T15:15:37Z) Diff: 334 upgraded, 34 downgraded, 143 removed, 58 added ● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360 Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z) Unlocked: development [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745046136 Available: 745046136 Inodes: Total: 300647616 Free: 300574188 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21779 objects with > 32 bit inode numbers and 18888 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 0 ostree-container images OK no corrupted images found [root@cosa-devsh ~]# [root@cosa-devsh ~]# [root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel Pulling manifest: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel Importing: ostree-unverified-image:docker://quay.io/fedora/fedora-coreos:testing-devel (digest: sha256:da542dbe516f4cbf6498f9a64b4702e1d11885b800675265b55ab36028fb3d8e) ''' ''' Changes queued for next boot. Run "systemctl reboot" to start a reboot [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745046052 Available: 745046052 Inodes: Total: 300647616 Free: 300578976 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 22673 objects with > 32 bit inode numbers and 17806 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_726728f89dea4a2f5aafe923b606d9bc9783a63475a698dbc9edc525e5dd9700 Image has 0 derived layers OK image docker://quay.io/fedora/fedora-coreos:testing-devel (verified=11 OK no corrupted images found
Verification done based on comment 2: aaradhak@fedora ~/rhcos/rhcos-4.13 $ cosa run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size '+3T' --qemu-memory 8192 -B reprovision-xfs.bu COREOS_ASSEMBLER_CONTAINER=quay.io/coreos-assembler/coreos-assembler:rhcos-4.13 COREOS_ASSEMBLER_ADD_CERTS=y + podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/rhcos/rhcos-4.13:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa -v=/etc/pki/ca-trust:/etc/pki/ca-trust:ro quay.io/coreos-assembler/coreos-assembler:rhcos-4.13 run --qemu-image rhcos-413.92.202305021736-0-qemu.x86_64.qcow2 --qemu-size +3T --qemu-memory 8192 -B reprovision-xfs.bu Red Hat Enterprise Linux CoreOS 413.92.202305021736-0 Part of OpenShift 4.13, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.13/architecture/architecture-rhcos.html --- Last login: Thu Jul 27 15:40:59 2023 [core@cosa-devsh ~]$ xfs_info / |grep agcount= meta-data=/dev/vda4 isize=512 agcount=4, agsize=187904768 blks [core@cosa-devsh ~]$ sudo -i [root@cosa-devsh ~]# alias verify-ostree="podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run" [root@cosa-devsh ~]# verify-ostree Trying to pull quay.io/cgwalters/ostree-ext-dev:latest... Getting image source signatures Copying blob dbc9fb14b970 done Copying blob ad5077952f52 done Copying blob 19d2bba2169e done Copying config 991ff7104a done Writing manifest to image destination Storing signatures Running in container, assuming we can remount /rootfs/sysroot writable OK no derived commits found. [root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable ''' ''' [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745050770 Available: 745050770 Inodes: Total: 300647616 Free: 300578887 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21787 objects with > 32 bit inode numbers and 18928 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_df74f73ba7ae2bec107ad6f2d01fdc4d55e0a9cc4b97f3ca5a5e82295b184c75 Image has 0 derived layers warning: Found corrupted merge commit inode clashes: 915 unknown: 0 ok: 3506 warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable error: Found potential corruption, dry-run mode enabled [root@cosa-devsh ~]# rpm -qa ostree ostree-2022.6-3.el9.x86_64 [root@cosa-devsh ~]# rpm-ostree usroverlay Development mode enabled. A writable overlayfs is now mounted on /usr. All changes there will be discarded on reboot. [root@cosa-devsh ~]# curl -L -k --fail --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/ostree/2023.1/4.el9_2/x86_64/ostree-{,libs-,grub2-}2023.1-4.el9_2.x86_64.rpm % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 242k 100 242k 0 0 230k 0 0:00:01 0:00:01 --:--:-- 230k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 441k 100 441k 0 0 856k 0 --:--:-- --:--:-- --:--:-- 856k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10057 100 10057 0 0 148k 0 --:--:-- --:--:-- --:--:-- 148k [root@cosa-devsh ~]# rpm -Uvh ostree-* Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:ostree-libs-2023.1-4.el9_2 ################################# [ 17%] 2:ostree-2023.1-4.el9_2 ################################# [ 33%] 3:ostree-grub2-2023.1-4.el9_2 ################################# [ 50%] Cleaning up / removing... 4:ostree-grub2-2022.6-3.el9 ################################# [ 67%] 5:ostree-2022.6-3.el9 ################################# [ 83%] 6:ostree-libs-2022.6-3.el9 ################################# [100%] /usr/lib/tmpfiles.d/tmp.conf:12: Duplicate line for path "/var/tmp", ignoring. /usr/lib/tmpfiles.d/var.conf:14: Duplicate line for path "/var/log", ignoring. /usr/lib/tmpfiles.d/var.conf:19: Duplicate line for path "/var/cache", ignoring. /usr/lib/tmpfiles.d/var.conf:21: Duplicate line for path "/var/lib", ignoring. /usr/lib/tmpfiles.d/var.conf:23: Duplicate line for path "/var/spool", ignoring. "/home" already exists and is not a directory. "/srv" already exists and is not a directory. "/root" already exists and is not a directory. [root@cosa-devsh ~]# systemctl restart rpm-ostreed [root@cosa-devsh ~]# rpm -qa ostree ostree-2023.1-4.el9_2.x86_64 [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745047001 Available: 745047001 Inodes: Total: 300647616 Free: 300578758 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21787 objects with > 32 bit inode numbers and 18928 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 1 ostree-container images Verifying with base ostree layer ostree/container/blob/sha256_3A_df74f73ba7ae2bec107ad6f2d01fdc4d55e0a9cc4b97f3ca5a5e82295b184c75 Image has 0 derived layers warning: Found corrupted merge commit inode clashes: 915 unknown: 0 ok: 3506 warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable error: Found potential corruption, dry-run mode enabled [root@cosa-devsh ~]# rpm-ostree status State: idle Deployments: ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Digest: sha256:8c43eefdab92c34b9386ac8803a8ae5e6568b68fbadf1a47b861e6671afa3615 Version: 38.20230709.3.0 (2023-07-27T15:43:02Z) Diff: 335 upgraded, 33 downgraded, 143 removed, 58 added ● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360 Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z) Unlocked: development [root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo error: Pruning docker://quay.io/fedora/fedora-coreos:stable: unlink(ostree/container/image/docker_3A__2F__2F_quay_2E_io/fedora/fedora-coreos_3A_stable): Read-only file system [root@cosa-devsh ~]# mount -o remount,rw /sysroot [root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo Removed images: 1 layers: 0 [root@cosa-devsh ~]# rpm-ostree status State: idle Deployments: ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable Digest: sha256:8c43eefdab92c34b9386ac8803a8ae5e6568b68fbadf1a47b861e6671afa3615 Version: 38.20230709.3.0 (2023-07-27T15:43:02Z) Diff: 335 upgraded, 33 downgraded, 143 removed, 58 added ● f069b5aca6a53934461a7361690bd20ad4aa3e223b6d12e77d54dc3ab4f48360 Version: 413.92.202305021736-0 (2023-05-02T17:39:22Z) Unlocked: development [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745047001 Available: 745047001 Inodes: Total: 300647616 Free: 300578759 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree objects... Analyzed 21787 objects with > 32 bit inode numbers and 18928 objects with <= 32 bit inode numbers warning: 1717 potentially colliding inodes found Verifying 0 ostree-container images OK no corrupted images found [root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel ''' ''' [root@cosa-devsh ~]# verify-ostree Running in container, assuming we can remount /rootfs/sysroot writable Found 1 derived commits Backing filesystem information: File: "." ID: fc0400000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 751252071 Free: 745049981 Available: 745049981 Inodes: Total: 300647616 Free: 300578940 Attempting analysis of ostree state for files that may be incorrectly linked. For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 Gathering inodes for ostree object
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ostree bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:4375