Bug 2224107

Summary: backport https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 [rhel-8.8.0.z]
Product: Red Hat Enterprise Linux 8 Reporter: RHEL Program Management Team <pgm-rhel-tools>
Component: ostreeAssignee: Colin Walters <walters>
Status: VERIFIED --- QA Contact: Aashish Radhakrishnan <aaradhak>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.6CC: aaradhak, dornelas, rhcos-sst, rhcos-sst-qe, walters
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ostree-2022.2-7.el8_8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2224102 Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2224102    
Bug Blocks:    

Comment 1 Aashish Radhakrishnan 2023-07-24 20:13:47 UTC 
Pre-verification:
brew build - https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2605167

aaradhak@fedora ~/rhcos/rhcos-4.12 $ cosa run --qemu-image rhcos-412.86.202305030814-0-qemu.x86_64.qcow2 --qemu-size '+3T' --qemu-memory 8192 -B reprovision-xfs.bu
COREOS_ASSEMBLER_CONTAINER=quay.io/coreos-assembler/coreos-assembler:rhcos-4.12
COREOS_ASSEMBLER_ADD_CERTS=y
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/rhcos/rhcos-4.12:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa -v=/etc/pki/ca-trust:/etc/pki/ca-trust:ro quay.io/coreos-assembler/coreos-assembler:rhcos-4.12 run --qemu-image rhcos-412.86.202305030814-0-qemu.x86_64.qcow2 --qemu-size +3T --qemu-memory 8192 -B reprovision-xfs.bu
Red Hat Enterprise Linux CoreOS 412.86.202305030814-0
  Part of OpenShift 4.12, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.12/architecture/architecture-rhcos.html

---
Last login: Mon Jul 24 19:28:29 2023
[core@cosa-devsh ~]$ xfs_info / |grep agcount=
meta-data=/dev/vda4              isize=512    agcount=4, agsize=187904768 blks
[core@cosa-devsh ~]$ sudo -i
[root@cosa-devsh ~]# alias verify-ostree="podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run"
[root@cosa-devsh ~]# verify-ostree 
Trying to pull quay.io/cgwalters/ostree-ext-dev:latest...
Getting image source signatures
Copying blob 19d2bba2169e done  
Copying blob ad5077952f52 done  
Copying blob dbc9fb14b970 done  
Copying config 991ff7104a done  
Writing manifest to image destination
Storing signatures
Running in container, assuming we can remount /rootfs/sysroot writable
OK no derived commits found.
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
'''
'''
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 744866966  Available: 744866966
Inodes: Total: 300647616  Free: 300569832
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 25504 objects with > 32 bit inode numbers and 23380 objects with <= 32 bit inode numbers
warning: 1187 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
  inode clashes: 27
  unknown:       0
  ok:            2109

warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled
[root@cosa-devsh ~]# rpm -qa ostree
ostree-2022.2-5.el8.x86_64
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
                   Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132
                  Version: 38.20230625.3.0 (2023-07-24T19:39:17Z)
                     Diff: 337 upgraded, 13 downgraded, 153 removed, 76 added

* 2aa485c7967e1b147f05d41a773d725878989c826b3b3c8e927585d5b388d9ca
                  Version: 412.86.202305030814-0 (2023-05-03T08:17:34Z)
[root@cosa-devsh ~]# rpm-ostree usroverlay
Development mode enabled.  A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.
[root@cosa-devsh ~]# curl -L -k --fail --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/ostree/2022.2/7.el8_8/x86_64/ostree-{,libs-,grub2-}2022.2-7.el8_8.x86_64.rpm                                                   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  252k  100  252k    0     0   449k      0 --:--:-- --:--:-- --:--:--  449k
100  445k  100  445k    0     0   758k      0 --:--:-- --:--:-- --:--:--  758k
100 25712  100 25712    0     0   343k      0 --:--:-- --:--:-- --:--:--  343k
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm -Uvh ostree-*
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:ostree-libs-2022.2-7.el8_8       ################################# [ 17%]
   2:ostree-2022.2-7.el8_8            ################################# [ 33%]
   3:ostree-grub2-2022.2-7.el8_8      ################################# [ 50%]
Cleaning up / removing...
   4:ostree-grub2-2022.2-5.el8        ################################# [ 67%]
   5:ostree-2022.2-5.el8              ################################# [ 83%]
   6:ostree-libs-2022.2-5.el8         ################################# [100%]
[/usr/lib/tmpfiles.d/pkg-dbus-daemon.conf:1] Duplicate line for path "/var/lib/dbus", ignoring.
[/usr/lib/tmpfiles.d/tmp.conf:12] Duplicate line for path "/var/tmp", ignoring.
[/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
[/usr/lib/tmpfiles.d/var.conf:19] Duplicate line for path "/var/cache", ignoring.
[/usr/lib/tmpfiles.d/var.conf:21] Duplicate line for path "/var/lib", ignoring.
[/usr/lib/tmpfiles.d/var.conf:23] Duplicate line for path "/var/spool", ignoring.
"/home" already exists and is not a directory.
"/srv" already exists and is not a directory.
[root@cosa-devsh ~]# systemctl restart rpm-ostreed
[root@cosa-devsh ~]# rpm -qa ostree
ostree-2022.2-7.el8_8.x86_64
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 744864983  Available: 744864983
Inodes: Total: 300647616  Free: 300569683
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 25504 objects with > 32 bit inode numbers and 23380 objects with <= 32 bit inode numbers
warning: 1187 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
  inode clashes: 27
  unknown:       0
  ok:            2109

warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled
[root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo
error: Pruning docker://quay.io/fedora/fedora-coreos:stable: unlink(ostree/container/image/docker_3A__2F__2F_quay_2E_io/fedora/fedora-coreos_3A_stable): Read-only file system
[root@cosa-devsh ~]# mount -o remount,rw /sysroot
[root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo
Removed images: 1 layers: 0
[root@cosa-devsh ~]# rpm-ostree status 
State: idle
Deployments:
  ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
                   Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132
                  Version: 38.20230625.3.0 (2023-07-24T19:39:17Z)
                     Diff: 337 upgraded, 13 downgraded, 153 removed, 76 added

* 2aa485c7967e1b147f05d41a773d725878989c826b3b3c8e927585d5b388d9ca
                  Version: 412.86.202305030814-0 (2023-05-03T08:17:34Z)
                 Unlocked: development
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel
'''
'''
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 744864171  Available: 744864171
Inodes: Total: 300647616  Free: 300569627
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 25220 objects with > 32 bit inode numbers and 23696 objects with <= 32 bit inode numbers
warning: 1250 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_4258bb33aab6c4f5ef257854f1a22d03b4f6387cabbf5b4eaa956d9aca7d50ec
Image has 0 derived layers
OK image docker://quay.io/fedora/fedora-coreos:testing-devel (verified=11

OK no corrupted images found
Comment 5 Aashish Radhakrishnan 2023-07-28 16:35:43 UTC 
Verification done based on comment 1