2224107 – backport https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 [rhel-8.8.0.z]

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2224107 - backport https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912 [rhel-8.8.0.z]

Summary: backport https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	ostree
Sub Component:
Version:	8.6
Hardware:	All
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	---
Assignee:	Colin Walters
QA Contact:	Aashish Radhakrishnan
Docs Contact:
URL:
Whiteboard:
Depends On:	2224102
Blocks:
TreeView+	depends on / blocked

Reported:	2023-07-19 19:27 UTC by RHEL Program Management Team
Modified:	2023-09-19 14:40 UTC (History)
CC List:	5 users (show)
Fixed In Version:	ostree-2022.2-7.el8_8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	2224102
Environment:
Last Closed:	2023-09-19 14:35:44 UTC
Type:	---
Target Upstream Version:
Embargoed:
Dependent Products:
Flags:	pm-rhel: mirror+

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-162798	0	None	None	None	2023-07-19 19:29:55 UTC
Red Hat Product Errata	RHBA-2023:5260	0	None	None	None	2023-09-19 14:35:47 UTC

Comment 1 Aashish Radhakrishnan 2023-07-24 20:13:47 UTC

Pre-verification:
brew build - https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2605167

aaradhak@fedora ~/rhcos/rhcos-4.12 $ cosa run --qemu-image rhcos-412.86.202305030814-0-qemu.x86_64.qcow2 --qemu-size '+3T' --qemu-memory 8192 -B reprovision-xfs.bu
COREOS_ASSEMBLER_CONTAINER=quay.io/coreos-assembler/coreos-assembler:rhcos-4.12
COREOS_ASSEMBLER_ADD_CERTS=y
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/rhcos/rhcos-4.12:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa -v=/etc/pki/ca-trust:/etc/pki/ca-trust:ro quay.io/coreos-assembler/coreos-assembler:rhcos-4.12 run --qemu-image rhcos-412.86.202305030814-0-qemu.x86_64.qcow2 --qemu-size +3T --qemu-memory 8192 -B reprovision-xfs.bu
Red Hat Enterprise Linux CoreOS 412.86.202305030814-0
  Part of OpenShift 4.12, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.12/architecture/architecture-rhcos.html

---
Last login: Mon Jul 24 19:28:29 2023
[core@cosa-devsh ~]$ xfs_info / |grep agcount=
meta-data=/dev/vda4              isize=512    agcount=4, agsize=187904768 blks
[core@cosa-devsh ~]$ sudo -i
[root@cosa-devsh ~]# alias verify-ostree="podman run --rm -ti --pull=newer --privileged -v /:/rootfs --net=none quay.io/cgwalters/ostree-ext-dev provisional-repair repair --sysroot /rootfs/sysroot --dry-run"
[root@cosa-devsh ~]# verify-ostree 
Trying to pull quay.io/cgwalters/ostree-ext-dev:latest...
Getting image source signatures
Copying blob 19d2bba2169e done  
Copying blob ad5077952f52 done  
Copying blob dbc9fb14b970 done  
Copying config 991ff7104a done  
Writing manifest to image destination
Storing signatures
Running in container, assuming we can remount /rootfs/sysroot writable
OK no derived commits found.
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
'''
'''
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 744866966  Available: 744866966
Inodes: Total: 300647616  Free: 300569832
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 25504 objects with > 32 bit inode numbers and 23380 objects with <= 32 bit inode numbers
warning: 1187 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
  inode clashes: 27
  unknown:       0
  ok:            2109

warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled
[root@cosa-devsh ~]# rpm -qa ostree
ostree-2022.2-5.el8.x86_64
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
                   Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132
                  Version: 38.20230625.3.0 (2023-07-24T19:39:17Z)
                     Diff: 337 upgraded, 13 downgraded, 153 removed, 76 added

* 2aa485c7967e1b147f05d41a773d725878989c826b3b3c8e927585d5b388d9ca
                  Version: 412.86.202305030814-0 (2023-05-03T08:17:34Z)
[root@cosa-devsh ~]# rpm-ostree usroverlay
Development mode enabled.  A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.
[root@cosa-devsh ~]# curl -L -k --fail --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/ostree/2022.2/7.el8_8/x86_64/ostree-{,libs-,grub2-}2022.2-7.el8_8.x86_64.rpm                                                   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  252k  100  252k    0     0   449k      0 --:--:-- --:--:-- --:--:--  449k
100  445k  100  445k    0     0   758k      0 --:--:-- --:--:-- --:--:--  758k
100 25712  100 25712    0     0   343k      0 --:--:-- --:--:-- --:--:--  343k
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm -Uvh ostree-*
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:ostree-libs-2022.2-7.el8_8       ################################# [ 17%]
   2:ostree-2022.2-7.el8_8            ################################# [ 33%]
   3:ostree-grub2-2022.2-7.el8_8      ################################# [ 50%]
Cleaning up / removing...
   4:ostree-grub2-2022.2-5.el8        ################################# [ 67%]
   5:ostree-2022.2-5.el8              ################################# [ 83%]
   6:ostree-libs-2022.2-5.el8         ################################# [100%]
[/usr/lib/tmpfiles.d/pkg-dbus-daemon.conf:1] Duplicate line for path "/var/lib/dbus", ignoring.
[/usr/lib/tmpfiles.d/tmp.conf:12] Duplicate line for path "/var/tmp", ignoring.
[/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
[/usr/lib/tmpfiles.d/var.conf:19] Duplicate line for path "/var/cache", ignoring.
[/usr/lib/tmpfiles.d/var.conf:21] Duplicate line for path "/var/lib", ignoring.
[/usr/lib/tmpfiles.d/var.conf:23] Duplicate line for path "/var/spool", ignoring.
"/home" already exists and is not a directory.
"/srv" already exists and is not a directory.
[root@cosa-devsh ~]# systemctl restart rpm-ostreed
[root@cosa-devsh ~]# rpm -qa ostree
ostree-2022.2-7.el8_8.x86_64
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 744864983  Available: 744864983
Inodes: Total: 300647616  Free: 300569683
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 25504 objects with > 32 bit inode numbers and 23380 objects with <= 32 bit inode numbers
warning: 1187 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_074b4ac01d1e6fd65aed6f9c9d81fd0bf0158a30b02a91c5cf674f7f013bc74f
Image has 0 derived layers
warning: Found corrupted merge commit
  inode clashes: 27
  unknown:       0
  ok:            2109

warning: Corrupted image docker://quay.io/fedora/fedora-coreos:stable
error: Found potential corruption, dry-run mode enabled
[root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo
error: Pruning docker://quay.io/fedora/fedora-coreos:stable: unlink(ostree/container/image/docker_3A__2F__2F_quay_2E_io/fedora/fedora-coreos_3A_stable): Read-only file system
[root@cosa-devsh ~]# mount -o remount,rw /sysroot
[root@cosa-devsh ~]# ostree container image list --repo=/ostree/repo | xargs ostree container image remove --repo=/ostree/repo
Removed images: 1 layers: 0
[root@cosa-devsh ~]# rpm-ostree status 
State: idle
Deployments:
  ostree-unverified-registry:quay.io/fedora/fedora-coreos:stable
                   Digest: sha256:f01cbb6cb39878550a699468463809b7475631907af9dbae7e24c9a439dae132
                  Version: 38.20230625.3.0 (2023-07-24T19:39:17Z)
                     Diff: 337 upgraded, 13 downgraded, 153 removed, 76 added

* 2aa485c7967e1b147f05d41a773d725878989c826b3b3c8e927585d5b388d9ca
                  Version: 412.86.202305030814-0 (2023-05-03T08:17:34Z)
                 Unlocked: development
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/fedora/fedora-coreos:testing-devel
'''
'''
[root@cosa-devsh ~]# verify-ostree 
Running in container, assuming we can remount /rootfs/sysroot writable
Found 1 derived commits
Backing filesystem information:
  File: "."
    ID: fc0400000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 751252071  Free: 744864171  Available: 744864171
Inodes: Total: 300647616  Free: 300569627
Attempting analysis of ostree state for files that may be incorrectly linked.
For more information, see https://github.com/ostreedev/ostree/pull/2874/commits/de6fddc6adee09a93901243dc7074090828a1912

Gathering inodes for ostree objects...
Analyzed 25220 objects with > 32 bit inode numbers and 23696 objects with <= 32 bit inode numbers
warning: 1250 potentially colliding inodes found
Verifying 1 ostree-container images
Verifying with base ostree layer ostree/container/blob/sha256_3A_4258bb33aab6c4f5ef257854f1a22d03b4f6387cabbf5b4eaa956d9aca7d50ec
Image has 0 derived layers
OK image docker://quay.io/fedora/fedora-coreos:testing-devel (verified=11

OK no corrupted images found

Comment 5 Aashish Radhakrishnan 2023-07-28 16:35:43 UTC

Verification done based on comment 1

Comment 11 errata-xmlrpc 2023-09-19 14:35:44 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ostree bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:5260

Note You need to log in before you can comment on or make changes to this bug.