Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2105414

Summary:	Include latest ostree-container support
Product:	Red Hat Enterprise Linux 8	Reporter:	Colin Walters <walters>
Component:	rpm-ostree	Assignee:	Colin Walters <walters>
Status:	CLOSED ERRATA	QA Contact:	HuijingHei <hhei>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	8.7	CC:	hhei, mnguyen
Target Milestone:	rc	Keywords:	Rebase, Triaged
Target Release:	---	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rpm-ostree-2022.10.86.gd8f0c67a-3.el8	Doc Type:	Rebase: Bug Fixes and Enhancements
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-11-08 09:49:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2095528
Bug Blocks:
Deadline:	2022-08-08

Description Colin Walters 2022-07-08 18:17:24 UTC

This will obsolete https://bugzilla.redhat.com/show_bug.cgi?id=2095528

This is the RHEL8.7 version of the RHEL9 bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2105405
(We'll update the rhel8 branch to 2022.11 + reverts)

Comment 1 Colin Walters 2022-07-19 18:23:17 UTC

We need the latest ostree-container fixes for OCP 4.12.

Comment 2 Colin Walters 2022-07-19 18:55:42 UTC

Specifically after this lands,

$ rpm-ostree rebase --experimental ostree-unverified-registry:registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest

should work. 

(Remember to do mkdir -p /run/ostree and write your pull secret to /run/ostree/auth.json )

Comment 3 HuijingHei 2022-07-20 14:02:58 UTC

Start 412.86.202207190921-0 qcow2 image, upgrade to rpm-ostree-2022.10.86.gd8f0c67a-3.el8.x86_64 (and librhsm-0.0.3-4.el8.x86_64.rpm), setup /run/ostree/auth.json, run rebase command which works, reboot and check the new rebase version


Steps:
1) Upgrade rpm-ostree to brew version and reboot
$ sudo rpm-ostree install http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/librhsm/0.0.3/4.el8/x86_64/librhsm-0.0.3-4.el8.x86_64.rpm

sudo rpm-ostree override replace http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.86.gd8f0c67a/3.el8/x86_64/rpm-ostree-2022.10.86.gd8f0c67a-3.el8.x86_64.rpm http://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.86.gd8f0c67a/3.el8/x86_64/rpm-ostree-libs-2022.10.86.gd8f0c67a-3.el8.x86_64.rpm

$ sudo systemctl reboot


2) Setup /run/ostree/auth.json, run rebase command
$ rpm -q rpm-ostree
rpm-ostree-2022.10.86.gd8f0c67a-3.el8.x86_64

$ sudo mkdir -p /run/ostree
$ oc login --token=xx --server=https://api.ci.l2s4.p1.openshiftapps.com:6443

$ oc registry login --to txt
$ sudo cp txt /run/ostree/auth.json

$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest
Pulling manifest: ostree-unverified-image:docker://registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest
Importing: ostree-unverified-image:docker://registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest (digest: sha256:92f2f9932d95ad5eaf32e14173f32fd2e35dd34bfb39c3e7be8c922d6faed35b)
ostree chunk layers stored: 0 needed: 1 (1.1 GB)
custom layers stored: 0 needed: 1 (42 bytes)
Fetching ostree chunk sha256:abb52211ddad (1.1 GB)
Fetched ostree chunk sha256:abb52211ddad
Fetching layer sha256:4ca545ee6d5d (42 bytes)
Fetched layer sha256:4ca545ee6d5d
Checking out tree 1e384cd... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 2 overrides and 1 overlay
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Downgraded:
  coreos-installer 0.15.0-2.rhaos4.12.el8 -> 0.15.0-1.rhaos4.12.el8
  coreos-installer-bootinfra 0.15.0-2.rhaos4.12.el8 -> 0.15.0-1.rhaos4.12.el8
  openshift-clients 4.12.0-202207190537.p0.ga1119e4.assembly.stream.el8 -> 4.12.0-202207132056.p0.gddfe9e9.assembly.stream.el8
Changes queued for next boot. Run "systemctl reboot" to start a reboot


3) After reboot and check the result
$ sudo systemctl reboot
$ rpm-ostree status
State: idle
Deployments:
● ostree-unverified-registry:registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest
                   Digest: sha256:92f2f9932d95ad5eaf32e14173f32fd2e35dd34bfb39c3e7be8c922d6faed35b
                Timestamp: 2022-07-20T13:47:31Z
           LocalOverrides: rpm-ostree-libs rpm-ostree 2022.2-2.el8 -> 2022.10.86.gd8f0c67a-3.el8
            LocalPackages: librhsm-0.0.3-4.el8.x86_64

  f0a3db7803989213abb2e9f2900c3c43aa14e8c1ab35a2fe02fa6a4dc37a49c4
                  Version: 412.86.202207190921-0 (2022-07-19T09:24:41Z)
           LocalOverrides: rpm-ostree-libs rpm-ostree 2022.2-2.el8 -> 2022.10.86.gd8f0c67a-3.el8
            LocalPackages: librhsm-0.0.3-4.el8.x86_64

Comment 4 Colin Walters 2022-07-20 14:50:46 UTC

I think this is sufficient to mark this VERIFIED; per discussion we're going to be backfilling more automated CI here and we have time to address any regressions.

Comment 5 HuijingHei 2022-07-20 15:02:42 UTC

Thanks Colin for your confirmation!

Based on last step in Comment 3, after rebase, run `rpm-ostree db list` and get error.
IMU, this should work and will list installed packages. If my understanding is wrong, correct me freely

$ rpm-ostree db list 92f2f9932d95ad5eaf32e14173f32fd2e35dd34bfb39c3e7be8c922d6faed35b
ostree commit: 92f2f9932d95ad5eaf32e14173f32fd2e35dd34bfb39c3e7be8c922d6faed35b
error: Loading package list: No such metadata object 92f2f9932d95ad5eaf32e14173f32fd2e35dd34bfb39c3e7be8c922d6faed35b.commit

Comment 6 Colin Walters 2022-07-20 15:09:21 UTC

There's two (sha256) digests for things here - the ostree commit and the container image.  Unfortunately right now, `rpm-ostree db list` only understands ostree commits, not container images.

You can get the ostree commit by looking at `rpm-ostree status -v` - then pass *that* sha256 to `db diff`.

Comment 9 HuijingHei 2022-08-17 07:36:57 UTC

Change status to MODIFIED according to errata workflow

Comment 13 HuijingHei 2022-08-18 10:04:56 UTC

Test with rpm-ostree-2022.10.86.gd8f0c67a-3.el8.x86_64 in 412.86.202208161753-0, result is passed

[root@cosa-devsh ~]# rpm -q rpm-ostree
rpm-ostree-2022.10.86.gd8f0c67a-3.el8.x86_64
[core@cosa-devsh ~]$ sudo rpm-ostree rebase --experimental ostree-unverified-registry:registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest
Pulling manifest: ostree-unverified-image:docker://registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest
Importing: ostree-unverified-image:docker://registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest (digest: sha256:9a0eda5e820cbd8fd25db9bd9911c89d7e7685676950d3936f2412d323828b55)
ostree chunk layers stored: 0 needed: 1 (1.1 GB)
custom layers stored: 0 needed: 1 (35 bytes)
Fetching ostree chunk sha256:2478135e612a (1.1 GB)
Fetched ostree chunk sha256:2478135e612a
Fetching layer sha256:acab339ca1e8 (35 bytes)
Fetched layer sha256:acab339ca1e8
Staging deployment... done
Downgraded:
  cri-o 1.25.0-32.rhaos4.12.gita73a2d7.el8 -> 1.25.0-28.rhaos4.12.git822b484.el8
  cri-tools 1.24.2-5.el8 -> 1.24.2-4.el8
  openshift-clients 4.12.0-202208101826.p0.g1e33c4a.assembly.stream.el8 -> 4.12.0-202208031044.p0.g3c85519.assembly.stream.el8
  openshift-hyperkube 4.12.0-202208161547.p0.ged93380.assembly.stream.el8 -> 4.12.0-202207292135.p0.ga9d6306.assembly.stream.el8
  toolbox 0.1.0-1.rhaos4.12.el8 -> 0.0.9-1.rhaos4.12.el8
Changes queued for next boot. Run "systemctl reboot" to start a reboot

[root@cosa-devsh ~]# rpm-ostree status -v
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree-unverified-registry:registry.ci.openshift.org/rhcos-devel/rhel-coreos:latest (index: 0)
                   Digest: sha256:9a0eda5e820cbd8fd25db9bd9911c89d7e7685676950d3936f2412d323828b55
                Timestamp: 2022-08-18T08:31:28Z
                   Staged: no
                StateRoot: rhcos

  3293fabf0586fc80478d79fb82b8e3051707f06fb5e023044f36ef34a84437f1 (index: 1)
                  Version: 412.86.202208161753-0 (2022-08-16T17:56:27Z)
                   Commit: 3293fabf0586fc80478d79fb82b8e3051707f06fb5e023044f36ef34a84437f1
                           ├─ rhel-8-appstream (2022-08-16T17:15:20Z)
                           ├─ rhel-8-baseos (2022-08-08T14:05:51Z)
                           ├─ rhel-8-fast-datapath (2022-08-01T13:46:17Z)
                           └─ rhel-8-server-ose (2022-08-16T17:14:41Z)
                StateRoot: rhcos

Comment 15 errata-xmlrpc 2022-11-08 09:49:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rpm-ostree bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7612