Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2177225

Summary: intermittent core dump in hex2bin (librpm.so.8)
Product: Red Hat Enterprise Linux 8 Reporter: Colin Walters <walters>
Component: rpm-ostreeAssignee: RHCOS SST <rhcos-sst>
Status: CLOSED ERRATA QA Contact: RHCOS SST QE <rhcos-sst-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.8CC: aaradhak, dornelas, hhei, jwboyer, swm-qe
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2177088
: 2177787 2177788 (view as bug list) Environment:
Last Closed: 2023-05-16 09:59:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2177787, 2177788    
Deadline: 2023-06-12   

Description Colin Walters 2023-03-10 13:49:12 UTC 
+++ This bug was initially created as a clone of Bug #2177088 +++

See that bug for more info.  Basically, patch rpm-ostree to stop looking at RPM IMA signatures unless explicitly enabled, which will work around a librpm bug.
Comment 5 Colin Walters 2023-03-13 13:49:22 UTC 
To verify this bug, see https://github.com/coreos/rpm-ostree/pull/4333#issuecomment-1464218998

Basically:

- Provision rhel8 coreos (or edge probably too)
- *Queue* (but do not reboot into) an upgrade to rhel9 coreos xref https://github.com/openshift/os/blob/master/docs/faq.md#q-how-do-i-upgrade-the-os-manually-or-outside-of-a-cluster
- Run rpm-ostree install on a RHEL9 RPM that includes IMA signatures (I think this may be all of them?) but definitely includes kernel-rt; I used the edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm inside the extensions container

On versions before this fix, it may or may not crash.  So you'll need to do it in a loop:

$ while rpm-ostree reset -oil && rpm-ostree install <rhel9 rpm>; do systemctl stop rpm-ostreed; done

On versions with this fix, it should successfully loop indefinitely.  In my testing without this fix I got a crash pretty quickly, on the order of 1/8 runs or so.
Comment 8 Colin Walters 2023-03-13 19:18:32 UTC 
Er, sorry I just remembered that for OCP 4.12 we'd been tagging in the 8.8 builds.  On one hand, Edge could hit this.  On the other hand, I think they discourage package layering.  So...I'd say let's lift the zstream from this - if anyone else does hit this we can backport to earlier RHEL8 (or they can upgrade to 8.8 first).
Comment 12 HuijingHei 2023-03-14 10:08:48 UTC 
Hi Colin, what do you mean `for OCP 4.12 we'd been tagging in the 8.8 builds`?

Test with 412.86.202303011010-0 which includes rpm-ostree-2022.10.112.g3d0ac35b-3.el8.x86_64 (without the fixed patch), running the loop install using the following steps for 30 mins, only get 1 failed segfault

===============================
Mar 14 08:45:15 cosa-devsh rpm-ostree[11542]: Initiated txn UpdateDeployment for client(id:cli dbus:1.218 unit:session-2.scope uid:0): /org/projectatomic/rpmo
stree1/rhcos
Mar 14 08:45:15 cosa-devsh rpm-ostree[11542]: Process [pid: 11556 uid: 0 unit: session-2.scope] connected to transaction progress
Mar 14 08:45:15 cosa-devsh rpm-ostree[11542]: Librepo version: 1.14.2 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0)
Mar 14 08:45:15 cosa-devsh rpm-ostree[11542]: warning: Found SQLITE rpmdb.sqlite database while attempting bdb backend: using sqlite backend.
Mar 14 08:45:16 cosa-devsh rpm-ostree[11563]: warning: Found SQLITE rpmdb.sqlite database while attempting bdb backend: using sqlite backend.
Mar 14 08:45:17 cosa-devsh rpm-ostree[11542]: Preparing pkg txn; enabled repos: [] solvables: 0
Mar 14 08:45:17 cosa-devsh rpm-ostree[11542]: Relabeled 0/1 pkgs
Mar 14 08:45:17 cosa-devsh kernel: pool[11548]: segfault at 7fab2fee1000 ip 00007fab4d3870f2 sp 00007fab41022c50 error 4 in librpm.so.8.2.0[7fab4d345000+77000]
Mar 14 08:45:17 cosa-devsh kernel: Code: 76 44 44 8d 41 9f 41 80 f8 05 76 5c 44 8d 41 bf 41 80 f8 05 77 05 83 e9 37 09 ca 48 83 c0 02 88 16 48 83 c6 01 48 39 
f8 74 2f <0f> b6 08 8d 51 d0 80 fa 09 77 a3 c1 e2 04 0f b6 48 01 44 8d 41 d0
===============================

Reproduce steps:
$ cosa run --qemu-image rhcos-412.86.202303011010-0-qemu.x86_64.qcow2 -m 4096

[root@cosa-devsh ~]# rpm -q rpm-ostree
rpm-ostree-2022.10.112.g3d0ac35b-3.el8.x86_64

# *Queue* (but do not reboot into) an upgrade to rhel9 coreos
[root@cosa-devsh ~]# vi /etc/ostree/auth.json
rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd299b2bf3cc98fb70907f152b4281633064fe33527b5d6a42ddc418ff00eec1

[root@cosa-devsh ~]# curl -LO https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm
[root@cosa-devsh ~]# rpm-ostree install ./edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm

[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd299b2bf3cc98fb70907f152b4281633064fe33527b5d6a42ddc418ff00eec1
                   Digest: sha256:cd299b2bf3cc98fb70907f152b4281633064fe33527b5d6a42ddc418ff00eec1
                  Version: 413.92.202303061740-0 (2023-03-14T07:41:27Z)
                     Diff: 445 upgraded, 7 downgraded, 51 removed, 60 added
            LocalPackages: edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch

* 336c2b60e9f33b292370f4dcbb4464c89a46c838f296f63470e3b7c21746c168
                  Version: 412.86.202303011010-0 (2023-03-01T10:13:30Z)

# run loop install edk2-ovmf
[root@cosa-devsh ~]# while rpm-ostree reset -oil && rpm-ostree install ./edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm; do systemctl stop rpm-ostreed; done
Comment 13 HuijingHei 2023-03-14 10:57:23 UTC 
One more question, should we verify this on internal build rhcos 4.12(or 4.13?) using RHEL8.8 or just upgrade rpm-ostree on 412-86?
Comment 14 Colin Walters 2023-03-14 11:59:59 UTC 
We don't currently plan to ship RHEL 8.8 content to customers for RHCOS, so let's test by upgrading just rpm-ostree (via e.g. rpm-ostree usroverlay && rpm -Uvh https://.../rpm-ostree.rpm && systemctl restart rpm-ostreed), but doing an override-replace and reboot or apply-live --allow-replacement also works.
Comment 15 HuijingHei 2023-03-15 06:32:37 UTC 
Thanks Colin for the confirmation, so we need to do testing after upgrade rpm-ostree and related packages on 412-86.

I believe Aashish is working on the verification, and mark the steps which might be helpful.

[root@cosa-devsh ~]# rpm-ostree override replace rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
[root@cosa-devsh ~]# rpm-ostree ex apply-live --allow-replacement
NOTICE: Experimental commands are subject to change.
Computing /etc diff to preserve... done
Updating /usr... done
Updating /etc... done
Running systemd-tmpfiles for /run and /var... done
Upgraded:
  rpm-ostree 2022.10.112.g3d0ac35b-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
  rpm-ostree-libs 2022.10.112.g3d0ac35b-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
Successfully updated running filesystem tree; some services may need to be restarted.

[root@cosa-devsh ~]# rpm -qa | grep rpm-ostree
rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64
rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
Comment 16 Aashish Radhakrishnan 2023-03-15 13:03:50 UTC 
I have been hitting the dependency error while executing the override

[root@cosa-devsh ~]# rpm-ostree override replace rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
Checking out tree 7dc0bd3... done
Rewriting rpmdb back to host bdb... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
error: Could not depsolve transaction; 2 problems detected:
 Problem 1: conflicting requests
  - nothing provides libcrypto.so.1.1()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libjson-c.so.4()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libjson-c.so.4(JSONC_0.14)(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librhsm.so.0()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librpm.so.8()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librpmio.so.8()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libssl.so.1.1()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libssl.so.1.1(OPENSSL_1_1_0)(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
 Problem 2: conflicting requests
  - nothing provides librpm.so.8()(64bit) needed by rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librpmio.so.8()(64bit) needed by rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64
Comment 17 Colin Walters 2023-03-15 16:37:08 UTC 
When reporting problems like this please add e.g. `rpm-ostree status` to show which starting point you're using.  In this case I'm guessing you are using an older RHEL/RHCOS perhaps based on 8.4?

I did this, starting from the current official qemu image for 4.12:

```
walters@toolbox /v/s/w/m/rhcos> curl -L https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.12/4.12.3/rhcos-qemu.x86_64.qcow2.gz | gunzip > rhcos-4.12.3-qemu.qcow2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1150M  100 1150M    0     0  32.6M      0  0:00:35  0:00:35 --:--:-- 30.1M
walters@toolbox /v/s/w/m/rhcos> cosa run --qemu-image rhcos-4.12.3-qemu.qcow2 --qemu-memory 4096
Red Hat Enterprise Linux CoreOS 412.86.202301311551-0
  Part of OpenShift 4.12, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.12/architecture/architecture-rhcos.html

---
Last login: Wed Mar 15 15:45:14 2023
[core@cosa-devsh ~]$ sudo su -
[root@cosa-devsh ~]# rpm-ostree override replace https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.115.g15eba7b1/2.el8_8/x86_64/rpm-ostree-{libs-,}2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
Downloading https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.115.g15eba7b1/2.el8_8/x86_64/rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm...failed
error: Preparing D-Bus arguments: [60] Peer certificate cannot be authenticated with given CA certificates (SSL certificate problem: unable to get local issuer certificate)
[root@cosa-devsh ~]# curl -k --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.115.g15eba7b1/2.el8_8/x86_64/rpm-ostree-{libs-,}2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54376  100 54376    0     0   188k      0 --:--:-- --:--:-- --:--:--  187k
100 3342k  100 3342k    0     0  10.6M      0 --:--:-- --:--:-- --:--:-- 10.6M
[root@cosa-devsh ~]# rpm-ostree override replace *.rpm
Checking out tree db83d20... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 2 overrides
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
  rpm-ostree 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
  rpm-ostree-libs 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
Use "rpm-ostree override reset" to undo overrides
Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# 

```
Comment 18 Aashish Radhakrishnan 2023-03-15 17:48:28 UTC 
I followed the same steps as in comment 17 , the rpms got upgraded but the dependency error is now seen at the rebase queue. I did the rpms install/upgrade before the rebase queue. 

 

aaradhak@fedora ~/rhcos/rhcos-4.12 $ curl -L https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.12/4.12.3/rhcos-qemu.x86_64.qcow2.gz | gunzip > rhcos-4.12.3-qemu.qcow2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1150M  100 1150M    0     0  4205k      0  0:04:40  0:04:40 --:--:-- 4463k
aaradhak@fedora ~/rhcos/rhcos-4.12 $ 
aaradhak@fedora ~/rhcos/rhcos-4.12 $ 
aaradhak@fedora ~/rhcos/rhcos-4.12 $ 
aaradhak@fedora ~/rhcos/rhcos-4.12 $ cosac run --qemu-image rhcos-4.12.3-qemu.qcow2 --qemu-memory 4096
COREOS_ASSEMBLER_CONTAINER=quay.io/coreos-assembler/coreos-assembler:rhcos-4.12
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/rhcos/rhcos-4.12:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa quay.io/coreos-assembler/coreos-assembler:rhcos-4.12 run --qemu-image rhcos-4.12.3-qemu.qcow2 --qemu-memory 4096
Red Hat Enterprise Linux CoreOS 412.86.202301311551-0
  Part of OpenShift 4.12, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.12/architecture/architecture-rhcos.html

---
Last login: Wed Mar 15 17:28:51 2023
[core@cosa-devsh ~]$ sudo su - 
[root@cosa-devsh ~]# curl -k --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.115.g15eba7b1/2.el8_8/x86_64/rpm-ostree-{libs-,}2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54376  100 54376    0     0   124k      0 --:--:-- --:--:-- --:--:--  123k
100 3342k  100 3342k    0     0  1142k      0  0:00:02  0:00:02 --:--:-- 1195k
[root@cosa-devsh ~]# ls
rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm  rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree override replace *.rpm
Checking out tree db83d20... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 2 overrides
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Upgraded:
  rpm-ostree 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
  rpm-ostree-libs 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
Use "rpm-ostree override reset" to undo overrides
Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  db83d20cf09a263777fcca78594b16da00af8acc245d29cc2a1344abc3f0dac2
                  Version: 412.86.202301311551-0 (2023-01-31T15:54:05Z)
                     Diff: 2 upgraded
           LocalOverrides: rpm-ostree rpm-ostree-libs 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8

* db83d20cf09a263777fcca78594b16da00af8acc245d29cc2a1344abc3f0dac2
                  Version: 412.86.202301311551-0 (2023-01-31T15:54:05Z)
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm -q rpm-ostree
rpm-ostree-2022.10.99.g0049dbdd-3.el8.x86_64
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# vi /etc/ostree/auth.json
[root@cosa-devsh ~]# curl -LO https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5396k  100 5396k    0     0  4269k      0  0:00:01  0:00:01 --:--:-- 4266k
[root@cosa-devsh ~]# rpm-ostree install ./edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm
Checking out tree db83d20... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 2 overrides and 1 overlay
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 68.3 MB (pkgcache branches: 0)
Upgraded:
  rpm-ostree 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
  rpm-ostree-libs 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
Added:
  edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  db83d20cf09a263777fcca78594b16da00af8acc245d29cc2a1344abc3f0dac2
                  Version: 412.86.202301311551-0 (2023-01-31T15:54:05Z)
                     Diff: 2 upgraded, 1 added
           LocalOverrides: rpm-ostree rpm-ostree-libs 2022.10.99.g0049dbdd-3.el8 -> 2022.10.115.g15eba7b1-2.el8_8
            LocalPackages: edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch

* db83d20cf09a263777fcca78594b16da00af8acc245d29cc2a1344abc3f0dac2
                  Version: 412.86.202301311551-0 (2023-01-31T15:54:05Z)
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev:4.13-9.2
Pulling manifest: ostree-unverified-image:docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev:4.13-9.2
Importing: ostree-unverified-image:docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev:4.13-9.2 (digest: sha256:cd299b2bf3cc98fb70907f152b4281633064fe33527b5d6a42ddc418ff00eec1)
ostree chunk layers stored: 0 needed: 51 (1.0 GB)
Fetching ostree chunk sha256:4a3ef361369a (115.5 MB)
Fetched ostree chunk sha256:4a3ef361369a
Fetching ostree chunk sha256:134c96a262c8 (297.6 MB)
Fetched ostree chunk sha256:134c96a262c8
Fetching ostree chunk sha256:20fb0580f41e (49.1 MB)
Fetched ostree chunk sha256:20fb0580f41e
Fetching ostree chunk sha256:5781d400b965 (56.9 MB)
Fetched ostree chunk sha256:5781d400b965
Fetching ostree chunk sha256:48863e3723fa (29.3 MB)
Fetched ostree chunk sha256:48863e3723fa
Fetching ostree chunk sha256:35773442207f (83.4 MB)
Fetched ostree chunk sha256:35773442207f
Fetching ostree chunk sha256:b6972d6eea8a (23.1 MB)
Fetched ostree chunk sha256:b6972d6eea8a
Fetching ostree chunk sha256:f992c085642c (18.5 MB)
Fetched ostree chunk sha256:f992c085642c
Fetching ostree chunk sha256:6673545b8252 (12.0 MB)
Fetched ostree chunk sha256:6673545b8252
Fetching ostree chunk sha256:6d873a444df8 (34.9 MB)
Fetched ostree chunk sha256:6d873a444df8
Fetching ostree chunk sha256:ffcf672093a1 (13.6 MB)
Fetched ostree chunk sha256:ffcf672093a1
Fetching ostree chunk sha256:629979efc71a (12.7 MB)
Fetched ostree chunk sha256:629979efc71a
Fetching ostree chunk sha256:796f3638b1a4 (32.9 MB)
Fetched ostree chunk sha256:796f3638b1a4
Fetching ostree chunk sha256:b267bbfbb036 (11.0 MB)
Fetched ostree chunk sha256:b267bbfbb036
Fetching ostree chunk sha256:eb37919270e0 (10.1 MB)
Fetched ostree chunk sha256:eb37919270e0
Fetching ostree chunk sha256:d4faf6a8f674 (8.5 MB)
Fetched ostree chunk sha256:d4faf6a8f674
Fetching ostree chunk sha256:1c80e1db3e3e (6.8 MB)
Fetched ostree chunk sha256:1c80e1db3e3e
Fetching ostree chunk sha256:db876d482c2d (6.7 MB)
Fetched ostree chunk sha256:db876d482c2d
Fetching ostree chunk sha256:2abd170c6803 (8.3 MB)
Fetched ostree chunk sha256:2abd170c6803
Fetching ostree chunk sha256:45f4d300a069 (13.5 MB)
Fetched ostree chunk sha256:45f4d300a069
Fetching ostree chunk sha256:184b46863e24 (5.2 MB)
Fetched ostree chunk sha256:184b46863e24
Fetching ostree chunk sha256:80d89e74e57d (4.4 MB)
Fetched ostree chunk sha256:80d89e74e57d
Fetching ostree chunk sha256:cb14fca223e6 (4.4 MB)
Fetched ostree chunk sha256:cb14fca223e6
Fetching ostree chunk sha256:be926691d4c3 (3.7 MB)
Fetched ostree chunk sha256:be926691d4c3
Fetching ostree chunk sha256:0bb93da2154a (4.0 MB)
Fetched ostree chunk sha256:0bb93da2154a
Fetching ostree chunk sha256:916e890fb94d (4.0 MB)
Fetched ostree chunk sha256:916e890fb94d
Fetching ostree chunk sha256:b8818bdbd465 (3.8 MB)
Fetched ostree chunk sha256:b8818bdbd465
Fetching ostree chunk sha256:c4db9cbd41fd (2.3 MB)
Fetched ostree chunk sha256:c4db9cbd41fd
Fetching ostree chunk sha256:c56feee022f3 (3.3 MB)
Fetched ostree chunk sha256:c56feee022f3
Fetching ostree chunk sha256:5f379f0decce (2.6 MB)
Fetched ostree chunk sha256:5f379f0decce
Fetching ostree chunk sha256:ba2ced78b90b (3.1 MB)
Fetched ostree chunk sha256:ba2ced78b90b
Fetching ostree chunk sha256:85faae1593b1 (3.6 MB)
Fetched ostree chunk sha256:85faae1593b1
Fetching ostree chunk sha256:782905a10a37 (4.5 MB)
Fetched ostree chunk sha256:782905a10a37
Fetching ostree chunk sha256:12b01c2762be (3.0 MB)
Fetched ostree chunk sha256:12b01c2762be
Fetching ostree chunk sha256:21b0b5ccae72 (2.3 MB)
Fetched ostree chunk sha256:21b0b5ccae72
Fetching ostree chunk sha256:7d48d3b7f9c3 (2.8 MB)
Fetched ostree chunk sha256:7d48d3b7f9c3
Fetching ostree chunk sha256:1b1f449648c8 (3.6 MB)
Fetched ostree chunk sha256:1b1f449648c8
Fetching ostree chunk sha256:7f1f9f5d6f81 (3.4 MB)
Fetched ostree chunk sha256:7f1f9f5d6f81
Fetching ostree chunk sha256:cd31543351ed (3.2 MB)
Fetched ostree chunk sha256:cd31543351ed
Fetching ostree chunk sha256:a35b5d4cf4fa (3.1 MB)
Fetched ostree chunk sha256:a35b5d4cf4fa
Fetching ostree chunk sha256:41a6b463763b (782.4 kB)
Fetched ostree chunk sha256:41a6b463763b
Fetching ostree chunk sha256:59f16e0c8229 (7.4 MB)
Fetched ostree chunk sha256:59f16e0c8229
Fetching ostree chunk sha256:f27c71d7ce96 (2.5 MB)
Fetched ostree chunk sha256:f27c71d7ce96
Fetching ostree chunk sha256:373ac11d3c10 (2.9 MB)
Fetched ostree chunk sha256:373ac11d3c10
Fetching ostree chunk sha256:067a388b22c6 (2.8 MB)
Fetched ostree chunk sha256:067a388b22c6
Fetching ostree chunk sha256:54dcdb987d76 (2.9 MB)
Fetched ostree chunk sha256:54dcdb987d76
Fetching ostree chunk sha256:1f8593495f9b (2.5 MB)
Fetched ostree chunk sha256:1f8593495f9b
Fetching ostree chunk sha256:cb7679789b56 (2.2 MB)
Fetched ostree chunk sha256:cb7679789b56
Fetching ostree chunk sha256:1ad793acba28 (2.7 MB)
Fetched ostree chunk sha256:1ad793acba28
Fetching ostree chunk sha256:859a10379262 (103.2 MB)
Fetched ostree chunk sha256:859a10379262
Fetching ostree chunk sha256:69f5dee936b5 (1.5 MB)
Fetched ostree chunk sha256:69f5dee936b5
Checking out tree 9bd384b... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
error: Could not depsolve transaction; 2 problems detected:
 Problem 1: conflicting requests
  - nothing provides libcrypto.so.1.1()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libjson-c.so.4()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libjson-c.so.4(JSONC_0.14)(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librhsm.so.0()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librpm.so.8()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librpmio.so.8()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libssl.so.1.1()(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides libssl.so.1.1(OPENSSL_1_1_0)(64bit) needed by rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
 Problem 2: conflicting requests
  - nothing provides librpm.so.8()(64bit) needed by rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64
  - nothing provides librpmio.so.8()(64bit) needed by rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64
Comment 19 Colin Walters 2023-03-15 17:54:31 UTC 
Right, rpm-ostree is trying to preserve the overrides across the rebase.

It's probably a lot easier to do:

rpm-ostree usroverlay
rpm -Uvh rpm-ostree*.rpm
systemctl restart rpm-ostreed

Instead of using `override replace`.

But to fix this you could also `rpm-ostree reset -ol` before queuing the rebase.
Comment 20 Aashish Radhakrishnan 2023-03-15 18:56:26 UTC 
The dependency error seems to occur with override replace even after the reboot. 

With the process in comment 19 the dependency error is not shown.


[core@cosa-devsh ~]$ sudo su -
[root@cosa-devsh ~]# curl -k --remote-name-all https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.115.g15eba7b1/2.el8_8/x86_64/rpm-ostree-{libs-,}2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54376  100 54376    0     0   111k      0 --:--:-- --:--:-- --:--:--  111k
100 3342k  100 3342k    0     0   962k      0  0:00:03  0:00:03 --:--:--  951k
[root@cosa-devsh ~]# rpm-ostree usroverlay
Development mode enabled.  A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.
[root@cosa-devsh ~]# ls
rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm  rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
[root@cosa-devsh ~]# rpm -Uvh rpm-ostree*.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:rpm-ostree-libs-2022.10.115.g15eb################################# [ 25%]
   2:rpm-ostree-2022.10.115.g15eba7b1-################################# [ 50%]
Cleaning up / removing...
   3:rpm-ostree-2022.10.99.g0049dbdd-3################################# [ 75%]
   4:rpm-ostree-libs-2022.10.99.g0049d################################# [100%]
[root@cosa-devsh ~]# systemctl restart rpm-ostreed
[root@cosa-devsh ~]# rpm -qa | grep rpm
rpm-libs-4.14.3-24.el8_6.x86_64
rpm-4.14.3-24.el8_6.x86_64
rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64
rpm-plugin-selinux-4.14.3-24.el8_6.x86_64
rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# vi /etc/ostree/auth.json
[root@cosa-devsh ~]# curl -LO https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5396k  100 5396k    0     0  1273k      0  0:00:04  0:00:04 --:--:-- 1273k
[root@cosa-devsh ~]# ls
edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm  rpm-ostree-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm  rpm-ostree-libs-2022.10.115.g15eba7b1-2.el8_8.x86_64.rpm
[root@cosa-devsh ~]# rpm-ostree install ./edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm
Checking out tree db83d20... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Added:
  edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  db83d20cf09a263777fcca78594b16da00af8acc245d29cc2a1344abc3f0dac2
                  Version: 412.86.202301311551-0 (2023-01-31T15:54:05Z)
                     Diff: 1 added
            LocalPackages: edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch

* db83d20cf09a263777fcca78594b16da00af8acc245d29cc2a1344abc3f0dac2
                  Version: 412.86.202301311551-0 (2023-01-31T15:54:05Z)
                 Unlocked: development

[root@cosa-devsh ~]# rpm-ostree reset -ol
Pinned to commit; no upgrade available

[root@cosa-devsh ~]# rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev:4.13-9.2
Pulling manifest: ostree-unverified-image:docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev:4.13-9.2
Importing: ostree-unverified-image:docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev:4.13-9.2 (digest: sha256:cd299b2bf3cc98fb70907f152b4281633064fe33527b5d6a42ddc418ff00eec1)
```
```
Changes queued for next boot. Run "systemctl reboot" to start a reboot

[root@cosa-devsh ~]# rpm-ostree status
State: idle
Deployments:
  ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev:4.13-9.2
                   Digest: sha256:cd299b2bf3cc98fb70907f152b4281633064fe33527b5d6a42ddc418ff00eec1
                  Version: 413.92.202303061740-0 (2023-03-15T18:20:50Z)
                     Diff: 445 upgraded, 7 downgraded, 51 removed, 60 added
            LocalPackages: edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch

* db83d20cf09a263777fcca78594b16da00af8acc245d29cc2a1344abc3f0dac2
                  Version: 412.86.202301311551-0 (2023-01-31T15:54:05Z)
                 Unlocked: development


# run loop install edk2-ovmf
[root@cosa-devsh ~]# while rpm-ostree reset -oil && rpm-ostree install ./edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch.rpm; do systemctl stop rpm-ostreed; done
Comment 21 Aashish Radhakrishnan 2023-03-16 13:43:26 UTC 
There were no segfaults or crashes when the loop install edk2-ovmf ran overnight.
Comment 26 errata-xmlrpc 2023-05-16 09:59:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rpm-ostree bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3086