Bug 1876194 - rpm-ostree update fails in filesystem checkout
Summary: rpm-ostree update fails in filesystem checkout
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm-ostree
Version: 32
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Jonathan Lebon
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException
Depends On: 1838691
Blocks: F33BetaFreezeException
TreeView+ depends on / blocked
 
Reported: 2020-09-06 08:09 UTC by Vadim Rutkovsky
Modified: 2020-09-23 14:08 UTC (History)
34 users (show)

Fixed In Version: rpm-ostree-2020.5-1.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1838691
Environment:
Last Closed: 2020-09-21 20:48:52 UTC
Type: Bug


Attachments (Terms of Use)

Description Vadim Rutkovsky 2020-09-06 08:09:29 UTC
(removed a few comments for clarity)

Still hitting this on Silverblue F32:

```
$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora:fedora/32/x86_64/silverblue
                   Version: 32.20200905.0 (2020-09-05T18:11:38Z)
                BaseCommit: 4150be4c9953a25b2ce8dc4197df6e2221014d23d8a162550dc0391ba13d692b
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
           LayeredPackages: gstreamer1-plugins-ugly-free libvirt libvirt-daemon-kvm make qemu-kvm tilix zsh
             LocalPackages: rpmfusion-nonfree-release-32-1.noarch rpmfusion-free-release-32-1.noarch
                    Pinned: yes

$ rpm-ostree rebase fedora/33/x86_64/testing/silverblue --uninstall=rpmfusion-free-release-32-1.noarch --uninstall=rpmfusion-nonfree-release-32-1.noarch --uninstall=gstreamer1-plugins-ugly-free
1 metadata, 0 content objects fetched; 592 B transferred in 2 seconds; 0 bytes content written
Checking out tree c9d135b... done
Enabled rpm-md repositories: updates updates-testing fedora
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z
rpm-md repo 'updates-testing' (cached); generated: 2020-09-05T21:31:13Z
rpm-md repo 'fedora' (cached); generated: 2020-09-04T10:59:23Z
Importing rpm-md... done
Resolving dependencies... done
Will download: 479 packages (244.7 MB)
Downloading from 'fedora'... done
Downloading from 'updates-testing'... done
Importing packages... done
Checking out packages... done
error: Checkout filesystem-3.14-3.fc33.x86_64: opendir(local): No such file or directory
```

libsolv from F33 requires newer RPM:
```
rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm 

Downloading 'https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm'... done!
Checking out tree 4150be4... done
Enabled rpm-md repositories: updates updates-testing fedora
rpm-md repo 'updates' (cached); generated: 2020-09-05T18:22:52Z
rpm-md repo 'updates-testing' (cached); generated: 2020-09-05T17:50:28Z
rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:36Z
Importing rpm-md... done
Resolving dependencies... done
error: Could not depsolve transaction; 1 problem detected:
 Problem: conflicting requests
  - nothing provides rpm(x86-64) >= 4.16.0 needed by libsolv-0.7.14-2.fc33.x86_64
```

+++ This bug was initially created as a clone of Bug #1838691 +++

rpm-ostree fails 

error: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory

...

--- Additional comment from Colin Walters on 2020-05-22 14:23:14 UTC ---

OK that's concerning, it's possible that something broke in rawhide related to this; handling of the `filesystem` package is special in rpm-ostree.

...

--- Additional comment from Colin Walters on 2020-05-24 23:37:10 UTC ---

OK I think what's going on here probably has something to do with the rpm switch to use the sqlite backend.

Depsolving isn't finding the "base" packages - one can see this with e.g.
`rpm-ostree install --dry-run wget`
and notice that depsolving includes `filesystem` and `bash` etc.

Trying to re-layer `filesystem` failed because it tries to unpack `/usr/local` but that's really `/var/usrlocal` which the unpacking process can't see.
(This is part of how rpm-ostree protects user data, it's very hard even if there are bugs for us to walk off and affect the real /var where e.g. the homedir is)

Going to need to add some debugging prints.

--- Additional comment from Colin Walters on 2020-05-25 20:46:34 UTC ---

https://github.com/openSUSE/libsolv/pull/386

--- Additional comment from Colin Walters on 2020-05-25 20:56:55 UTC ---

https://src.fedoraproject.org/rpms/libsolv/c/c6160aa95a361d9ac498e1586b9227e55753314d?branch=master
https://koji.fedoraproject.org/koji/taskinfo?taskID=44965714

...

--- Additional comment from Fedora Blocker Bugs Application on 2020-05-28 21:10:45 UTC ---

Proposed as a Blocker for 33-beta by Fedora user lruzicka using the blocker tracking app because:

 The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated.

--- Additional comment from Daniel Mach on 2020-06-01 11:43:05 UTC ---

Colin,
could you provide some details on why do you need the paths?

Many tools use the paths because they detect rpmdb changes,
but maybe we should provide another way as part of librpm API.
The database should remain an implementation detail, a private API.

--- Additional comment from Colin Walters on 2020-06-01 14:10:54 UTC ---

For rpm-ostree, the database must be in /usr - this has been extensively discussed before, see e.g.
http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html

I'd also like to use this opportunity of changing the backend to sqlite to *also* move the rpm database to /usr/lib/sysimage/rpm for all editions/spins.

Ideally as long as librpm supports probing per above, we wouldn't need to know the specific
path.

That said, in order for us to support transitioning from /usr/share/rpm we'd probably need to make it configurable where to place the rpmdb, which might require some support for both configuring and probing the path explicitly.

...

--- Additional comment from Colin Walters on 2020-06-03 14:07:19 UTC ---

> What can be workaround. I want to stick to Rawhide and not to rebase. How to force update?

You can use `rpm-ostree override replace` with e.g. the builds from https://koji.fedoraproject.org/koji/buildinfo?buildID=1519658
or temporarily remove your layered packages, upgrade, then re-add them.

...

--- Additional comment from Jonathan Lebon on 2020-06-05 19:36:06 UTC ---

Try:

```
$ rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm
$ reboot
---
$ rpm-ostree upgrade
```
...

--- Additional comment from Adam Williamson on 2020-06-22 20:19:21 UTC ---

So, this bug is still proposed as a blocker for F33 Beta, but I'm confused as to the current status at this point. Is there still a serious problem with ostree here, Lukas?

...

--- Additional comment from Colin Walters on 2020-06-22 22:52:28 UTC ---

This should have been fixed in libsolv as of https://src.fedoraproject.org/rpms/libsolv/c/c6160aa95a361d9ac498e1586b9227e55753314d?branch=master

I just tested the latest rpm-ostree+libsolv in 

$ rpm-ostree status -b
State: idle
BootedDeployment:
● ostree://fedora:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20200622.n.0 (2020-06-22T06:03:56Z)
                    Commit: d44b3db4d47b448adb6a19bb5cfe4f58de02e4d3960829ef695e3f3288f9630f
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
$

and things look fine.


...

Comment 1 Vadim Rutkovsky 2020-09-06 08:31:15 UTC
Workaround:
```
rpm-ostree reset
...
rpm-ostree rebase fedora/33/x86_64/testing/silverblue
...
reboot
```

That however has nuked my shell (zsh), so I had to create a new user to login and run the rebase

Comment 2 Kamil Páral 2020-09-07 14:39:46 UTC
If I understand it correctly, this got fixed in F33 (bug 1838691) but not in F32. So I'm removing the F33 proposed blocker.

Comment 3 Jonathan Lebon 2020-09-08 15:56:44 UTC
Right, this will cause issues in the f32 -> f33 rebase because f32 rpm-ostree doesn't understand the new rpmdb since it has old libsolv and old librpm, so systems with layered packages will fail to rebase.

The fix for this would be to backport the libsolv patch *and* rpm 4.16 to f32 (but make the rpmdb migration still gate on upgrading to f33), but I don't see that happening (/cc Panu in case this came up already). We'll probably have to document this on the "Common F33 bugs" page and tell users with overlays to `rpm-ostree reset` first (see https://bugzilla.redhat.com/show_bug.cgi?id=1838691#c34).

Comment 4 Colin Walters 2020-09-08 16:04:27 UTC
Yeah, unfortunately.  We could also teach rpm-ostree how to do this itself with an extra reboot (or run the new base OS as a container, which is trickier; would work most of the time but introduces fun new failure modes with a particularly old kernel).

Comment 5 Jonathan Lebon 2020-09-08 16:06:30 UTC
Cross-linking: https://github.com/coreos/fedora-coreos-tracker/issues/609#issuecomment-688977962. Interesting to note for FCOS we could work around this relatively easily thanks to update barriers.

Comment 6 Adam Williamson 2020-09-08 16:33:13 UTC
"We could also teach rpm-ostree how to do this itself with an extra reboot"

I'd really like it if we could do this, because as everyone knows, lots of people never read the docs...

Comment 7 Colin Walters 2020-09-10 00:38:55 UTC
Hmm I'm leaning more and more towards making f33 use bdb by default for at least Silverblue and FCOS.

Comment 8 Jonathan Lebon 2020-09-15 14:06:50 UTC
This is fixed by https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f1c4e8bae. Notice it's against f33 even though this bug is opened against f32. Once it enters the f33 repos, f33 Silverblue composes will use BDB again and rebases from f32 should work.

Comment 9 Adam Williamson 2020-09-15 21:09:32 UTC
This should probably get a freeze exception, then, because otherwise that won't go stable until after Beta release. Proposing as an FE so we can fix rebase from Silverblue 32 to 33, seems reasonable to fix that during freeze.

Comment 10 Fedora Update System 2020-09-16 18:05:33 UTC
FEDORA-2020-0f1c4e8bae has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f1c4e8bae

Comment 11 Fedora Update System 2020-09-16 18:56:42 UTC
FEDORA-2020-0f1c4e8bae has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0f1c4e8bae`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f1c4e8bae

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Adam Williamson 2020-09-18 21:10:46 UTC
+5 votes in the ticket (https://pagure.io/fedora-qa/blocker-review/issue/65 ), setting accepted. Note it'd still be good to hear from pwhalen and/or pbrobinson to confirm this change is OK for IoT.

Comment 13 Paul Whalen 2020-09-19 02:35:17 UTC
(In reply to Adam Williamson from comment #12)
> +5 votes in the ticket (https://pagure.io/fedora-qa/blocker-review/issue/65
> ), setting accepted. Note it'd still be good to hear from pwhalen and/or
> pbrobinson to confirm this change is OK for IoT.

No issues on IoT, +1

Comment 14 Fedora Update System 2020-09-21 20:48:52 UTC
FEDORA-2020-0f1c4e8bae has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Tomas Popela 2020-09-23 05:34:07 UTC
Tried it today and I can't still successfully rebase to the F33 branch:

0 $ rpm-ostree rebase fedora/33/x86_64/silverblue
1 metadata, 0 content objects fetched; 592 B transferred in 3 seconds; 0 bytes content written
Checking out tree 2d91e02... done
Enabled rpm-md repositories: google-chrome fedora-cisco-openh264 google-chrome-beta fedora updates
rpm-md repo 'google-chrome' (cached); generated: 2020-09-16T17:43:15Z
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z
rpm-md repo 'google-chrome-beta' (cached); generated: 2020-09-16T17:43:15Z
rpm-md repo 'fedora' (cached); generated: 2020-09-20T10:51:30Z
rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z
Importing rpm-md... done
Resolving dependencies... done
Will download: 427 packages (319,3 MB)
Downloading from 'fedora'... done
Importing packages... done
Checking out packages... done
error: Checkout filesystem-3.14-3.fc33.x86_64: opendir(local): No such file or directory

I thought that the once the update gets into stable repos, then this will be resolved, but we are probably missing something else?

Comment 16 Colin Walters 2020-09-23 13:22:49 UTC
```
# ostree pull --commit-metadata-only fedora:fedora/33/x86_64/silverblue

GPG: Verification enabled, found 1 signature:

  Signature made Mon 21 Sep 2020 04:05:40 AM EDT using RSA key ID 6C13026D12C944D0
  Good signature from "Fedora <fedora-32-primary@fedoraproject.org>"
2 metadata, 0 content objects fetched; 56 KiB transferred in 4 seconds; 0 bytes content written                                                                                                                                                                 

# ostree show fedora:fedora/33/x86_64/silverblue | grep Date:

Date:  2020-09-21 08:05:26 +0000

#
```

The ostree commit needs to be regenerated.

Comment 17 Tomas Popela 2020-09-23 14:08:32 UTC
(In reply to Colin Walters from comment #16)
> The ostree commit needs to be regenerated.

Opened https://pagure.io/fedora-infrastructure/issue/9344

Thank you Colin!


Note You need to log in before you can comment on or make changes to this bug.