Bug 1876194
| Summary: | rpm-ostree update fails in filesystem checkout | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Vadim Rutkovsky <vrutkovs> |
| Component: | rpm-ostree | Assignee: | Jonathan Lebon <jlebon> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 32 | CC: | awilliam, dmach, dustymabe, ego.cordatus, extras-qa, feborges, igor.raits, jlebon, jmracek, jonathan, jrohel, juha.koskiniemi, kparal, lruzicka, mcermak, miabbott, mjw, ngompa13, nielsenb, packaging-team-maint, philip.wyett, pkratoch, pmatilai, pmoravco, pwhalen, rfairley, robatino, robertthomasfairley, sdodson, thiago.frmoraes, tpopela, travier, vmukhame, walters |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | AcceptedFreezeException | ||
| Fixed In Version: | rpm-ostree-2020.5-1.fc33 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1838691 | Environment: | |
| Last Closed: | 2020-09-21 20:48:52 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1838691 | ||
| Bug Blocks: | 1766776 | ||
Workaround: ``` rpm-ostree reset ... rpm-ostree rebase fedora/33/x86_64/testing/silverblue ... reboot ``` That however has nuked my shell (zsh), so I had to create a new user to login and run the rebase If I understand it correctly, this got fixed in F33 (bug 1838691) but not in F32. So I'm removing the F33 proposed blocker. Right, this will cause issues in the f32 -> f33 rebase because f32 rpm-ostree doesn't understand the new rpmdb since it has old libsolv and old librpm, so systems with layered packages will fail to rebase. The fix for this would be to backport the libsolv patch *and* rpm 4.16 to f32 (but make the rpmdb migration still gate on upgrading to f33), but I don't see that happening (/cc Panu in case this came up already). We'll probably have to document this on the "Common F33 bugs" page and tell users with overlays to `rpm-ostree reset` first (see https://bugzilla.redhat.com/show_bug.cgi?id=1838691#c34). Yeah, unfortunately. We could also teach rpm-ostree how to do this itself with an extra reboot (or run the new base OS as a container, which is trickier; would work most of the time but introduces fun new failure modes with a particularly old kernel). Cross-linking: https://github.com/coreos/fedora-coreos-tracker/issues/609#issuecomment-688977962. Interesting to note for FCOS we could work around this relatively easily thanks to update barriers. "We could also teach rpm-ostree how to do this itself with an extra reboot" I'd really like it if we could do this, because as everyone knows, lots of people never read the docs... Hmm I'm leaning more and more towards making f33 use bdb by default for at least Silverblue and FCOS. This is fixed by https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f1c4e8bae. Notice it's against f33 even though this bug is opened against f32. Once it enters the f33 repos, f33 Silverblue composes will use BDB again and rebases from f32 should work. This should probably get a freeze exception, then, because otherwise that won't go stable until after Beta release. Proposing as an FE so we can fix rebase from Silverblue 32 to 33, seems reasonable to fix that during freeze. FEDORA-2020-0f1c4e8bae has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f1c4e8bae FEDORA-2020-0f1c4e8bae has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0f1c4e8bae` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f1c4e8bae See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. +5 votes in the ticket (https://pagure.io/fedora-qa/blocker-review/issue/65 ), setting accepted. Note it'd still be good to hear from pwhalen and/or pbrobinson to confirm this change is OK for IoT. (In reply to Adam Williamson from comment #12) > +5 votes in the ticket (https://pagure.io/fedora-qa/blocker-review/issue/65 > ), setting accepted. Note it'd still be good to hear from pwhalen and/or > pbrobinson to confirm this change is OK for IoT. No issues on IoT, +1 FEDORA-2020-0f1c4e8bae has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report. Tried it today and I can't still successfully rebase to the F33 branch: 0 $ rpm-ostree rebase fedora/33/x86_64/silverblue 1 metadata, 0 content objects fetched; 592 B transferred in 3 seconds; 0 bytes content written Checking out tree 2d91e02... done Enabled rpm-md repositories: google-chrome fedora-cisco-openh264 google-chrome-beta fedora updates rpm-md repo 'google-chrome' (cached); generated: 2020-09-16T17:43:15Z rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z rpm-md repo 'google-chrome-beta' (cached); generated: 2020-09-16T17:43:15Z rpm-md repo 'fedora' (cached); generated: 2020-09-20T10:51:30Z rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z Importing rpm-md... done Resolving dependencies... done Will download: 427 packages (319,3 MB) Downloading from 'fedora'... done Importing packages... done Checking out packages... done error: Checkout filesystem-3.14-3.fc33.x86_64: opendir(local): No such file or directory I thought that the once the update gets into stable repos, then this will be resolved, but we are probably missing something else? ``` # ostree pull --commit-metadata-only fedora:fedora/33/x86_64/silverblue GPG: Verification enabled, found 1 signature: Signature made Mon 21 Sep 2020 04:05:40 AM EDT using RSA key ID 6C13026D12C944D0 Good signature from "Fedora <fedora-32-primary>" 2 metadata, 0 content objects fetched; 56 KiB transferred in 4 seconds; 0 bytes content written # ostree show fedora:fedora/33/x86_64/silverblue | grep Date: Date: 2020-09-21 08:05:26 +0000 # ``` The ostree commit needs to be regenerated. (In reply to Colin Walters from comment #16) > The ostree commit needs to be regenerated. Opened https://pagure.io/fedora-infrastructure/issue/9344 Thank you Colin! |
(removed a few comments for clarity) Still hitting this on Silverblue F32: ``` $ rpm-ostree status State: idle Deployments: ● ostree://fedora:fedora/32/x86_64/silverblue Version: 32.20200905.0 (2020-09-05T18:11:38Z) BaseCommit: 4150be4c9953a25b2ce8dc4197df6e2221014d23d8a162550dc0391ba13d692b GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 LayeredPackages: gstreamer1-plugins-ugly-free libvirt libvirt-daemon-kvm make qemu-kvm tilix zsh LocalPackages: rpmfusion-nonfree-release-32-1.noarch rpmfusion-free-release-32-1.noarch Pinned: yes $ rpm-ostree rebase fedora/33/x86_64/testing/silverblue --uninstall=rpmfusion-free-release-32-1.noarch --uninstall=rpmfusion-nonfree-release-32-1.noarch --uninstall=gstreamer1-plugins-ugly-free 1 metadata, 0 content objects fetched; 592 B transferred in 2 seconds; 0 bytes content written Checking out tree c9d135b... done Enabled rpm-md repositories: updates updates-testing fedora rpm-md repo 'updates' (cached); generated: 2018-02-20T19:18:14Z rpm-md repo 'updates-testing' (cached); generated: 2020-09-05T21:31:13Z rpm-md repo 'fedora' (cached); generated: 2020-09-04T10:59:23Z Importing rpm-md... done Resolving dependencies... done Will download: 479 packages (244.7 MB) Downloading from 'fedora'... done Downloading from 'updates-testing'... done Importing packages... done Checking out packages... done error: Checkout filesystem-3.14-3.fc33.x86_64: opendir(local): No such file or directory ``` libsolv from F33 requires newer RPM: ``` rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm Downloading 'https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm'... done! Checking out tree 4150be4... done Enabled rpm-md repositories: updates updates-testing fedora rpm-md repo 'updates' (cached); generated: 2020-09-05T18:22:52Z rpm-md repo 'updates-testing' (cached); generated: 2020-09-05T17:50:28Z rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:36Z Importing rpm-md... done Resolving dependencies... done error: Could not depsolve transaction; 1 problem detected: Problem: conflicting requests - nothing provides rpm(x86-64) >= 4.16.0 needed by libsolv-0.7.14-2.fc33.x86_64 ``` +++ This bug was initially created as a clone of Bug #1838691 +++ rpm-ostree fails error: Checkout filesystem-3.14-2.fc32.x86_64: opendir(local): No such file or directory ... --- Additional comment from Colin Walters on 2020-05-22 14:23:14 UTC --- OK that's concerning, it's possible that something broke in rawhide related to this; handling of the `filesystem` package is special in rpm-ostree. ... --- Additional comment from Colin Walters on 2020-05-24 23:37:10 UTC --- OK I think what's going on here probably has something to do with the rpm switch to use the sqlite backend. Depsolving isn't finding the "base" packages - one can see this with e.g. `rpm-ostree install --dry-run wget` and notice that depsolving includes `filesystem` and `bash` etc. Trying to re-layer `filesystem` failed because it tries to unpack `/usr/local` but that's really `/var/usrlocal` which the unpacking process can't see. (This is part of how rpm-ostree protects user data, it's very hard even if there are bugs for us to walk off and affect the real /var where e.g. the homedir is) Going to need to add some debugging prints. --- Additional comment from Colin Walters on 2020-05-25 20:46:34 UTC --- https://github.com/openSUSE/libsolv/pull/386 --- Additional comment from Colin Walters on 2020-05-25 20:56:55 UTC --- https://src.fedoraproject.org/rpms/libsolv/c/c6160aa95a361d9ac498e1586b9227e55753314d?branch=master https://koji.fedoraproject.org/koji/taskinfo?taskID=44965714 ... --- Additional comment from Fedora Blocker Bugs Application on 2020-05-28 21:10:45 UTC --- Proposed as a Blocker for 33-beta by Fedora user lruzicka using the blocker tracking app because: The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated. --- Additional comment from Daniel Mach on 2020-06-01 11:43:05 UTC --- Colin, could you provide some details on why do you need the paths? Many tools use the paths because they detect rpmdb changes, but maybe we should provide another way as part of librpm API. The database should remain an implementation detail, a private API. --- Additional comment from Colin Walters on 2020-06-01 14:10:54 UTC --- For rpm-ostree, the database must be in /usr - this has been extensively discussed before, see e.g. http://lists.rpm.org/pipermail/rpm-maint/2017-October/006681.html I'd also like to use this opportunity of changing the backend to sqlite to *also* move the rpm database to /usr/lib/sysimage/rpm for all editions/spins. Ideally as long as librpm supports probing per above, we wouldn't need to know the specific path. That said, in order for us to support transitioning from /usr/share/rpm we'd probably need to make it configurable where to place the rpmdb, which might require some support for both configuring and probing the path explicitly. ... --- Additional comment from Colin Walters on 2020-06-03 14:07:19 UTC --- > What can be workaround. I want to stick to Rawhide and not to rebase. How to force update? You can use `rpm-ostree override replace` with e.g. the builds from https://koji.fedoraproject.org/koji/buildinfo?buildID=1519658 or temporarily remove your layered packages, upgrade, then re-add them. ... --- Additional comment from Jonathan Lebon on 2020-06-05 19:36:06 UTC --- Try: ``` $ rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/libsolv/0.7.14/2.fc33/x86_64/libsolv-0.7.14-2.fc33.x86_64.rpm $ reboot --- $ rpm-ostree upgrade ``` ... --- Additional comment from Adam Williamson on 2020-06-22 20:19:21 UTC --- So, this bug is still proposed as a blocker for F33 Beta, but I'm confused as to the current status at this point. Is there still a serious problem with ostree here, Lukas? ... --- Additional comment from Colin Walters on 2020-06-22 22:52:28 UTC --- This should have been fixed in libsolv as of https://src.fedoraproject.org/rpms/libsolv/c/c6160aa95a361d9ac498e1586b9227e55753314d?branch=master I just tested the latest rpm-ostree+libsolv in $ rpm-ostree status -b State: idle BootedDeployment: ● ostree://fedora:fedora/rawhide/x86_64/silverblue Version: Rawhide.20200622.n.0 (2020-06-22T06:03:56Z) Commit: d44b3db4d47b448adb6a19bb5cfe4f58de02e4d3960829ef695e3f3288f9630f GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31 $ and things look fine. ...