Relocate RPM database to /usr/lib/sysimage/rpm https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr#Scope Richard W.M. Jones via lists.fedoraproject.org wrote: Please can you make sure two bugs are filed against libguestfs and supermin components to track this change (if it happens). We now use librpm to parse the RPM database so I think we're OK from the inspection side, but we do use the RPM database's real location in two places: [supermin] To test if a package has been installed/upated/removed so that we can rebuild our cache [libguestfs] To build a "phony" Fedora image for testing with a fake RPM database. https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/B37AFXH3TFGKO3H5IHKIYH6TA3YIJNGK/
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle. Changing version to 36.
Hey Chris, could you comment on the status of this change? In current Rawhide I see that /usr/lib/sysimage/rpm/ has symlinks to the old location: $ ls -l /usr/lib/sysimage/rpm/ total 0 lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal It's unclear to me if this is the final situation, or if there will be a later change that moves the database itself? Or if this is something to do with the Rawhide machine being an upgrade instead of a fresh install.
The database should get moved. Sounds like Rawhide missed a step. Initially there's symlinks pointing to the old location. But then it switches with the new location have rebuilt RPM database, i.e. the rebuilding is what causes the move. And then a new symlink in the old location pointing to the new location. >/var/lib/rpm -> ../../usr/lib/sysimage/rpm
Maybe the journal still contains a hint at the failure? But when I do 'journalctl --no-hostname | grep rpmdb' I get two lines per boot on a system that has already done the migration: >Oct 27 00:01:43 systemd[1]: rpmdb-migrate.service - RPM database migration to /usr was skipped because of a failed condition check (ConditionPathExists=/var/lib/rpm/.migratedb). >Oct 27 00:01:43 systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). But try that anyway and attach the whole thing as a file if it's long, and we'll see if there's something that stands out like an attempted migration or rebuild. Maybe migrate happened and rebuild didn't?
# journalctl --no-hostname There are no lines at all that match any of: - rpmdb - database migration - var-lib-rpm - migrate # except some obviously irrelevant ones The only lines referring to /var/lib/rpm were small variations of: Aug 31 13:02:51 userhelper[2620763]: running '/usr/libexec/mock/mock -r fedora-37-x86_64 --no-cleanup-after --no-clean --resultdir=/var/tmp/2121258-python-textual/results --shell 'rm -f /var/lib/rpm/__db*'' with root privileges on behalf of 'rjones' which seems not relevant. The journal goes back to April 13th so about 6 months. BTW I now have rpm-4.18.0-3.fc38.x86_64 installed.
Pretty sure the change would have landed in Rawhide before April 13. I'm not sure if this is a one off or if there might be a bug here? Maybe we should start a devel thread and ask people to check /usr/lib/rpm and /var/lib/rpm, and report back if they find something unexpected? $ ls -la /var/lib/rpm lrwxrwxrwx. 1 root root 26 May 4 17:29 /var/lib/rpm -> ../../usr/lib/sysimage/rpm $ ls -la /usr/lib/rpm total 148 drwxr-xr-x. 1 root root 404 Oct 25 12:59 . dr-xr-xr-x. 1 root root 790 Oct 11 10:04 .. -rwxr-xr-x. 1 root root 1465 Jul 20 21:45 brp-boot-efi-times drwxr-xr-x. 1 root root 172 Oct 25 12:59 fileattrs -rwxr-xr-x. 1 root root 950 Oct 14 04:57 gstreamer1.prov drwxr-xr-x. 1 root root 12 Sep 21 06:52 lua -rw-r--r--. 1 root root 42661 Sep 21 06:52 macros drwxr-xr-x. 1 root root 1416 Oct 11 10:04 macros.d -rwxr-xr-x. 1 root root 6156 Aug 17 10:30 perl.prov -rwxr-xr-x. 1 root root 12315 Aug 17 10:30 perl.req drwxr-xr-x. 1 root root 1690 Sep 21 06:52 platform -rwxr-xr-x. 1 root root 7754 Jul 22 13:46 postscriptdriver.prov drwxr-xr-x. 1 root root 900 Oct 25 12:59 redhat -rwxr-xr-x. 1 root root 1597 Sep 2 01:48 rpm2cpio.sh -rw-r--r--. 1 root root 296 Apr 7 2022 rpm.daily -rwxr-xr-x. 1 root root 41 Apr 7 2022 rpmdb_dump -rwxr-xr-x. 1 root root 41 Apr 7 2022 rpmdb_load -rwxr-xr-x. 1 root root 1020 Sep 21 06:52 rpmdb_migrate -rw-r--r--. 1 root root 61 Apr 7 2022 rpm.log -rw-r--r--. 1 root root 10793 Sep 21 06:51 rpmpopt-4.18.0 -rw-r--r--. 1 root root 17872 Sep 21 06:51 rpmrc -rw-r--r--. 1 root root 688 Apr 7 2022 rpm.supp -rwxr-xr-x. 1 root root 937 Apr 7 2022 tgpg Uhh, does this seem a bit cluttered?
Did you mean /usr/lib/sysimage/rpm? Anyway on this machine: $ ls -la /var/lib/rpm total 339004 drwxr-xr-x. 2 root root 4096 Feb 16 2022 . drwxr-xr-x. 76 root root 4096 Aug 23 13:28 .. -rw-r--r--. 1 root root 0 Oct 18 14:28 .migratedb -rw-r--r--. 1 root root 347095040 Oct 26 16:50 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 27 15:28 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 26 16:50 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Feb 16 2022 .rpm.lock $ ls -la /usr/lib/sysimage/rpm total 8 drwxr-xr-x. 2 root root 4096 Oct 5 12:05 . drwxr-xr-x. 3 root root 4096 Feb 9 2022 .. lrwxrwxrwx. 1 root root 34 Oct 18 14:28 .migratedb -> ../../../../var/lib/rpm/.migratedb lrwxrwxrwx. 1 root root 36 Oct 18 14:28 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx. 1 root root 40 Oct 18 14:28 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal lrwxrwxrwx. 1 root root 33 Oct 18 14:28 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock
Yes. :facepalm: This is what I have: $ ls -la /usr/lib/sysimage/rpm total 200340 drwxr-xr-x. 1 root root 154 Sep 25 00:53 . drwxr-xr-x. 1 root root 6 Aug 9 09:27 .. -rw-r--r--. 1 root root 0 May 4 17:29 .rpmdbdirsymlink_created -rw-r--r--. 1 root root 205115392 Oct 26 22:11 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 27 11:33 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 26 22:11 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 May 4 17:26 .rpm.lock Your system appears to not have been migrated. Yet you have the .migratedb hidden file. For some reason that is not triggering the migration, and looks like it's in some in-between state. I haven't seen this before. I've asked on #devel channel if anyone can check this. Of course this was tested a lot from f34 and f35 to f36. But seems plausible a bug specific to Rawhide could be going uncaught, and difficult to test now without just polling people because we don't have old enough Rawhide images lying around anywhere.
Following the request on IRC: I have state very similar to comment #7: $ ls -la /var/lib/rpm total 79072 drwxr-xr-x 2 root root 4096 Feb 7 2022 . drwxr-xr-x. 27 root root 4096 Aug 9 15:27 .. -rw-r--r-- 1 root root 0 Sep 26 19:15 .migratedb -rw-r--r-- 1 root root 0 Feb 7 2022 .rpm.lock -rw-r--r-- 1 root root 80924672 Oct 4 19:25 rpmdb.sqlite -rw-r--r-- 1 root root 32768 Oct 27 20:49 rpmdb.sqlite-shm -rw-r--r-- 1 root root 0 Oct 4 19:25 rpmdb.sqlite-wal $ ls -la /usr/lib/sysimage/rpm total 8 drwxr-xr-x 2 root root 4096 Sep 21 12:32 . drwxr-xr-x. 5 root root 4096 Oct 4 19:25 .. lrwxrwxrwx 1 root root 34 Sep 26 19:13 .migratedb -> ../../../../var/lib/rpm/.migratedb lrwxrwxrwx 1 root root 33 Sep 26 19:13 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock lrwxrwxrwx 1 root root 36 Sep 26 19:13 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx 1 root root 40 Sep 26 19:13 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx 1 root root 40 Sep 26 19:13 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal $ rpm -q rpm rpm-4.18.0-1.fc38.x86_64 $ journalctl --no-hostname --grep=rpmdb Apr 07 09:50:55 systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). (and then it repeats ~40 times, once per boot)
Posted here, let's see if this problem is rare or widespread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/G6YTC4E5CZGIKSO5FUTYLI2NJL3A4EDF/
Fedora 35->36 distro-sync 3 days ago: Current (F36) Name : rpm Version : 4.17.1 Release : 3.fc36 Architecture: x86_64 Install Date: Mi 26 Okt 2022 12:05:55 CEST # ll /var/lib/rpm insgesamt 116804 -rw-r--r-- 1 root root 119570432 28. Okt 18:02 rpmdb.sqlite -rw-r--r-- 1 root root 32768 28. Okt 20:45 rpmdb.sqlite-shm -rw-r--r-- 1 root root 0 28. Okt 18:02 rpmdb.sqlite-wal [root@s113 ~]# pathdiscover /var/lib/rpm '/var/lib/rpm' translates to '/var/lib/rpm' 4096 Bytes root/root drwxr-xr-x : rpm ( directory ) 4096 Bytes root/root drwxr-xr-x : lib ( directory ) 4096 Bytes root/root drwxr-xr-x : var ( directory ) # ll /usr/lib/sysimage/ insgesamt 4 drwxr-xr-x 2 root root 4096 2. Aug 14:32 rpm # ll /usr/lib/sysimage/rpm/ insgesamt 0 lrwxrwxrwx 1 root root 36 26. Okt 12:05 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx 1 root root 40 26. Okt 12:05 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx 1 root root 40 26. Okt 12:05 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal
I am seeing same kind of thing as Richard, but I'm on Fedora 36. I upgraded from Fedora 35 using dnf system-upgrade back on September 19 it looks like from the logs. I haven't tried running the migrate service yet in case there is some other data to gather. $ ls -la /var/lib/rpm total 371280 drwxr-xr-x. 2 root root 4096 Sep 19 13:19 ./ drwxr-xr-x. 76 root root 4096 Sep 19 12:28 ../ -rw-r--r--. 1 root root 0 Sep 19 12:36 .migratedb -rw-r--r--. 1 root root 380145664 Oct 24 08:09 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Oct 28 15:41 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Oct 24 08:09 rpmdb.sqlite-wal -rw-r--r--. 1 root root 0 Sep 19 13:19 .rpm.lock $ ls -la /usr/lib/sysimage/rpm total 8 drwxr-xr-x. 2 root root 4096 Aug 2 05:32 ./ drwxr-xr-x. 3 root root 4096 Sep 19 12:27 ../ lrwxrwxrwx. 1 root root 36 Sep 19 12:27 rpmdb.sqlite -> ../../../../var/lib/rpm/rpmdb.sqlite lrwxrwxrwx. 1 root root 40 Sep 19 12:27 rpmdb.sqlite-shm -> ../../../../var/lib/rpm/rpmdb.sqlite-shm lrwxrwxrwx. 1 root root 40 Sep 19 12:27 rpmdb.sqlite-wal -> ../../../../var/lib/rpm/rpmdb.sqlite-wal lrwxrwxrwx. 1 root root 33 Sep 19 12:27 .rpm.lock -> ../../../../var/lib/rpm/.rpm.lock $ systemctl status rpmdb-migrate.service ○ rpmdb-migrate.service - RPM database migration to /usr Loaded: loaded (/usr/lib/systemd/system/rpmdb-migrate.service; disabled; vendor preset: enabled) Active: inactive (dead) $ journalctl --no-hostnam -a | grep rpmdb (main-azure-sap $%) Sep 19 13:17:48 systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). . . . Oct 24 08:13:05 systemd[1]: rpmdb-rebuild.service - RPM database rebuild was skipped because of a failed condition check (ConditionPathExists=/usr/lib/sysimage/rpm/.rebuilddb). Installation of rpmdb-rebuild.service was back in 2021: 2021-05-03T08:40:53-0700 SUBDEBUG Upgraded: rpm-4.15.1.1-1.fc32.1.x86_64 2021-05-03T08:40:53-0700 INFO Created symlink /etc/systemd/system/basic.target.wants/rpmdb-rebuild.service → /usr/lib/systemd/system/rpmdb-rebuild.service. And while there is /usr/lib/systemd/system/rpmdb-migrate.service there is no symlink to it from anywhere so it looks like it was not enabled at install time.
I posted this to the devel list¹, but it's probably worth having here as well. (That is, assuming I'm not just wildly mistaken.) I upgraded two systems from f35 to f36 in the past week via dnf system-upgrade. Neither of them completed the rpm migration. I _think_ the issue is due to the version-release in the %triggerun which should enable the rpmdb-migrate service. That is: %triggerun -- rpm < 4.17.0-7 # Handle rpmdb migrate service on erasure of old to avoid ordering issues if [ -x /usr/bin/systemctl ]; then systemctl --no-reload preset rpmdb-migrate ||: fi That was accurate when it was added in 0b9f813 (Migrate rpmdb to /usr/lib/sysimage/rpm (#2042099), 2022-01-26). However, when rpm was updated in f35 in e9927df (Rebase to rpm 4.17.1 (http://rpm.org/wiki/Releases/4.17.1), 2022-07-01), which never had the migration code, it prevented any up-to-date f35 system from triggering the migration. Anyone upgrading from f35 after rpm-4.17.1 was pushed to f35 won't have the rpmdb-migrate service enabled and the database will remain in /var/lib/rpm (with .migratedb) and symlinks in /usr/lib/sysimage/rpm. It seems that it's not so much a failed migration as a migration which is never attempted. :/ ¹ https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/6AGSBAACILHNG7SU7C7CC4UQ2ZZIGXHH/
I upgraded today (2023-02-04) from Fedora 36 to Fedora 37 and for the first time encountered this bug in slightly different form: Prior advise to manually > # systemctl enable rpmdb-migrate > # systemctl start rpmdb-migrate Did **NOT** solve the issue. Found this discussion and checked ls -la /usr/lib/sysimage/rpm. Found a `.migratedb` in there. Also found following files: .rpm.lock rpmdb.sqlite rpmdb.sqlite-shm rpmdb.sqlite-wal What I did to (hopefully) fix it: 1. Deleted the `.migratedb` via `sudo rm .migratedb`. 2. Ran the `RPM database migration to /usr` Systemservice via Fedora Cockpit (This should have been equivalent to `systemctl start rpmdb-migrate`). 3. Following error was triggered: rpmdb-migrate.service - RPM database migration to /usr was skipped because of a failed condition check (ConditionPathExists=/var/lib/rpm/.migratedb). 4. Remembered, the `RPM database migration to /usr` Systemservice was supposed to run at boot, so I rebooted my machine. 5. After restart, the `RPM database migration to /usr` Systemservice` did not trigger Fedora cockpit to give a warning anymore. So, I checked both folders (i should have checked BOTH folders before deleting stuff btw.!): $ ls -la /var/lib/rpm lrwxrwxrwx. 1 root root 26 Jun 25 2022 /var/lib/rpm -> ../../usr/lib/sysimage/rpm $ ls -la /usr/lib/sysimage/rpm total 122940 drwxr-xr-x. 2 root root 123 Feb 4 19:21 . drwxr-xr-x. 3 root root 17 Aug 9 15:27 .. -rw-r--r--. 1 root root 0 Jul 1 2022 .rpm.lock -rw-r--r--. 1 root root 125857792 Feb 4 18:34 rpmdb.sqlite -rw-r--r--. 1 root root 32768 Feb 4 19:25 rpmdb.sqlite-shm -rw-r--r--. 1 root root 0 Feb 4 18:34 rpmdb.sqlite-wal which seemed pretty close to a migration. I noticed `.rpmdbdirsymlink_created` was missing, so i manually created the file via `sudo touch .rpmdbdirsymlink_created` `-rw-r--r--. 1 root root 0 Feb 4 19:21 .rpmdbdirsymlink_created` Hopefully this was correct. Hopefully this will help anybody encountering the same issue. @maintainers: please fix it. Bye.
The actual (supermin) bug described in this report has been fixed. https://github.com/libguestfs/supermin/commit/86fd6f3e86ab99d54a22b475aecccfc19bdff07e If there are other issues with the RPM database location I'd suggest asking on the Fedora thread mentioned in comment 10.
I created a new bug for my problem: Bug 2171863