Red Hat Bugzilla – Bug 1282188
LVM unable to access volume group after upgrade to F23
Last modified: 2015-11-25 16:51:55 EST
Description of problem:
After upgrading from F21 to F23 through the normal procedures, I was left with an unbootable system where LVM was be unable to access the volume group that makes my lvm-cached /home and stop the boot sequence with absolutely no error message or failure indicator (besides just waiting forever for the devices to come up).
Booting from a F23-LiveDVD I found out LVM indeed failed to access the volume group, blaming the lvm-cache's main logical volume with "LV [...] has invalid cache's feature flag." and "LV [...] is missing cache policy name."
The VG worked just fine before rebooting so I suspected the lvm toolkit from F23 for being overzealous about what was created under F21. Also, the error message above was added just recently with commit 533ac4d47dcecc976c05ba503acff11b9546b67c in August 2015. Accessing the VG with tools built from an earlier tarball indeed allowed me to access the VG and remove the cache-pool. After that, the F23-built lvm was able to access the VG and booting now succeeds.
I suspect the F21-built lvm tools create cache pools that the F23-build tools trip over.
Actual results: After upgrading from F21 to F23, a lvm-cached VG is inaccessible, causing the system to fail to boot. Additionally, no error is given whatsoever.
Expected results: LVM in F23 should always be able to access a VG created under F21...
Additional info: These are the new checks that cause the F23-built lvm tools to reject even looking at my VG that worked just fine under F21: https://git.fedorahosted.org/cgit/lvm2.git/tree/lib/metadata/merge.c#n223
Please provide 'metadata' from your f21 system.
And also please attach 'vgchange -ay -vvvv' trace.
Will do when I get to the box. The cache's pv has already been removed however to get my system back up running.
Glancing at the other code that deals with unset flags / policy_names, it seems the recently introduced checks in merge.c should indeed allow unset values, as other places deal with that explicitly. For example, cache_segtype/cache.c reads
seg->cleaner_policy ? "cleaner" :
/* undefined policy name -> likely an old "mq" */
cache_pool_seg->policy_name ? : "mq",
As the original lv that caused the problem is already gone, here is a vgcfgbackup from a dummy vg I just created using a F21-LiveDVD with LVM2 version 2.02.111.
After booting the LiveDVD, the sequence was:
vgcreate dump /dev/sda2
lvcreate -L 20M -n lv_cache_meta dump
lvcreate -l 20%FREE -n lv_cache dump
lvcreate -l 99%FREE -n main dump
lvconvert --type cache-pool --poolmetadata dump/lv_cache_meta dump/lv_cache
lvconvert --type cache --cachepool dump/lv_cache dump/main
As you can see in the attached vgcfgbackup-output, the logical volume lv_cache does not have keys "cache_mode" and "policy" set. Executing the same steps with a F23-LiveDVD and LVM2 version 2.02.130 results in "cache_mode" being present and set to "writethrough" and "policy" present and set to "smq".
Created attachment 1094415 [details]
vgcfgbackup output from a fresh F21 cached VG
A cached VG created by a F22-LiveDVD includes the "cache_mode" and "policy" keys for the cache LV.
I couldnt test installing a F21 system on a cached VG and then upgrading it because anaconda keeps crashing.
So lvm2 slightly updated 'logic' for cache creation.
It's now focused on entering all needed parameter right when user is caching some LV and not when just cache pool is created.
This change was need for better support of --commandprofile - which now can hold presets for 'user-liked' cache types.
To restore support for metadata created with older lvm2 - this patch has been upstreamed:
I've done a clean install of Fedora 21 in a virtual machine and added the cache device using the lvm tools that came with the LiveDVD. The system worked just fine.
Upgrading to Fedora 23 left the system unbootable exactly as I described above.
It seems the version of lvm shipped with Fedora 21 up to some point exposes this behaviour. Doing a clean install of Fedora 21, updating all packages, then adding the cache device and only then upgrading to Fedora 23 does not cause problems.
lvm2-2.02.132-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-85c183f27a
One may add this to the Common-Bugs-Wiki since upgrading from F21 to F23 by the documented path may lead to an unbootable system.
lvm2-2.02.132-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update lvm2'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-85c183f27a
lvm2-2.02.132-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.