Bug 1282188 - LVM unable to access volume group after upgrade to F23
LVM unable to access volume group after upgrade to F23
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Zdenek Kabelac
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2015-11-15 06:40 EST by Lukas Lueg
Modified: 2015-11-25 16:51 EST (History)
12 users (show)

See Also:
Fixed In Version: lvm2-2.02.132-2.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-11-25 16:51:55 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
vgcfgbackup output from a fresh F21 cached VG (3.36 KB, text/plain)
2015-11-15 08:04 EST, Lukas Lueg
no flags Details

  None (edit)
Description Lukas Lueg 2015-11-15 06:40:45 EST
Description of problem:

After upgrading from F21 to F23 through the normal procedures, I was left with an unbootable system where LVM was be unable to access the volume group that makes my lvm-cached /home and stop the boot sequence with absolutely no error message or failure indicator (besides just waiting forever for the devices to come up).
Booting from a F23-LiveDVD I found out LVM indeed failed to access the volume group, blaming the lvm-cache's main logical volume with "LV [...] has invalid cache's feature flag." and "LV [...] is missing cache policy name."
The VG worked just fine before rebooting so I suspected the lvm toolkit from F23 for being overzealous about what was created under F21. Also, the error message above was added just recently with commit 533ac4d47dcecc976c05ba503acff11b9546b67c in August 2015. Accessing the VG with tools built from an earlier tarball indeed allowed me to access the VG and remove the cache-pool. After that, the F23-built lvm was able to access the VG and booting now succeeds.

I suspect the F21-built lvm tools create cache pools that the F23-build tools trip over.

Actual results: After upgrading from F21 to F23, a lvm-cached VG is inaccessible, causing the system to fail to boot. Additionally, no error is given whatsoever. 

Expected results: LVM in F23 should always be able to access a VG created under F21...

Additional info:
Comment 1 Lukas Lueg 2015-11-15 06:42:40 EST
Additional info: These are the new checks that cause the F23-built lvm tools to reject even looking at my VG that worked just fine under F21: https://git.fedorahosted.org/cgit/lvm2.git/tree/lib/metadata/merge.c#n223
Comment 2 Zdenek Kabelac 2015-11-15 06:48:04 EST
Please provide  'metadata' from your f21 system.

And also please attach 'vgchange -ay -vvvv' trace.
Comment 3 Lukas Lueg 2015-11-15 07:17:31 EST
Will do when I get to the box. The cache's pv has already been removed however to get my system back up running.

Glancing at the other code that deals with unset flags / policy_names, it seems the recently introduced checks in merge.c should indeed allow unset values, as other places deal with that explicitly. For example, cache_segtype/cache.c  reads 

seg->cleaner_policy ? "cleaner" :
/* undefined policy name -> likely an old "mq" */
cache_pool_seg->policy_name ? : "mq",
Comment 4 Lukas Lueg 2015-11-15 08:03:27 EST
As the original lv that caused the problem is already gone, here is a vgcfgbackup from a dummy vg I just created using a F21-LiveDVD with LVM2 version 2.02.111.

After booting the LiveDVD, the sequence was:

fdisk /dev/sda
vgcreate dump /dev/sda2
lvcreate -L 20M -n lv_cache_meta dump
lvcreate -l 20%FREE -n lv_cache dump
lvcreate -l 99%FREE -n main dump
lvconvert --type cache-pool --poolmetadata dump/lv_cache_meta dump/lv_cache
lvconvert --type cache --cachepool dump/lv_cache dump/main
vgcfgbackup -v

As you can see in the attached vgcfgbackup-output, the logical volume lv_cache does not have keys "cache_mode" and "policy" set. Executing the same steps with a F23-LiveDVD and LVM2 version 2.02.130 results in "cache_mode" being present and set to "writethrough" and "policy" present and set to "smq".
Comment 5 Lukas Lueg 2015-11-15 08:04 EST
Created attachment 1094415 [details]
vgcfgbackup output from a fresh F21 cached VG
Comment 6 Lukas Lueg 2015-11-15 14:25:07 EST
A cached VG created by a F22-LiveDVD includes the "cache_mode" and "policy" keys for the cache LV.

I couldnt test installing a F21 system on a cached VG and then upgrading it because anaconda keeps crashing.
Comment 7 Zdenek Kabelac 2015-11-16 04:00:22 EST
So lvm2 slightly updated 'logic' for cache creation.

It's now focused on entering all needed parameter right when user is caching some LV and not when just cache pool is created.

This change was need for better support of --commandprofile - which now can hold presets for 'user-liked' cache types.

To restore support for metadata created with older lvm2 - this patch has been upstreamed:

Comment 8 Lukas Lueg 2015-11-16 06:08:54 EST
I've done a clean install of Fedora 21 in a virtual machine and added the cache device using the lvm tools that came with the LiveDVD. The system worked just fine.
Upgrading to Fedora 23 left the system unbootable exactly as I described above.

It seems the version of lvm shipped with Fedora 21 up to some point exposes this behaviour. Doing a clean install of Fedora 21, updating all packages, then adding the cache device and only then upgrading to Fedora 23 does not cause problems.
Comment 9 Fedora Update System 2015-11-24 06:29:17 EST
lvm2-2.02.132-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-85c183f27a
Comment 10 Lukas Lueg 2015-11-24 07:25:59 EST
One may add this to the Common-Bugs-Wiki since upgrading from F21 to F23 by the documented path may lead to an unbootable system.
Comment 11 Fedora Update System 2015-11-24 21:53:43 EST
lvm2-2.02.132-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update lvm2'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-85c183f27a
Comment 12 Fedora Update System 2015-11-25 16:51:45 EST
lvm2-2.02.132-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.