Bug 1282188

Summary: LVM unable to access volume group after upgrade to F23
Product: [Fedora] Fedora Reporter: Lukas Lueg <knabberknusperhaus>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 23CC: agk, bmarzins, bmr, dwysocha, heinzm, jonathan, knabberknusperhaus, lvm-team, msnitzer, prajnoha, prockai, zkabelac
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: lvm2-2.02.132-2.fc23 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-25 21:51:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vgcfgbackup output from a fresh F21 cached VG none

Description Lukas Lueg 2015-11-15 11:40:45 UTC
Description of problem:

After upgrading from F21 to F23 through the normal procedures, I was left with an unbootable system where LVM was be unable to access the volume group that makes my lvm-cached /home and stop the boot sequence with absolutely no error message or failure indicator (besides just waiting forever for the devices to come up).
Booting from a F23-LiveDVD I found out LVM indeed failed to access the volume group, blaming the lvm-cache's main logical volume with "LV [...] has invalid cache's feature flag." and "LV [...] is missing cache policy name."
The VG worked just fine before rebooting so I suspected the lvm toolkit from F23 for being overzealous about what was created under F21. Also, the error message above was added just recently with commit 533ac4d47dcecc976c05ba503acff11b9546b67c in August 2015. Accessing the VG with tools built from an earlier tarball indeed allowed me to access the VG and remove the cache-pool. After that, the F23-built lvm was able to access the VG and booting now succeeds.

I suspect the F21-built lvm tools create cache pools that the F23-build tools trip over.


Actual results: After upgrading from F21 to F23, a lvm-cached VG is inaccessible, causing the system to fail to boot. Additionally, no error is given whatsoever. 

Expected results: LVM in F23 should always be able to access a VG created under F21...


Additional info:

Comment 1 Lukas Lueg 2015-11-15 11:42:40 UTC
Additional info: These are the new checks that cause the F23-built lvm tools to reject even looking at my VG that worked just fine under F21: https://git.fedorahosted.org/cgit/lvm2.git/tree/lib/metadata/merge.c#n223

Comment 2 Zdenek Kabelac 2015-11-15 11:48:04 UTC
Please provide  'metadata' from your f21 system.
(vgcfgbackup)


And also please attach 'vgchange -ay -vvvv' trace.

Comment 3 Lukas Lueg 2015-11-15 12:17:31 UTC
Will do when I get to the box. The cache's pv has already been removed however to get my system back up running.

Glancing at the other code that deals with unset flags / policy_names, it seems the recently introduced checks in merge.c should indeed allow unset values, as other places deal with that explicitly. For example, cache_segtype/cache.c  reads 

seg->cleaner_policy ? "cleaner" :
/* undefined policy name -> likely an old "mq" */
cache_pool_seg->policy_name ? : "mq",

Comment 4 Lukas Lueg 2015-11-15 13:03:27 UTC
As the original lv that caused the problem is already gone, here is a vgcfgbackup from a dummy vg I just created using a F21-LiveDVD with LVM2 version 2.02.111.

After booting the LiveDVD, the sequence was:

fdisk /dev/sda
...
vgcreate dump /dev/sda2
lvcreate -L 20M -n lv_cache_meta dump
lvcreate -l 20%FREE -n lv_cache dump
lvcreate -l 99%FREE -n main dump
lvconvert --type cache-pool --poolmetadata dump/lv_cache_meta dump/lv_cache
lvconvert --type cache --cachepool dump/lv_cache dump/main
vgcfgbackup -v

As you can see in the attached vgcfgbackup-output, the logical volume lv_cache does not have keys "cache_mode" and "policy" set. Executing the same steps with a F23-LiveDVD and LVM2 version 2.02.130 results in "cache_mode" being present and set to "writethrough" and "policy" present and set to "smq".

Comment 5 Lukas Lueg 2015-11-15 13:04:33 UTC
Created attachment 1094415 [details]
vgcfgbackup output from a fresh F21 cached VG

Comment 6 Lukas Lueg 2015-11-15 19:25:07 UTC
A cached VG created by a F22-LiveDVD includes the "cache_mode" and "policy" keys for the cache LV.

I couldnt test installing a F21 system on a cached VG and then upgrading it because anaconda keeps crashing.

Comment 7 Zdenek Kabelac 2015-11-16 09:00:22 UTC
So lvm2 slightly updated 'logic' for cache creation.

It's now focused on entering all needed parameter right when user is caching some LV and not when just cache pool is created.

This change was need for better support of --commandprofile - which now can hold presets for 'user-liked' cache types.

To restore support for metadata created with older lvm2 - this patch has been upstreamed:

https://www.redhat.com/archives/lvm-devel/2015-November/msg00098.html

Comment 8 Lukas Lueg 2015-11-16 11:08:54 UTC
I've done a clean install of Fedora 21 in a virtual machine and added the cache device using the lvm tools that came with the LiveDVD. The system worked just fine.
Upgrading to Fedora 23 left the system unbootable exactly as I described above.

It seems the version of lvm shipped with Fedora 21 up to some point exposes this behaviour. Doing a clean install of Fedora 21, updating all packages, then adding the cache device and only then upgrading to Fedora 23 does not cause problems.

Comment 9 Fedora Update System 2015-11-24 11:29:17 UTC
lvm2-2.02.132-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-85c183f27a

Comment 10 Lukas Lueg 2015-11-24 12:25:59 UTC
One may add this to the Common-Bugs-Wiki since upgrading from F21 to F23 by the documented path may lead to an unbootable system.

Comment 11 Fedora Update System 2015-11-25 02:53:43 UTC
lvm2-2.02.132-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update lvm2'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-85c183f27a

Comment 12 Fedora Update System 2015-11-25 21:51:45 UTC
lvm2-2.02.132-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.