Description of problem: One of my original LV partition could not be activitied during boot session (the other is OK). After entering the system, the 'lvs' command results a NOT AVAILABLE feedback. By using 'vgchange -a y ' command, it could temporally turn the LV being usable, but after rebooting, issue is there.. Version-Release number of selected component (if applicable): By downgrading to 2.02.184-4, issue completely resolved. How reproducible: After getting lvm2 package updated to 2.02.184-6, this issue occurs. Actual results: The journalctl results during the boot-up has the following information, the propoer results should have two 'LVM2 PV scan' items, there is only one: Created slice system-lvm2\x2dpvscan.slice. Starting LVM2 PV scan on device xxx:1... pvscan[583] WARNING: lvmetad is being updated, retrying (setup) for 10 more seconds. Additional info: OS: Archlinux The /etc/lvm/lvm.conf has not being modified yet.
Archlinux seems to be depending on some unwanted fallback error handling (potentially even bugs) in pvscan. Specifically, when run with a single device, we do not want pvscan to scan/activate all devs, and we don't want pvscan to activate a VG until it's complete, and we don't intend for pvscan to be used without lvmetad (in 2.02 versions). Archlinux seems to depend on these three unwanted behaviors, and perhaps others, to activate LVs during boot. It also seems to be using pvscan for activation in initramfs which it is not meant for. The system should never need to try to activate a VG a second time, but archlinux seems to depend on this. I've not seen any good solutions for doing LVM activation of the root LV in the initramfs (something we're beginning to look at), but this one seems to be quite wrong. I've recently been looking at the fedora/rhel approach which is also poor. I haven't seen the specifics of how archlinux intends for this to work.
On archlinux is /var/run/lvm/ on tmpfs? pvscan activation depends on those temp files being cleared on boot.
Analyzing one set of logs more closely, a "working" archlinux boot seems to be: 1. initramfs runs a pvscan --cache -aay when the first device appears (other devs still missing) 2. pvscan error fallback code ends up activating any LVs present on the first device 3. the lvm2-pvscan service fails because the pvscan was not successful 4. switch-root happens 5. all devices are now present 6. main system runs pvscan --cache -aay on all devices, including the one already done above. 7. all pvscans attempt to activate the VG, and one or more end up activating all VGs. With the recent change, pvscan created a temp file in /run/lvm/vgs_online between steps 2 and 3 which prevents the pvscans in step 7 from attempting to activate the VG again. One hack to work around this might be for archlinux to run "rm /run/lvm/vgs_online/*" before switch-root. This doesn't fix the incorrect approach to activating LVs, though. A correct approach would be something like: a) do not use pvscan or lvmetad to activate LVs in the initramfs. b) the initramfs should attempt a direct activation of the root LV by calling lvchange -ay vg/lv. (Attempting this lvchange command after every device or PV appears might be necessary.) c) the main system should have lvmetad and lvm2-pvscan services enabled for event based autoactivation.
I've found that there is clearly a problem with the new commit "pvscan: avoid redundant activation". I'll push out a fix today or tomorrow. It should at least help this issue.
A temp branch with the fix I'm testing: https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=6b12930860a993624d6325aec2e9c561f4412aa9
Pushed the fix to the stable-2.02 branch https://sourceware.org/git/?p=lvm2.git;a=commit;h=6b12930860a993624d6325aec2e9c561f4412aa9 With this change, pvscan should only activate LVs in a complete VG. I think it should be workable to use pvscan to activate LVs in the initramfs if that's what archlinux wants to do (contrary to what I wrote above). Note that it would work differently (and more correctly) than the sequence I tried to describe in comment 3 (1-7). If the root VG has two PVs, it should look something like: 1. first PV appears on the system 2. initramfs runs pvscan --cache -aay on the first PV, activates nothing 3. second PV appears on the system 4. initramfs runs pvscan --cache -aay on the second PV, activates all LVs in the root VG 5. switch-root happens 6. if pvscan --cache -aay is run again on the same PVs, the pvscans should skip activating the root VG again because of the /run/lvm/vgs_online temp file. (unless the content of /run/lvm/ is lost when doing switch root)
Hi David, (In reply to David Teigland from comment #6) > Pushed the fix to the stable-2.02 branch > https://sourceware.org/git/?p=lvm2.git;a=commit; > h=6b12930860a993624d6325aec2e9c561f4412aa9 > > With this change, pvscan should only activate LVs in a complete VG. > May I ask why this restriction has been introduced recently? > I think > it should be workable to use pvscan to activate LVs in the initramfs if > that's what archlinux wants to do (contrary to what I wrote above). Note > that it would work differently (and more correctly) than the sequence I > tried to describe in comment 3 (1-7). > > If the root VG has two PVs, it should look something like: > > 1. first PV appears on the system > 2. initramfs runs pvscan --cache -aay on the first PV, activates nothing > 3. second PV appears on the system > 4. initramfs runs pvscan --cache -aay on the second PV, activates all LVs in > the root VG > 5. switch-root happens > 6. if pvscan --cache -aay is run again on the same PVs, the pvscans should > skip activating the root VG again because of the /run/lvm/vgs_online temp > file. (unless the content of /run/lvm/ is lost when doing switch root) This won't work if a given root VG is composed of several PVs, which all but one are not visible until switch-root happens and e.g. systemd activates underlying LUKS devices using /etc/crypttab.
(In reply to Aleksander Trofimowicz from comment #7) > Hi David, > > (In reply to David Teigland from comment #6) > > Pushed the fix to the stable-2.02 branch > > https://sourceware.org/git/?p=lvm2.git;a=commit; > > h=6b12930860a993624d6325aec2e9c561f4412aa9 > > > > With this change, pvscan should only activate LVs in a complete VG. > > > May I ask why this restriction has been introduced recently? As far as I know, trying to autoactivate an incomplete VG was never intended, it was a bug (I haven't gone back to find where the regression appeared.) If there's some reference somewhere that indicates otherwise, I'd be happy to read that. Perhaps some special cases or exceptions have been made for activating the root LV? We could look into adding this as a feature, but we'd need to think about how that should work, when/why it would be used, etc. > This won't work if a given root VG is composed of several PVs, which all but > one are > not visible until switch-root happens and e.g. systemd activates underlying > LUKS devices using > /etc/crypttab. OK, I wonder how it was meant to work (without depending on a pvscan bug), someone must have thought about it at some point and can explain it to me. I'm not familiar with the various approaches (or hacks) that have been used in initramfs previously.
(In reply to David Teigland from comment #8) > (In reply to Aleksander Trofimowicz from comment #7) > > May I ask why this restriction has been introduced recently? > > As far as I know, trying to autoactivate an incomplete VG was never > intended, it was a bug (I haven't gone back to find where the regression > appeared.) If there's some reference somewhere that indicates otherwise, > I'd be happy to read that. Perhaps some special cases or exceptions have > been made for activating the root LV? > There is a notion of partial VGs to be found in man pages, but is it a synonym for an incomplete VG? > We could look into adding this as a feature, but we'd need to think about > how that should work, when/why it would be used, etc. > Would you consider reverting the change until these uses cases are ironed out? Or at least a reasonable grace period. There was no early warning at all, and user systems are broken now.
> There is a notion of partial VGs to be found in man pages, but is it a > synonym for an incomplete VG? The term "partial" refers in all cases I've seen partial LVs, where a PV used by an LV is missing. Handling of incomplete VGs is an area that seems to never have been well defined. > Would you consider reverting the change until these uses cases are ironed > out? > Or at least a reasonable grace period. > > There was no early warning at all, and user systems are broken now. This comment indicates it works again in one case: https://github.com/lvmteam/lvm2/issues/24#issuecomment-528149827 If there are other cases that don't work with all the latest changes, I'll look at the details and see if there's some other change we can do to make them work.
Concept of 'partial' activation goes along these notes: Lvm2 so far does not support 'normal' work with VG with incomplete PV set. The flag '--partial' was meant to be used for activation of LVs with PVs are missing in VG. The primary concept comes for 'clustering' support - where you activate on node A and activation happens on node B and you don't know what is missing on node B. So either you get 'normal' activation (all PVs are there) or you can try 'fail-recover' variant with '--partial' activation. This rules were quite simple to the point where 'partial' activation started to be used for activation of raid/mirror LVs with missing legs without modification of lvm2 metadata (IMHO this was always design bug and should be fixed). The concept of 'component' activation of LVs on PVs that are 'ready' while whole VG isn't there is not supported (and would be kind of useless and buggy). We simply need to see ALL PVs since we don't know WHICH of the PVs has the last metadata. So requesting to support this kind of activation doesn't make much sense (except maybe couple millisecond faster boot) - but this doesn't pay off compared with lot's of trouble we could have if the 'later' arrived PV would have newer metadata. Also note '--partial' was only meant as activation for 'recovery' purposes - so it should be ONLY activated data 'rescued' and deactivated - there is not much support to continue to work with partial activated LVs - they are simply meant to be deactivated right after that sort of use (and raid/mirror needs to be fixed)
(In reply to Aleksander Trofimowicz from comment #7) > Hi David, > > (In reply to David Teigland from comment #6) > > Pushed the fix to the stable-2.02 branch > > https://sourceware.org/git/?p=lvm2.git;a=commit; > > h=6b12930860a993624d6325aec2e9c561f4412aa9 > > > > With this change, pvscan should only activate LVs in a complete VG. > > > May I ask why this restriction has been introduced recently? > > > I think > > it should be workable to use pvscan to activate LVs in the initramfs if > > that's what archlinux wants to do (contrary to what I wrote above). Note > > that it would work differently (and more correctly) than the sequence I > > tried to describe in comment 3 (1-7). > > > > If the root VG has two PVs, it should look something like: > > > > 1. first PV appears on the system > > 2. initramfs runs pvscan --cache -aay on the first PV, activates nothing > > 3. second PV appears on the system > > 4. initramfs runs pvscan --cache -aay on the second PV, activates all LVs in > > the root VG > > 5. switch-root happens > > 6. if pvscan --cache -aay is run again on the same PVs, the pvscans should > > skip activating the root VG again because of the /run/lvm/vgs_online temp > > file. (unless the content of /run/lvm/ is lost when doing switch root) > > This won't work if a given root VG is composed of several PVs, which all but > one are > not visible until switch-root happens and e.g. systemd activates underlying > LUKS devices using > /etc/crypttab. This sounds like strange setup to me. If you require more than one LUKS device (encrypted PV in this case) to properly assembly specific VG hosting rootfs, I'd expect all those LUKS devices specified on kernel command line (rd.luks...). What if you extend rootfs on another PV later? Multiple rd.luks... arguments are supported by initramfs/systemd as far as I know and this would solved the problem properly. Is it Archlinux specific limitation that only single LUKS is being activated during boot or what's the reason for it exactly? Regards O.
Hi, (In reply to Ondrej Kozina from comment #12) > > This sounds like strange setup to me. If you require more than one LUKS > device (encrypted PV in this case) to properly assembly specific VG hosting > rootfs, I'd expect all those LUKS devices specified on kernel command line > (rd.luks...). > And that many passwords to be provided at the boot time? I can live with an idea that a one encrypted PV/FS pair holds keys for the others, as my major concerns are addressed anyway. > What if you extend rootfs on another PV later? Multiple rd.luks... arguments > are supported by initramfs/systemd as far as I know and this would solved > the problem properly. > I would have hard time arguing that typing more is a proper solution. Just to be clear: rootfs is not stored on another PV. > Is it Archlinux specific limitation that only single LUKS is being activated > during boot or what's the reason for it exactly? > I tend to be lazy, so if a one password could do the job, then why not? Thanks, Aleksander
(In reply to Aleksander Trofimowicz from comment #13) > Hi, > > (In reply to Ondrej Kozina from comment #12) > > > > This sounds like strange setup to me. If you require more than one LUKS > > device (encrypted PV in this case) to properly assembly specific VG hosting > > rootfs, I'd expect all those LUKS devices specified on kernel command line > > (rd.luks...). > > > And that many passwords to be provided at the boot time? I can live with an > idea that a one encrypted PV/FS pair holds keys for the others, as my major > concerns are addressed anyway. The simple advice would be to put rootfs LV on VG that uses a single encrypted PV. As mentioned in comment 11 - the standard activation is supposed to work ONLY when all PVs from a VG are available. Lvm2 simply cannot know which of the PV has the latest metadata and would be kind of 'bad idea' to activation LV by version A and when the newer version with next PV arrives (especially with raids/mirrors....) So if someone had designed activation of rootfs based on arrival of a single PV which would look like the only provider of physical segments for a particular LV - made an invalid design (from lvm2 perspective). > I would have hard time arguing that typing more is a proper solution. Just > to be clear: rootfs is not stored on another PV. VG is meant to be used ONLY when all PVs are present - the use with 'partial' activation is purely a recovery variant and must never be used (seriously!) in a standard environment. Simple rule: With current design of lvm2 normal activation needs whole VG - if this has ever worked differently in past - it's been a pure bug we can't support for safety reason. AFAIK 'pvscan -aay' never uses partial activation - unless there was a temporary bug present. > > Is it Archlinux specific limitation that only single LUKS is being activated > > during boot or what's the reason for it exactly? > > > I tend to be lazy, so if a one password could do the job, then why not? If it's possible and rootfs runs from a single disk - maybe you can run a 'vgsplit' command and have two separate VGs - then you can easily activate rootfs by unlocking a single PV. Of course you get the limitation, that you no longer can use space betweeen LVs in different VGs.
Hi, (In reply to Zdenek Kabelac from comment #14) > > So if someone had designed activation of rootfs based on arrival of a single > PV which would look like the only provider of physical segments for a > particular LV - made an invalid design (from lvm2 perspective). >w Point taken. However if existing behaviour was considered erroneous, I would expect to see runtime deprecation warning for at least two releases before an actual change makes its way into the code. IIRC a VG with incomplete PV set could have been activated without resorting to the partial mode on RHEL5/7 too. > If it's possible and rootfs runs from a single disk - maybe you can run a > 'vgsplit' command and have two separate > VGs - then you can easily activate rootfs by unlocking a single PV. Of > course you get the limitation, that you no longer can use space betweeen LVs > in different VGs. > That is a solution for me. Yet an idea there are multiple VGs on a single end user host isn't quite elegant. Anyway thank you all for the discussion.