Bug 1036437

Description of problem:

If a USB-connected external disk is disconnected, LVM does not seem to properly understand that its logical volumes are no longer present.  In addition, if the disk is reconnected, LVM does not seem to regain access to the logical volumes.

(This would not be very relevant if the logical volumes were mounted at the time, but it happens even if the logical volumes are not mounted.)

Version-Release number of selected component (if applicable):

kernel 3.11.9-200.fc19.x86_64

How reproducible:

reliably reproducible

Steps to Reproduce:
(summary)
1. Connect USB disk.
2. LVM logical volumes become accessible.
3. Disconnect USB disk.
4. LVM logical volumes become inaccessible (of course), but LVM does not seem to recognize this properly, e.g., /dev/mapper entries for the removed logical volumes remain present.
5. Connect USB disk.
6. LVM logical volumes are not accessible.
(detailed script is below)

Actual results:

When USB disk is disconnected, LVM does not seem to recognize this properly, e.g., /dev/mapper entries for the removed logical volumes remain present.
After USB disk is reconnected, LVM logical volumes are not accessible.

Expected results:

When USB disk is disconnected, LVM should remove the /dev entries for its logical volumes.
When USB disk is reconnected, LVM should recreate the /dev entries, and they should access the logical volumes.

Additional info:

Here is a script of a typical sequence of actions:

# uname -a
Linux hobgoblin.ariadne.com 3.11.9-200.fc19.x86_64 #1 SMP Wed Nov 20 21:22:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
Fedora release 19 (Schrödinger’s Cat)
# # Start with external USB disk disconnected.
# ll /dev/mapper
total 0
drwxr-xr-x.  2 root root     140 Dec  1 19:11 .
drwxr-xr-x. 19 root root    3580 Dec  1 19:12 ..
lrwxrwxrwx.  1 root root       7 Dec  1 19:11 Hobgoblin00-RootF16 -> ../dm-3
lrwxrwxrwx.  1 root root       7 Dec  1 19:11 Hobgoblin00-RootF19 -> ../dm-1
lrwxrwxrwx.  1 root root       7 Dec  1 19:11 Hobgoblin00-Swap -> ../dm-0
lrwxrwxrwx.  1 root root       7 Dec  1 19:12 Hobgoblin00-User -> ../dm-2
crw-------.  1 root root 10, 236 Dec  1 19:11 control
# Connect external USB disk.
# vgscan ; vgchange -ay
  Reading all physical volumes.  This may take a while...
  Found volume group "Freeze01" using metadata type lvm2
  Found volume group "Hobgoblin00" using metadata type lvm2
  1 logical volume(s) in volume group "Freeze01" now active
  4 logical volume(s) in volume group "Hobgoblin00" now active
# ll /dev/mapper
total 0
lrwxrwxrwx. 1 root root       7 Dec  1 19:14 Freeze01-Store -> ../dm-4
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-RootF16 -> ../dm-3
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-RootF19 -> ../dm-1
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-Swap -> ../dm-0
lrwxrwxrwx. 1 root root       7 Dec  1 19:12 Hobgoblin00-User -> ../dm-2
crw-------. 1 root root 10, 236 Dec  1 19:11 control
# dd if=/dev/mapper/Freeze01-Store of=/dev/null bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000937973 s, 546 kB/s
# # Disconnect external USB disk.
# vgscan ; vgchange -ay
  Reading all physical volumes.  This may take a while...
  Found volume group "Hobgoblin00" using metadata type lvm2
  4 logical volume(s) in volume group "Hobgoblin00" now active
# ll /dev/mapper
total 0
lrwxrwxrwx. 1 root root       7 Dec  1 19:14 Freeze01-Store -> ../dm-4
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-RootF16 -> ../dm-3
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-RootF19 -> ../dm-1
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-Swap -> ../dm-0
lrwxrwxrwx. 1 root root       7 Dec  1 19:12 Hobgoblin00-User -> ../dm-2
crw-------. 1 root root 10, 236 Dec  1 19:11 control
# dd if=/dev/mapper/Freeze01-Store of=/dev/null bs=512 count=1
dd: error reading ‘/dev/mapper/Freeze01-Store’: Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000627597 s, 0.0 kB/s
# vgscan ; vgchange -ay
  Reading all physical volumes.  This may take a while...
  Found volume group "Freeze01" using metadata type lvm2
  Found volume group "Hobgoblin00" using metadata type lvm2
  1 logical volume(s) in volume group "Freeze01" now active
  4 logical volume(s) in volume group "Hobgoblin00" now active
# ll /dev/mapper
total 0
lrwxrwxrwx. 1 root root       7 Dec  1 19:14 Freeze01-Store -> ../dm-4
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-RootF16 -> ../dm-3
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-RootF19 -> ../dm-1
lrwxrwxrwx. 1 root root       7 Dec  1 19:11 Hobgoblin00-Swap -> ../dm-0
lrwxrwxrwx. 1 root root       7 Dec  1 19:12 Hobgoblin00-User -> ../dm-2
crw-------. 1 root root 10, 236 Dec  1 19:11 control
# dd if=/dev/mapper/Freeze01-Store of=/dev/null bs=512 count=1
dd: error reading ‘/dev/mapper/Freeze01-Store’: Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000247883 s, 0.0 kB/s
# 

(BTW, there is some data field in the "create new bug" page which causes a page-submit when you type Return in it.  This is Not A Good Thing.)

(In reply to Dale R. Worley from comment #1)
> # Connect external USB disk.
> # vgscan ; vgchange -ay
>   Reading all physical volumes.  This may take a while...
>   Found volume group "Freeze01" using metadata type lvm2
>   Found volume group "Hobgoblin00" using metadata type lvm2
>   1 logical volume(s) in volume group "Freeze01" now active
>   4 logical volume(s) in volume group "Hobgoblin00" now active

The vgchange -ay is not actually needed when lvmetad daemon is used (global/use_lvmetad=1 in /etc/lvm/lvm.conf) - which is now used by default in F19.

Anyway, the problem here is known: we don't have LVM autodeactivation yet.

The LVM autodeactivation is a bit more complex as for this to be properly implemented, we need to deactivate the whole stack of devices from top to bottom. The LV can be just a part of the stack above which any other block device subsystem can be layered - there are no restrictions when it comes to the height of this stack and device composition.

So this problem is more a general one, encompassing the whole block device stack and all the possible block device subsystems, not just LVM.

However, we could support this partially - there's a new "blkdeactivate" script that deals with block device stack deactivation (normally used at shutdown) - currently it supports device-mapper/LVM devices only (so it would not work if there's any other type of the device included the stack). But I agree that majority of the situations are with a simple layout with a disk at the bottom on top of which LVM is defined and no more levels above it - so we could cover most situations with calling blkdeactivate on device removal.


Also, in recent versions of LVM (version >= 2.02.101), we always do a VG refresh before autoactivation which corrects any stale LV mappings, for example:

 - device plugged, LVs autoactivated

 - device unplugged (but LV mappings still present in the system as there's no autodeactivation on "device/PV unplugged" event, giving IO errors if accessed)

 - device plugged again, VG where LVs belong to is refreshed - this way the LV mappings are updated to be up to date with kernel state (major:minor device number)


As for the autodeactivation hook - we could probably think of that as an RFE for future version of Fedora (F21+). As for the VG refresh before autoactivation - I could backport that for F19 LVM - that's quite straightforward and easy patch.

(In reply to Peter Rajnoha from comment #2)
> As for the autodeactivation hook - we could probably think of that as an RFE
> for future version of Fedora (F21+). As for the VG refresh before
> autoactivation - I could backport that for F19 LVM - that's quite
> straightforward and easy patch.

I've opened a FutureFeature bug #1038608 for the autodeactivation hook. As for the "VG refresh before autodeactivation" - I can include that in next lvm2 update for F19.

Well, I really recommend upgrading to F20 where lvmetad is used by default (which enables LVM volume autoactivation, including the refresh before the autoactivation) and it also contain many fixes for lvmetad itself.

Unfortunately, I don't know a lot about LVM.  I think I understand the general problem regarding removing LVM volumes.  But that doesn't tell me how to solve the practical problem I have, which is: I have a USB LVM volume, and I wish to unplug it and at a later time plug it back in and then use it (without having to reboot in between).  What sequence of actions can I take that allows this scenario to work?

It looks like the version of LVM in Fedora 20 has this problem solved in a clean way.  But the discussion suggests that there is a way to make it work in the current Fedora 19.  Unfortunately, it isn't clear to me what that way is.

(In reply to Dale R. Worley from comment #6)
> Unfortunately, I don't know a lot about LVM.  I think I understand the
> general problem regarding removing LVM volumes.  But that doesn't tell me
> how to solve the practical problem I have, which is: I have a USB LVM
> volume, and I wish to unplug it and at a later time plug it back in and then
> use it (without having to reboot in between).  What sequence of actions can
> I take that allows this scenario to work?
> 

Calling "vgchange --refresh <vgname>" (or if not specifying the vgname, all VGs are refreshed then) after you plug the device back. This should refresh any existing VGs/LVs, including changed major:minor pair for the PV device that got unplugged and then plugged back.

> It looks like the version of LVM in Fedora 20 has this problem solved in a
> clean way.  But the discussion suggests that there is a way to make it work

F20 calls the "vgchange --refresh" automatically... But as we've found out recently, calling this automatically is not the perfect solution either since it's *always* called, even when not necessary - we're just fixing this.

So for the F19, just call that "vgchange --refresh" manually. That should do the job.

(In reply to Peter Rajnoha from comment #7)
> Calling "vgchange --refresh <vgname>" [...]

Many thanks!