800801 – Device stack including LVM and multipath should be properly deactivated on shutdown

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 800801 - Device stack including LVM and multipath should be properly deactivated on shutdown

Summary: Device stack including LVM and multipath should be properly deactivated on sh...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	6.4
Hardware:	All
OS:	Linux
Priority:	medium
Severity:	high
Target Milestone:	beta
Target Release:	6.4
Assignee:	Peter Rajnoha
QA Contact:	Cluster QE
Docs Contact:
URL:
Whiteboard:
Duplicates (2):	784584 892039 (view as bug list)
Depends On:	672530
Blocks:	784640 905536
TreeView+	depends on / blocked

Reported:	2012-03-07 10:06 UTC by Peter Rajnoha
Modified:	2018-12-03 17:30 UTC (History)
CC List:	37 users (show)
Fixed In Version:	lvm2-2.02.98-9.el6
Doc Type:	Enhancement
Doc Text:	Cause: Device-mapper devices (including LVM devices) were not deactivated at system shutdown or reboot. Consequence: When device-mapper devices were layered on top of other block devices and these were detached during the shutdown/reboot procedure, any further access to the device-mapper devices ended up with either IO errors or even a hang as the underlying devices were gone (e.g iSCSI or FCoE devices). Fix: A new blkdeactivate script together with the accompanying blk-availability shutdown script has been provided that unmounts and deactivates any existing device-mapper devices before deactivating and detaching the underlying devices on shutdown/reboot. Result: Clean shutdown/reboot without IO errors or hangs if using attached storage that detaches itself during shutdown/reboot procedure.
Clone Of:	672530
Environment:
Last Closed:	2013-02-21 08:03:30 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2013:0501	0	normal	SHIPPED_LIVE	lvm2 bug fix and enhancement update	2013-02-20 21:30:45 UTC

Description Peter Rajnoha 2012-03-07 10:06:15 UTC

+++ This bug was initially created as a clone of Bug #672530 +++
--- Additional comment from prajnoha on 2011-08-01 14:12:15 CEST ---
...
A script to deactivate any remaining LVs/VGs should still be provided though with the exception of the LV on which root fs and any system fs resides (together with unmounting any filesystems using them and any layered devices/mappings underneath). I'll try to complete the script using existing lsblk output that shows the device tree already based on information found in sysfs. The lsblk needs a small patch to make the output more suitable for parsing...

--- Additional comment from dwysocha on 2011-08-08 21:35:37 CEST ---

I've put together a Kbase article covering this issue, and a couple workarounds.  The current public version is here: https://access.redhat.com/kb/docs/DOC-60763

Comment 2 Peter Rajnoha 2012-03-07 10:13:44 UTC

The lsblk patch is under review/waiting upstream acceptance...

Comment 3 Peter Rajnoha 2012-03-23 07:05:29 UTC

The lsblk patch is upstream now (should appear in upcoming util-linux 2.22). I'll try to negotiate with initscripts maintainer to have the script using lsblk output for proper device shutdown included (as this is a general problem with stacked devices, not just lvm one).

Comment 4 Peter Rajnoha 2012-03-29 08:56:19 UTC

*** Bug 784584 has been marked as a duplicate of this bug. ***

Comment 5 Larry Troan 2012-06-19 00:17:24 UTC

Bug 784584 was CLOSED=DUP of this bug but the trackers were not carried forward.

I added Hitachi6.4bugs and the GSS_6_3_Proposed trackers but the latter needs to be updated to GSS_6_4_Proposed.

Comment 6 Jeremy West 2012-06-19 13:39:35 UTC

Thanks Larry. I've corrected it.

Comment 7 Mitsuhiro Tanino 2012-09-14 13:16:48 UTC

Hi Peter-san,

Could you share upstream URL of this bugfix?
Also could you share latest status?

Regards,
Mitsuhiro Tanino

Comment 8 Peter Rajnoha 2012-10-09 13:26:59 UTC

The script is not upstream yet, it'll be committed soon (during this week), I'll add documentation and a bit of a cleanup. I'll update once this is done.

Comment 9 Mitsuhiro Tanino 2012-10-09 13:37:51 UTC

Hi Peter-san,

Thank you for your update.
If you will have any update from now, please share it us.

Regards,
Mitsuhiro Tanino

Comment 11 Peter Rajnoha 2012-10-12 08:56:28 UTC

*** Bug 784584 has been marked as a duplicate of this bug. ***

Comment 12 Peter Rajnoha 2012-10-12 13:03:03 UTC

Patch introducing the "blkdeactivate" utility together with initscript/systemd unit to automatically call it on shutdown has been committed upstream (it'll be a part of lvm2 v2_02_98):

http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=c698ee14bbb1310cf2383c8977d14a8e29139f8c
http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=a70d6b350f98bd6e9068ce723a275f31e9f27b7e

The required lsblk patch is upstream since util-linux v 2.22 and backported to RHEL6.4 within the util-linux-ng-2.17.2-12.8.el6 build.

The blkdeactivate utility will be included in next lvm2 build for 6.4.

For now, the utility deactivates all the device-mapper-based (including LVM) devices. Support for more device types may be included later (e.g. the mdraid) - it should be fairly easy to do so, but let's have the basic version working now.

Comment 20 Peter Rajnoha 2012-10-15 08:21:42 UTC

To QA:

To test this, create an arbitrary stack of device-mapper-based (pure dm, mpath, cryptsetup...) devices arbitrarily stacked one over the other together with LVM and then use the new blkdeactivate script.

Example:
========
# lsblk -s
NAME                            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sdb                               8:16   0  128M  0 disk  
sdc                               8:32   0  128M  0 disk  
sde                               8:64   0  128M  0 disk  
sdj                               8:144  0  128M  0 disk  
sdk                               8:160  0  128M  0 disk  
sdl                               8:176  0  128M  0 disk  
sdm                               8:192  0  128M  0 disk  
sdn                               8:208  0  128M  0 disk  
sdo                               8:224  0  128M  0 disk  
sdp                               8:240  0  128M  0 disk  
sdq                              65:0    0  128M  0 disk  
sdr                              65:16   0  128M  0 disk  
sds                              65:32   0  128M  0 disk  
sdt                              65:48   0  128M  0 disk  
sdu                              65:64   0  128M  0 disk  
sdv                              65:80   0  128M  0 disk  
sdw                              65:96   0  128M  0 disk  
sdx                              65:112  0  128M  0 disk  
sdy                              65:128  0  128M  0 disk  
sdz                              65:144  0  128M  0 disk  
vda1                            252:1    0  512M  0 part  /boot
`-vda                           252:0    0   20G  0 disk  
vda2                            252:2    0   19G  0 part  /
`-vda                           252:0    0   20G  0 disk  
vda3                            252:3    0  512M  0 part  [SWAP]
`-vda                           252:0    0   20G  0 disk  
vg2-lvol0 (dm-6)                253:6    0  244M  0 lvm   /mnt/temp
|-sdg                             8:96   0  128M  0 disk  
`-crypted_one (dm-5)            253:5    0  122M  0 crypt 
  `-vg1-lvol0 (dm-4)            253:4    0  124M  0 lvm   
    |-vg1-lvol0_mlog (dm-1)     253:1    0    4M  0 lvm   
    | `-vg-lvol0 (dm-0)         253:0    0   32M  0 lvm   
    |   `-sda                     8:0    0  128M  0 disk  
    |-vg1-lvol0_mimage_0 (dm-2) 253:2    0  124M  0 lvm   
    | `-sdd                       8:48   0  128M  0 disk  
    `-vg1-lvol0_mimage_1 (dm-3) 253:3    0  124M  0 lvm   
      `-sdf                       8:80   0  128M  0 disk  
vg3-lvol0 (dm-8)                253:8    0   28M  0 lvm   /mnt/temp1
`-mpath_one (dm-7)              253:7    0   32M  0 dm    
  |-sdh                           8:112  0  128M  0 disk  
  `-sdi                           8:128  0  128M  0 disk  

# blkdeactivate
(this should cause the deactivation to be skipped as we haven't used the "-u"/"--umount" option and the devs at the top of each stack are mounted)
Deactivating block devices:
  [SKIP]: unmount of vg2-lvol0 (dm-6) mounted on /mnt/temp
  [SKIP]: unmount of vg3-lvol0 (dm-8) mounted on /mnt/temp1

# blkdeactivate -u
(the same, but with the "-u"/"--umount" option used - all should be deactivated as the devs on top will be unmounted as well)
Deactivating block devices:
  UMOUNT: unmounting vg2-lvol0 (dm-6) mounted on /mnt/temp
  LVM: deactivating Logical Volume vg2/lvol0
  DM: deactivating crypt device crypted_one (dm-5)
  LVM: deactivating Logical Volume vg1/lvol0
  LVM: deactivating Logical Volume vg/lvol0
  UMOUNT: unmounting vg3-lvol0 (dm-8) mounted on /mnt/temp1
  LVM: deactivating Logical Volume vg3/lvol0
  DM: deactivating dm device mpath_one (dm-7)

# lsblk -s
NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda     8:0    0  128M  0 disk 
sdb     8:16   0  128M  0 disk 
sdc     8:32   0  128M  0 disk 
sdd     8:48   0  128M  0 disk 
sde     8:64   0  128M  0 disk 
sdf     8:80   0  128M  0 disk 
sdg     8:96   0  128M  0 disk 
sdh     8:112  0  128M  0 disk 
sdi     8:128  0  128M  0 disk 
sdj     8:144  0  128M  0 disk 
sdk     8:160  0  128M  0 disk 
sdl     8:176  0  128M  0 disk 
sdm     8:192  0  128M  0 disk 
sdn     8:208  0  128M  0 disk 
sdo     8:224  0  128M  0 disk 
sdp     8:240  0  128M  0 disk 
sdq    65:0    0  128M  0 disk 
sdr    65:16   0  128M  0 disk 
sds    65:32   0  128M  0 disk 
sdt    65:48   0  128M  0 disk 
sdu    65:64   0  128M  0 disk 
sdv    65:80   0  128M  0 disk 
sdw    65:96   0  128M  0 disk 
sdx    65:112  0  128M  0 disk 
sdy    65:128  0  128M  0 disk 
sdz    65:144  0  128M  0 disk 
vda1  252:1    0  512M  0 part /boot
`-vda 252:0    0   20G  0 disk 
vda2  252:2    0   19G  0 part /
`-vda 252:0    0   20G  0 disk 
vda3  252:3    0  512M  0 part [SWAP]
`-vda 252:0    0   20G  0 disk 


============================================
(create the same stack again)

# blkdeactivate -u -l wholevg
(the -l wholevg option will cause the whole VGs to be deactivated at once instead of separate LV deactivation)
Deactivating block devices:
  UMOUNT: unmounting vg2-lvol0 (dm-7) mounted on /mnt/temp
  LVM: deactivating Volume Group vg2
    0 logical volume(s) in volume group "vg2" now active
  DM: deactivating crypt device crypted_one (dm-5)
  LVM: deactivating Volume Group vg1
    0 logical volume(s) in volume group "vg1" now active
  LVM: deactivating Volume Group vg
    0 logical volume(s) in volume group "vg" now active
  UMOUNT: unmounting vg3-lvol0 (dm-8) mounted on /mnt/temp1
  LVM: deactivating Volume Group vg3
    0 logical volume(s) in volume group "vg3" now active
  DM: deactivating dm device mpath_one (dm-6)

[0] rawhide/~ # lsblk -s
NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda     8:0    0  128M  0 disk 
sdb     8:16   0  128M  0 disk 
sdc     8:32   0  128M  0 disk 
sdd     8:48   0  128M  0 disk 
sde     8:64   0  128M  0 disk 
sdf     8:80   0  128M  0 disk 
sdg     8:96   0  128M  0 disk 
sdh     8:112  0  128M  0 disk 
sdi     8:128  0  128M  0 disk 
sdj     8:144  0  128M  0 disk 
sdk     8:160  0  128M  0 disk 
sdl     8:176  0  128M  0 disk 
sdm     8:192  0  128M  0 disk 
sdn     8:208  0  128M  0 disk 
sdo     8:224  0  128M  0 disk 
sdp     8:240  0  128M  0 disk 
sdq    65:0    0  128M  0 disk 
sdr    65:16   0  128M  0 disk 
sds    65:32   0  128M  0 disk 
sdt    65:48   0  128M  0 disk 
sdu    65:64   0  128M  0 disk 
sdv    65:80   0  128M  0 disk 
sdw    65:96   0  128M  0 disk 
sdx    65:112  0  128M  0 disk 
sdy    65:128  0  128M  0 disk 
sdz    65:144  0  128M  0 disk 
vda1  252:1    0  512M  0 part /boot
`-vda 252:0    0   20G  0 disk 
vda2  252:2    0   19G  0 part /
`-vda 252:0    0   20G  0 disk 
vda3  252:3    0  512M  0 part [SWAP]
`-vda 252:0    0   20G  0 disk 

============================================
(create the same stack again + added one more mapping "zero_one")

# lsblk -s
NAME                            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sdb                               8:16   0  128M  0 disk  
sdc                               8:32   0  128M  0 disk  
sde                               8:64   0  128M  0 disk  
sdk                               8:160  0  128M  0 disk  
sdl                               8:176  0  128M  0 disk  
sdm                               8:192  0  128M  0 disk  
sdn                               8:208  0  128M  0 disk  
sdo                               8:224  0  128M  0 disk  
sdp                               8:240  0  128M  0 disk  
sdq                              65:0    0  128M  0 disk  
sdr                              65:16   0  128M  0 disk  
sds                              65:32   0  128M  0 disk  
sdt                              65:48   0  128M  0 disk  
sdu                              65:64   0  128M  0 disk  
sdv                              65:80   0  128M  0 disk  
sdw                              65:96   0  128M  0 disk  
sdx                              65:112  0  128M  0 disk  
sdy                              65:128  0  128M  0 disk  
sdz                              65:144  0  128M  0 disk  
vda1                            252:1    0  512M  0 part  /boot
`-vda                           252:0    0   20G  0 disk  
vda2                            252:2    0   19G  0 part  /
`-vda                           252:0    0   20G  0 disk  
vda3                            252:3    0  512M  0 part  [SWAP]
`-vda                           252:0    0   20G  0 disk  
vg2-lvol0 (dm-6)                253:6    0  244M  0 lvm   /mnt/temp
|-sdg                             8:96   0  128M  0 disk  
`-crypted_one (dm-5)            253:5    0  122M  0 crypt 
  `-vg1-lvol0 (dm-4)            253:4    0  124M  0 lvm   
    |-vg1-lvol0_mlog (dm-1)     253:1    0    4M  0 lvm   
    | `-vg-lvol0 (dm-0)         253:0    0   32M  0 lvm   
    |   `-sda                     8:0    0  128M  0 disk  
    |-vg1-lvol0_mimage_0 (dm-2) 253:2    0  124M  0 lvm   
    | `-sdd                       8:48   0  128M  0 disk  
    `-vg1-lvol0_mimage_1 (dm-3) 253:3    0  124M  0 lvm   
      `-sdf                       8:80   0  128M  0 disk  
zero_one (dm-8)                 253:8    0   32M  0 dm    
`-sdj                             8:144  0  128M  0 disk  
vg3-lvol0 (dm-9)                253:9    0   28M  0 lvm   /mnt/temp1
`-mpath_one (dm-7)              253:7    0   32M  0 dm    
  |-sdh                           8:112  0  128M  0 disk  
  `-sdi                           8:128  0  128M  0 disk  

[0] rawhide/~ # blkdeactivate -u -l wholevg /dev/mapper/mpath_one /dev/mapper/zero_one
(specifying the devs to deactivate directly on the command line, if the dev is in the middle of the device stack, all holders of the dev will be deactivated first - like the /dev/mapper/mpath_one which has vg-lvol0 stacked on top)
Deactivating block devices:
  UMOUNT: unmounting vg3-lvol0 (dm-7) mounted on /mnt/temp1
  LVM: deactivating Volume Group vg3
    0 logical volume(s) in volume group "vg3" now active
  DM: deactivating dm device mpath_one (dm-8)
  DM: deactivating dm device zero_one (dm-9)


# lsblk -s
(the vg2-lvol0 and downwards the stack stays untouched - no dev specified on the command line for blkdeactivate above is included in the stack)
NAME                            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sdb                               8:16   0  128M  0 disk  
sdc                               8:32   0  128M  0 disk  
sde                               8:64   0  128M  0 disk  
sdh                               8:112  0  128M  0 disk  
sdi                               8:128  0  128M  0 disk  
sdj                               8:144  0  128M  0 disk  
sdk                               8:160  0  128M  0 disk  
sdl                               8:176  0  128M  0 disk  
sdm                               8:192  0  128M  0 disk  
sdn                               8:208  0  128M  0 disk  
sdo                               8:224  0  128M  0 disk  
sdp                               8:240  0  128M  0 disk  
sdq                              65:0    0  128M  0 disk  
sdr                              65:16   0  128M  0 disk  
sds                              65:32   0  128M  0 disk  
sdt                              65:48   0  128M  0 disk  
sdu                              65:64   0  128M  0 disk  
sdv                              65:80   0  128M  0 disk  
sdw                              65:96   0  128M  0 disk  
sdx                              65:112  0  128M  0 disk  
sdy                              65:128  0  128M  0 disk  
sdz                              65:144  0  128M  0 disk  
vda1                            252:1    0  512M  0 part  /boot
`-vda                           252:0    0   20G  0 disk  
vda2                            252:2    0   19G  0 part  /
`-vda                           252:0    0   20G  0 disk  
vda3                            252:3    0  512M  0 part  [SWAP]
`-vda                           252:0    0   20G  0 disk  
vg2-lvol0 (dm-6)                253:6    0  244M  0 lvm   /mnt/temp
|-sdg                             8:96   0  128M  0 disk  
`-crypted_one (dm-5)            253:5    0  122M  0 crypt 
  `-vg1-lvol0 (dm-4)            253:4    0  124M  0 lvm   
    |-vg1-lvol0_mlog (dm-1)     253:1    0    4M  0 lvm   
    | `-vg-lvol0 (dm-0)         253:0    0   32M  0 lvm   
    |   `-sda                     8:0    0  128M  0 disk  
    |-vg1-lvol0_mimage_0 (dm-2) 253:2    0  124M  0 lvm   
    | `-sdd                       8:48   0  128M  0 disk  
    `-vg1-lvol0_mimage_1 (dm-3) 253:3    0  124M  0 lvm   
      `-sdf                       8:80   0  128M  0 disk  


============================================

This should be all done automatically by initscript called on shutdown if blk-availability initscript is enabled (this is actually a "shutdown" only initscript). Also, it should be done before the iscsi service is disabled.

N.B.: the swap, /boot, /, /lib, /lib64, /bin, /sbin and /usr mount (including subdirs /bin, /sbin, /lib, /lib64) is *not* unmounted with the -u option!

Comment 22 Peter Rajnoha 2012-10-15 12:08:17 UTC

(In reply to comment #20)
> This should be all done automatically by initscript called on shutdown if
> blk-availability initscript is enabled (this is actually a "shutdown" only
> initscript). Also, it should be done before the iscsi service is disabled.

Example snippet of shutdown (where there was one VG called "vg", just for an example...):

...
Stopping HAL daemon:                                      [  OK  ]
Stopping blk-availability Deactivating block devices:
  [SKIP]: unmount of vg_rhel6a-lv_root (dm-0) mounted on /
  LVM: deactivating Volume Group vg
   0 logical volume(s) in volume group "vg" now active
                                                          [  OK  ]
Stopping NetworkManager daemon:                           [  OK  ]
Stopping systemd message bus:                             [  OK  ]
Stopping rpcbind:                                         [  OK  ]
Stopping iscsi:                                           [  OK  ]
Shutting down system logger:                              [  OK  ]
...

Comment 23 Peter Rajnoha 2012-10-15 13:02:13 UTC

(In reply to comment #20)
> it should be done before the iscsi service is disabled.

..the same applies for fcoe service if enabled.

Comment 25 Peter Rajnoha 2013-01-07 08:20:24 UTC

*** Bug 892039 has been marked as a duplicate of this bug. ***

Comment 26 Nenad Peric 2013-01-22 17:16:39 UTC

Based on Comment #20 I created an arbitrarily stacked and mounted devices.

But the reboot is not what you would call pretty and graceful, and blkdeactivate -u needs to be ran several times in order to unmount and deactivate all LVs/VGs.

I think there is something to do with thin pool and thin LVs, and the order those are handled during this deactivation process. 

Setup (mountpoint did not really fit):

(11:13:53) [root@r6-node02:~]$ lsblk -s
NAME                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda1                                  252:1    0  500M  0 part /boot
└─vda                                 252:0    0   10G  0 disk 
VolGroup-lv_root (dm-0)               253:0    0  7.6G  0 lvm  /
└─vda2                                252:2    0  9.5G  0 part 
  └─vda                               252:0    0   10G  0 disk 
VolGroup-lv_swap (dm-1)               253:1    0    2G  0 lvm  [SWAP]
└─vda2                                252:2    0  9.5G  0 part 
  └─vda                               252:0    0   10G  0 disk 
sdb1                                    8:17   0   10G  0 part 
└─sdb                                   8:16   0   10G  0 disk 
sdc1                                    8:33   0   10G  0 part 
└─sdc                                   8:32   0   10G  0 disk 
sdf1                                    8:81   0   10G  0 part 
└─sdf                                   8:80   0   10G  0 disk 
sdi1                                    8:129  0   10G  0 part 
└─sdi                                   8:128  0   10G  0 disk 
thinny-pool (dm-5)                    253:5    0    5G  0 lvm  
└─thinny-pool-tpool (dm-4)            253:4    0    5G  0 lvm  
  ├─thinny-pool_tmeta (dm-2)          253:2    0    8M  0 lvm  
  │ └─sdh1                              8:113  0   10G  0 part 
  │   └─sdh                             8:112  0   10G  0 disk 
  └─thinny-pool_tdata (dm-3)          253:3    0    5G  0 lvm  
    └─sdh1                              8:113  0   10G  0 part 
      └─sdh                             8:112  0   10G  0 disk 
thinny-lvolthin (dm-6)                253:6    0    5G  0 lvm  /mnt/write_snap_1
└─thinny-pool-tpool (dm-4)            253:4    0    5G  0 lvm  
  ├─thinny-pool_tmeta (dm-2)          253:2    0    8M  0 lvm  
  │ └─sdh1                              8:113  0   10G  0 part 
  │   └─sdh                             8:112  0   10G  0 disk 
  └─thinny-pool_tdata (dm-3)          253:3    0    5G  0 lvm  
    └─sdh1                              8:113  0   10G  0 part 
      └─sdh                             8:112  0   10G  0 disk 
thinny-thinlv (dm-7)                  253:7    0    5G  0 lvm  /mnt/write_snap_1
└─thinny-pool-tpool (dm-4)            253:4    0    5G  0 lvm  
  ├─thinny-pool_tmeta (dm-2)          253:2    0    8M  0 lvm  
  │ └─sdh1                              8:113  0   10G  0 part 
  │   └─sdh                             8:112  0   10G  0 disk 
  └─thinny-pool_tdata (dm-3)          253:3    0    5G  0 lvm  
    └─sdh1                              8:113  0   10G  0 part 
      └─sdh                             8:112  0   10G  0 disk 
helter_skelter-syncd_secondary_core_4legs_1 (dm-14)
                                      253:14   0  500M  0 lvm  /mnt/syncd_primar
└─helter_skelter-syncd_secondary_core_4legs_1-real (dm-13)
                                      253:13   0  500M  0 lvm  
  ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_0 (dm-10)
                                      253:10   0  500M  0 lvm  
  │ └─sdj1                              8:145  0   10G  0 part 
  │   └─sdj                             8:144  0   10G  0 disk 
  ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_2 (dm-11)
                                      253:11   0  500M  0 lvm  
  │ └─sdg1                              8:97   0   10G  0 part 
  │   └─sdg                             8:96   0   10G  0 disk 
  └─helter_skelter-syncd_secondary_core_4legs_1_mimage_3 (dm-12)
                                      253:12   0  500M  0 lvm  
    └─sde1                              8:65   0   10G  0 part 
      └─sde                             8:64   0   10G  0 disk 
helter_skelter-hs_snap1 (dm-16)       253:16   0  500M  0 lvm  
├─helter_skelter-syncd_secondary_core_4legs_1-real (dm-13)
                                      253:13   0  500M  0 lvm  
│ ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_0 (dm-10)
                                      253:10   0  500M  0 lvm  
│ │ └─sdj1                              8:145  0   10G  0 part 
│ │   └─sdj                             8:144  0   10G  0 disk 
│ ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_2 (dm-11)
                                      253:11   0  500M  0 lvm  
│ │ └─sdg1                              8:97   0   10G  0 part 
│ │   └─sdg                             8:96   0   10G  0 disk 
│ └─helter_skelter-syncd_secondary_core_4legs_1_mimage_3 (dm-12)
                                      253:12   0  500M  0 lvm  
│   └─sde1                              8:65   0   10G  0 part 
│     └─sde                             8:64   0   10G  0 disk 
└─helter_skelter-hs_snap1-cow (dm-15) 253:15   0  252M  0 lvm  
  └─sdj1                                8:145  0   10G  0 part 
    └─sdj                               8:144  0   10G  0 disk 
raid-lvol0 (dm-21)                    253:21   0  120M  0 lvm  /mnt/write_snap_2
├─raid-lvol0_rmeta_0 (dm-17)          253:17   0    4M  0 lvm  
│ └─sda1                                8:1    0   10G  0 part 
│   └─sda                               8:0    0   10G  0 disk 
├─raid-lvol0_rimage_0 (dm-18)         253:18   0  120M  0 lvm  
│ └─sda1                                8:1    0   10G  0 part 
│   └─sda                               8:0    0   10G  0 disk 
├─raid-lvol0_rmeta_1 (dm-19)          253:19   0    4M  0 lvm  
│ └─sdd1                                8:49   0   10G  0 part 
│   └─sdd                               8:48   0   10G  0 disk 
└─raid-lvol0_rimage_1 (dm-20)         253:20   0  120M  0 lvm  
  └─sdd1                                8:49   0   10G  0 part 
    └─sdd                               8:48   0   10G  0 disk 
stacked-lvol0 (dm-22)                 253:22   0  200M  0 lvm  /mnt/write_snap_1
├─thinny-thpv1 (dm-8)                 253:8    0    1G  0 lvm  
│ └─thinny-pool-tpool (dm-4)          253:4    0    5G  0 lvm  
│   ├─thinny-pool_tmeta (dm-2)        253:2    0    8M  0 lvm  
│   │ └─sdh1                            8:113  0   10G  0 part 
│   │   └─sdh                           8:112  0   10G  0 disk 
│   └─thinny-pool_tdata (dm-3)        253:3    0    5G  0 lvm  
│     └─sdh1                            8:113  0   10G  0 part 
│       └─sdh                           8:112  0   10G  0 disk 
└─thinny-thpv2 (dm-9)                 253:9    0    1G  0 lvm  
  └─thinny-pool-tpool (dm-4)          253:4    0    5G  0 lvm  
    ├─thinny-pool_tmeta (dm-2)        253:2    0    8M  0 lvm  
    │ └─sdh1                            8:113  0   10G  0 part 
    │   └─sdh                           8:112  0   10G  0 disk 
    └─thinny-pool_tdata (dm-3)        253:3    0    5G  0 lvm  
      └─sdh1                            8:113  0   10G  0 part 
        └─sdh                           8:112  0   10G  0 disk 


mounted as:

/dev/mapper/helter_skelter-syncd_secondary_core_4legs_1 on /mnt/syncd_primary_2devices_1 type ext3 (rw)
/dev/mapper/raid-lvol0 on /mnt/write_snap_2 type ext3 (rw)
/dev/mapper/thinny-thinlv on /mnt/write_snap_1/test type ext4 (rw)
/dev/mapper/stacked-lvol0 on /mnt/write_snap_1/test/again type ext4 (rw)
/dev/mapper/thinny-lvolthin on /mnt/write_snap_1/test/again/deep type ext3 (rw)


Nothing has been written into those locations - they were only mounted.


Reboot:

Stopping block device availability: Deactivating block devices:
  [SKIP]: unmount of VolGroup-lv_root (dm-0) mounted on /
  UMOUNT: unmounting thinny-lvolthin (dm-6) mounted on /mnt/write_snap_1/test/again/deep
  UMOUNT: unmounting thinny-thinlv (dm-7) mounted on /mnt/write_snap_1/test
umount: /mnt/write_snap_1/test: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
  UMOUNT: unmounting helter_skelter-syncd_secondary_core_4legs_1 (dm-14) mounted on /mnt/syncd_primary_2devices_1
  LVM: deactivating Volume Group helter_skelter
    0 logical volume(s) in volume group "helter_skelter" now active
  UMOUNT: unmounting raid-lvol0 (dm-21) mounted on /mnt/write_snap_2
  LVM: deactivating Volume Group raid
    0 logical volume(s) in volume group "raid" now active
  UMOUNT: unmounting stacked-lvol0 (dm-22) mounted on /mnt/write_snap_1/test/again
  LVM: deactivating Volume Group stacked
    0 logical volume(s) in volume group "stacked" now active
[  OK  ]

Stopping iscsi: device-mapper: thin: commit failed, error = -19
device-mapper: thin: switching pool to read-only mode
device-mapper: thin: aborting transaction failed
device-mapper: thin: switching pool to failure mode
Buffer I/O error on device dm-7, logical block 0
lost page write due to I/O error on dm-7

Stopping monitoring for VG VolGroup:   /dev/mapper/thinny-pool: read failed after 0 of 4096 at 5368643584: Input/output error
  /dev/mapper/thinny-pool: read failed after 0 of 4096 at 5368700928: Input/output error
  /dev/mapper/thinny-pool: read failed after 0 of 4096 at 0: Input/output error
  /dev/mapper/thinny-pool: read failed after 0 of 4096 at 4096: Input/output error
  /dev/thinny/lvolthin: read failed after 0 of 4096 at 5368643584: Input/output error
  /dev/thinny/lvolthin: read failed after 0 of 4096 at 5368700928: Input/output error
  /dev/thinny/lvolthin: read failed after 0 of 4096 at 0: Input/output error
  /dev/thinny/lvolthin: read failed after 0 of 4096 at 4096: Input/output error
  /dev/thinny/thinlv: read failed after 0 of 4096 at 5368643584: Input/output error
  /dev/thinny/thinlv: read failed after 0 of 4096 at 5368700928: Input/output error
  /dev/thinny/thinlv: read failed after 0 of 4096 at 0: Input/output error
  /dev/thinny/thinlv: read failed after 0 of 4096 at 4096: Input/output error
  /dev/thinny/thpv1: read failed after 0 of 4096 at 1073676288: Input/output error
  /dev/thinny/thpv1: read failed after 0 of 4096 at 1073733632: Input/output error
  /dev/thinny/thpv1: read failed after 0 of 4096 at 0: Input/output error
  /dev/thinny/thpv1: read failed after 0 of 4096 at 4096: Input/output error
  /dev/thinny/thpv2: read failed after 0 of 4096 at 1073676288: Input/output error
  /dev/thinny/thpv2: read failed after 0 of 4096 at 1073733632: Input/output error
  /dev/thinny/thpv2: read failed after 0 of 4096 at 0: Input/output error
  /dev/thinny/thpv2: read failed after 0 of 4096 at 4096: Input/output error
  2 logical volume(s) in volume group "VolGroup" unmonitored
[  OK  ]


Is there something I did too arbitrarily, or is this something that can be fixed in the way blkdeactivate is reading dependencies, mount order and umounting/deactivating things.

Comment 27 Peter Rajnoha 2013-01-23 13:55:02 UTC

(In reply to comment #26)
> Is there something I did too arbitrarily, or is this something that can be
> fixed in the way blkdeactivate is reading dependencies, mount order and
> umounting/deactivating things.

Your test is OK. Thanks a lot for spotting this!

I've committed a fix for this (the problem was in nested mountpoints):

http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=f7da1caf8deaf50e0597ec377fb035ea3de1ac36

(I've also found that mangled mount paths were not processed correctly, so this is fixed now as well.)

Comment 28 Peter Rajnoha 2013-01-23 14:01:28 UTC

The additional patch needs to be added to 6.4!

For example the simplest reproducer:
    
    [root@rhel6-a ~]# lsblk
    NAME                        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda                           8:0    0    4G  0 disk
    |-vg-lvol0 (dm-2)           253:2    0   32M  0 lvm  /mnt/a
    `-vg-lvol1 (dm-3)           253:3    0   32M  0 lvm  /mnt/a/b
    
Before this patch:
    
    [root@rhel6-a ~]# blkdeactivate -u
    Deactivating block devices:
      UMOUNT: unmounting vg-lvol0 (dm-2) mounted on /mnt/a
    umount: /mnt/a: device is busy.
            (In some cases useful info about processes that use 
             the device is found by lsof(8) or fuser(1))
      UMOUNT: unmounting vg-lvol1 (dm-3) mounted on /mnt/a/b
      LVM: deactivating Logical Volume vg/lvol1
    
    (deactivation of vg/lvol0 is skipped as /mnt/a that is on lvol0
    can't be unmounted - it still has /mnt/a/b as nested mountpoint!)
    
With this patch applied:
    
    [root@rhel6-a ~]# blkdeactivate -u
    Deactivating block devices:
      UMOUNT: unmounting vg-lvol1 (dm-3) mounted on /mnt/a/b
      UMOUNT: unmounting vg-lvol0 (dm-2) mounted on /mnt/a
      LVM: deactivating Logical Volume vg/lvol0
      LVM: deactivating Logical Volume vg/lvol1
    
===

Also:

    [root@rhel6-a ~]# lsblk
    NAME                        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda                           8:0    0    4G  0 disk
    `-vg-lvol0 (dm-2)           253:2    0   32M  0 lvm  /mnt/x y z 
    
    [root@rhel6-a ~]# lsblk -r
    vg-lvol0 253:2 0 32M 0 lvm /mnt/x\x20y\x20z
    
(the mount point is mangled with \xNN that is visible in raw 
lsblk output only and which is used in blkdeactive)
    
Before this patch:
    
    [root@rhel6-a ~]# blkdeactivate -u
    Deactivating block devices:
      umount: /mnt/x\x20y\x20z: not found

After this patch applied:
    
   [root@rhel6-a ~]# blkdeactivate -u
    Deactivating block devices:
      UMOUNT: unmounting vg-lvol0 (dm-2) mounted on /mnt/x\x20y\x20z
      LVM: deactivating Logical Volume vg/lvol0

Comment 29 Nenad Peric 2013-01-23 15:55:15 UTC

Tested with scratch build lvm2-2.02.98-9.el6.x86_64:

It works without issues and clean as expected, this is a paste from reboot:

Stopping block device availability: Deactivating block devices:
  [SKIP]: unmount of VolGroup-lv_swap (dm-1) mounted on [SWAP]
  UMOUNT: unmounting vg_thin-lvol1 (dm-27) mounted on /mnt/thin
  UMOUNT: unmounting vg_thin-lvol0 (dm-26) mounted on /mnt/regular/skinny
  UMOUNT: unmounting first-stripe (dm-2) mounted on /mnt/regular
  UMOUNT: unmounting stacked-stripe (dm-28) mounted on /mnt/raid/stack
  UMOUNT: unmounting raid-raid1_lv (dm-11) mounted on /mnt/raid
  [SKIP]: unmount of VolGroup-lv_root (dm-0) mounted on /
  LVM: deactivating Volume Group first
    0 logical volume(s) in volume group "first" now active
  LVM: deactivating Volume Group stacked
    0 logical volume(s) in volume group "stacked" now active
  LVM: deactivating Volume Group raid
    0 logical volume(s) in volume group "raid" now active
  LVM: deactivating Volume Group vg_thin
    0 logical volume(s) in volume group "vg_thin" now active
[  OK  ]

Comment 30 Peter Rajnoha 2013-01-23 16:02:05 UTC

(In reply to comment #29)
> Tested with scratch build lvm2-2.02.98-9.el6.x86_64:
> 

(which includes this addendum too: http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=2be83f45431313947619ac57e9554c59621c371b)

Comment 32 Nenad Peric 2013-01-24 09:50:56 UTC

Marking verified based on my last comment.

Comment 33 errata-xmlrpc 2013-02-21 08:03:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0501.html

Comment 36 Vitor Zanoni 2013-03-20 21:32:59 UTC

=== In Red Hat Customer Portal Case 00801196 ===
--- Comment by Zanoni, Vitor on 3/20/2013 6:32 PM ---

Hi

the message is the following:

Stopping block device availability: Deactivating block defices:
  UMOUNT: umounting vg01-lv_audit (dm-5) mounted on /var/log/audit
umount: /var/log/audit:device is busy.
        (In some cases useful info about processes that use  
         the device is found by lsof(8) or fuser(1)) 
  UMOUNT: umounting vg01-lv_log (dm-6) mounted on /var/log
umount: /var/log:device is busy.
        (In some cases useful info about processes that use  
         the device is found by lsof(8) or fuser(1)) 
  UMOUNT: umounting vg01-lv_var (dm-1) mounted on /var
umount: /var:device is busy.
        (In some cases useful info about processes that use  
         the device is found by lsof(8) or fuser(1)) 
  [SKIP]: umount of vg01-lv_usr (dm-4) mounted on /usr
  UMOUNT: umounting vg01-lv_tmp (dm-3) mounted on /tmp
umount: /tmp:device is busy.
        (In some cases useful info about processes that use  
         the device is found by lsof(8) or fuser(1)) 
  [SKIP]: umount of vg01-lv_swap (dm-0) mounted [SWAP]
  UMOUNT: umounting vg01-lv_home (dm-3) mounted on /home

According with customer, is possible to shutdown this machine normally, however, these messages always are showed during this process.

This host is a VMware guest.

tks

Comment 39 Peter Rajnoha 2013-03-21 09:38:58 UTC

Thanks for the feedback. The messages you see comes from the umount call that needs to be wrapped better. Anyway, this should not cause any harm, they're just messages that should have been better wrapped (and possibly the 'still-used' mountpoints better detected).

I've opened a new bug #924137 for this.

Comment 43 BugMasta 2013-04-23 02:13:00 UTC

RHEL 6 
The OS that cannot even shutdown cleanly.

FYI I have a 6.4 system here which has just hung on this garbage script, "Stopping block device availability: Deactivating block devices:"

Brilliant work guys.

Comment 44 Peter Rajnoha 2013-04-23 06:46:54 UTC

Please, attach the output of "lsblk", "lvs" and "dmsetup info -c" command and, if possible, the /var/log/messages content. Thanks.

Comment 49 Mike Lee 2013-07-26 19:51:42 UTC

I am getting the same errors as Vitor and this is also a VM guest.  Here is the output of the above requested commands.  Running CentOS 6.4:

[root@mdthlnvmwu036 ~]# lsblk
NAME                                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                                        11:0    1 1024M  0 rom  
sda                                         8:0    0   35G  0 disk 
├─sda1                                      8:1    0  500M  0 part /boot
└─sda2                                      8:2    0 34.5G  0 part 
  ├─vg_mdthlnvmwu036-LVRoot (dm-0)        253:0    0   12G  0 lvm  /
  ├─vg_mdthlnvmwu036-LVSwap (dm-1)        253:1    0  3.9G  0 lvm  [SWAP]
  ├─vg_mdthlnvmwu036-LVVar (dm-2)         253:2    0  3.9G  0 lvm  /var
  ├─vg_mdthlnvmwu036-LVVarLogAudit (dm-3) 253:3    0  3.9G  0 lvm  /var/log/audit
  ├─vg_mdthlnvmwu036-LVTmp (dm-4)         253:4    0  6.9G  0 lvm  /tmp
  └─vg_mdthlnvmwu036-LVVarLog (dm-5)      253:5    0  3.9G  0 lvm  /var/log
[root@mdthlnvmwu036 ~]# lvs
  LV            VG               Attr      LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  LVRoot        vg_mdthlnvmwu036 -wi-ao--- 12.05g                                             
  LVSwap        vg_mdthlnvmwu036 -wi-ao---  3.91g                                             
  LVTmp         vg_mdthlnvmwu036 -wi-ao---  6.84g                                             
  LVVar         vg_mdthlnvmwu036 -wi-ao---  3.91g                                             
  LVVarLog      vg_mdthlnvmwu036 -wi-ao---  3.91g                                             
  LVVarLogAudit vg_mdthlnvmwu036 -wi-ao---  3.91g                                             
[root@mdthlnvmwu036 ~]# dmsetup info -c
Name                           Maj Min Stat Open Targ Event  UUID                                                                
vg_mdthlnvmwu036-LVSwap        253   1 L--w    1    1      0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpS3qzq1yQJ0vPk905PxiQFP9Kj2UR39Z0V
vg_mdthlnvmwu036-LVVarLogAudit 253   3 L--w    1    1      0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpS98soFjJVqv2Afre62T48x9MWtuItS4Qn
vg_mdthlnvmwu036-LVRoot        253   0 L--w    1    1      0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpSYAhfzybVBPU2Hd4pRuavqK2V6J8BRKHK
vg_mdthlnvmwu036-LVTmp         253   4 L--w    1    1      0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpShPKhLdtDNogjoiJHIZpc3SSDB1bsTPGt
vg_mdthlnvmwu036-LVVar         253   2 L--w    1    1      0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpSbyCKBR35cYPBtbgdkZhDlWHwn00vAi1h
vg_mdthlnvmwu036-LVVarLog      253   5 L--w    1    1      0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpSw28SI5M4YC0hC2ZPG7QtCoPBFV2VTMHp

Comment 50 Mike Lee 2013-09-05 16:55:14 UTC

Any progress on this issue?  Thanks.

Mike

Comment 51 Peter Rajnoha 2013-09-09 08:31:16 UTC

There's a patch added for blkdeactivate in RHEL 6.5 which changes the way the tool reports messages from external tools. Also, the /var and /var/log is not unmounted as other tools make use of it during shutdown (see also bug #924137 comment 10).

Note You need to log in before you can comment on or make changes to this bug.

agk
bmarzins
cmarthal
dkelson
dwysocha
heinzm
jbrassow
jcpunk
jwest
kay
kfujii
lpoetter
ltroan
masaki.kimura.kz
mchristi
mitsuhiro.tanino.gm
mlee
msnitzer
myamazak
noboru.obata.ar
notting
nperic
patrickrowan
peljasz
prajnoha
prockai
saguchi
Sean.Stewart
syeghiay
takahiro.yasui.mp
teruaki.ishizaki
thornber
tlavigne
tomasz.kepczynski
tsekiyam
vorpal
zkabelac