Bug 800801
| Summary: | Device stack including LVM and multipath should be properly deactivated on shutdown | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Peter Rajnoha <prajnoha> |
| Component: | lvm2 | Assignee: | Peter Rajnoha <prajnoha> |
| Status: | CLOSED ERRATA | QA Contact: | Cluster QE <mspqa-list> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.4 | CC: | agk, bmarzins, cmarthal, dkelson, dwysocha, heinzm, jbrassow, jcpunk, jwest, kay, kfujii, lpoetter, ltroan, masaki.kimura.kz, mchristi, mitsuhiro.tanino.gm, mlee, msnitzer, myamazak, noboru.obata.ar, notting, nperic, patrickrowan, peljasz, prajnoha, prockai, saguchi, Sean.Stewart, syeghiay, takahiro.yasui.mp, teruaki.ishizaki, thornber, tlavigne, tomasz.kepczynski, tsekiyam, vorpal, zkabelac |
| Target Milestone: | beta | ||
| Target Release: | 6.4 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | lvm2-2.02.98-9.el6 | Doc Type: | Enhancement |
| Doc Text: |
Cause:
Device-mapper devices (including LVM devices) were not deactivated at system shutdown or reboot.
Consequence:
When device-mapper devices were layered on top of other block devices and these were detached during the shutdown/reboot procedure, any further access to the device-mapper devices ended up with either IO errors or even a hang as the underlying devices were gone (e.g iSCSI or FCoE devices).
Fix:
A new blkdeactivate script together with the accompanying blk-availability shutdown script has been provided that unmounts and deactivates any existing device-mapper devices before deactivating and detaching the underlying devices on shutdown/reboot.
Result:
Clean shutdown/reboot without IO errors or hangs if using attached storage that detaches itself during shutdown/reboot procedure.
|
Story Points: | --- |
| Clone Of: | 672530 | Environment: | |
| Last Closed: | 2013-02-21 08:03:30 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 672530 | ||
| Bug Blocks: | 784640, 905536 | ||
|
Description
Peter Rajnoha
2012-03-07 10:06:15 UTC
The lsblk patch is under review/waiting upstream acceptance... The lsblk patch is upstream now (should appear in upcoming util-linux 2.22). I'll try to negotiate with initscripts maintainer to have the script using lsblk output for proper device shutdown included (as this is a general problem with stacked devices, not just lvm one). *** Bug 784584 has been marked as a duplicate of this bug. *** Bug 784584 was CLOSED=DUP of this bug but the trackers were not carried forward. I added Hitachi6.4bugs and the GSS_6_3_Proposed trackers but the latter needs to be updated to GSS_6_4_Proposed. Thanks Larry. I've corrected it. Hi Peter-san, Could you share upstream URL of this bugfix? Also could you share latest status? Regards, Mitsuhiro Tanino The script is not upstream yet, it'll be committed soon (during this week), I'll add documentation and a bit of a cleanup. I'll update once this is done. Hi Peter-san, Thank you for your update. If you will have any update from now, please share it us. Regards, Mitsuhiro Tanino *** Bug 784584 has been marked as a duplicate of this bug. *** Patch introducing the "blkdeactivate" utility together with initscript/systemd unit to automatically call it on shutdown has been committed upstream (it'll be a part of lvm2 v2_02_98): http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=c698ee14bbb1310cf2383c8977d14a8e29139f8c http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=a70d6b350f98bd6e9068ce723a275f31e9f27b7e The required lsblk patch is upstream since util-linux v 2.22 and backported to RHEL6.4 within the util-linux-ng-2.17.2-12.8.el6 build. The blkdeactivate utility will be included in next lvm2 build for 6.4. For now, the utility deactivates all the device-mapper-based (including LVM) devices. Support for more device types may be included later (e.g. the mdraid) - it should be fairly easy to do so, but let's have the basic version working now. To QA:
To test this, create an arbitrary stack of device-mapper-based (pure dm, mpath, cryptsetup...) devices arbitrarily stacked one over the other together with LVM and then use the new blkdeactivate script.
Example:
========
# lsblk -s
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 128M 0 disk
sdc 8:32 0 128M 0 disk
sde 8:64 0 128M 0 disk
sdj 8:144 0 128M 0 disk
sdk 8:160 0 128M 0 disk
sdl 8:176 0 128M 0 disk
sdm 8:192 0 128M 0 disk
sdn 8:208 0 128M 0 disk
sdo 8:224 0 128M 0 disk
sdp 8:240 0 128M 0 disk
sdq 65:0 0 128M 0 disk
sdr 65:16 0 128M 0 disk
sds 65:32 0 128M 0 disk
sdt 65:48 0 128M 0 disk
sdu 65:64 0 128M 0 disk
sdv 65:80 0 128M 0 disk
sdw 65:96 0 128M 0 disk
sdx 65:112 0 128M 0 disk
sdy 65:128 0 128M 0 disk
sdz 65:144 0 128M 0 disk
vda1 252:1 0 512M 0 part /boot
`-vda 252:0 0 20G 0 disk
vda2 252:2 0 19G 0 part /
`-vda 252:0 0 20G 0 disk
vda3 252:3 0 512M 0 part [SWAP]
`-vda 252:0 0 20G 0 disk
vg2-lvol0 (dm-6) 253:6 0 244M 0 lvm /mnt/temp
|-sdg 8:96 0 128M 0 disk
`-crypted_one (dm-5) 253:5 0 122M 0 crypt
`-vg1-lvol0 (dm-4) 253:4 0 124M 0 lvm
|-vg1-lvol0_mlog (dm-1) 253:1 0 4M 0 lvm
| `-vg-lvol0 (dm-0) 253:0 0 32M 0 lvm
| `-sda 8:0 0 128M 0 disk
|-vg1-lvol0_mimage_0 (dm-2) 253:2 0 124M 0 lvm
| `-sdd 8:48 0 128M 0 disk
`-vg1-lvol0_mimage_1 (dm-3) 253:3 0 124M 0 lvm
`-sdf 8:80 0 128M 0 disk
vg3-lvol0 (dm-8) 253:8 0 28M 0 lvm /mnt/temp1
`-mpath_one (dm-7) 253:7 0 32M 0 dm
|-sdh 8:112 0 128M 0 disk
`-sdi 8:128 0 128M 0 disk
# blkdeactivate
(this should cause the deactivation to be skipped as we haven't used the "-u"/"--umount" option and the devs at the top of each stack are mounted)
Deactivating block devices:
[SKIP]: unmount of vg2-lvol0 (dm-6) mounted on /mnt/temp
[SKIP]: unmount of vg3-lvol0 (dm-8) mounted on /mnt/temp1
# blkdeactivate -u
(the same, but with the "-u"/"--umount" option used - all should be deactivated as the devs on top will be unmounted as well)
Deactivating block devices:
UMOUNT: unmounting vg2-lvol0 (dm-6) mounted on /mnt/temp
LVM: deactivating Logical Volume vg2/lvol0
DM: deactivating crypt device crypted_one (dm-5)
LVM: deactivating Logical Volume vg1/lvol0
LVM: deactivating Logical Volume vg/lvol0
UMOUNT: unmounting vg3-lvol0 (dm-8) mounted on /mnt/temp1
LVM: deactivating Logical Volume vg3/lvol0
DM: deactivating dm device mpath_one (dm-7)
# lsblk -s
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 128M 0 disk
sdb 8:16 0 128M 0 disk
sdc 8:32 0 128M 0 disk
sdd 8:48 0 128M 0 disk
sde 8:64 0 128M 0 disk
sdf 8:80 0 128M 0 disk
sdg 8:96 0 128M 0 disk
sdh 8:112 0 128M 0 disk
sdi 8:128 0 128M 0 disk
sdj 8:144 0 128M 0 disk
sdk 8:160 0 128M 0 disk
sdl 8:176 0 128M 0 disk
sdm 8:192 0 128M 0 disk
sdn 8:208 0 128M 0 disk
sdo 8:224 0 128M 0 disk
sdp 8:240 0 128M 0 disk
sdq 65:0 0 128M 0 disk
sdr 65:16 0 128M 0 disk
sds 65:32 0 128M 0 disk
sdt 65:48 0 128M 0 disk
sdu 65:64 0 128M 0 disk
sdv 65:80 0 128M 0 disk
sdw 65:96 0 128M 0 disk
sdx 65:112 0 128M 0 disk
sdy 65:128 0 128M 0 disk
sdz 65:144 0 128M 0 disk
vda1 252:1 0 512M 0 part /boot
`-vda 252:0 0 20G 0 disk
vda2 252:2 0 19G 0 part /
`-vda 252:0 0 20G 0 disk
vda3 252:3 0 512M 0 part [SWAP]
`-vda 252:0 0 20G 0 disk
============================================
(create the same stack again)
# blkdeactivate -u -l wholevg
(the -l wholevg option will cause the whole VGs to be deactivated at once instead of separate LV deactivation)
Deactivating block devices:
UMOUNT: unmounting vg2-lvol0 (dm-7) mounted on /mnt/temp
LVM: deactivating Volume Group vg2
0 logical volume(s) in volume group "vg2" now active
DM: deactivating crypt device crypted_one (dm-5)
LVM: deactivating Volume Group vg1
0 logical volume(s) in volume group "vg1" now active
LVM: deactivating Volume Group vg
0 logical volume(s) in volume group "vg" now active
UMOUNT: unmounting vg3-lvol0 (dm-8) mounted on /mnt/temp1
LVM: deactivating Volume Group vg3
0 logical volume(s) in volume group "vg3" now active
DM: deactivating dm device mpath_one (dm-6)
[0] rawhide/~ # lsblk -s
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 128M 0 disk
sdb 8:16 0 128M 0 disk
sdc 8:32 0 128M 0 disk
sdd 8:48 0 128M 0 disk
sde 8:64 0 128M 0 disk
sdf 8:80 0 128M 0 disk
sdg 8:96 0 128M 0 disk
sdh 8:112 0 128M 0 disk
sdi 8:128 0 128M 0 disk
sdj 8:144 0 128M 0 disk
sdk 8:160 0 128M 0 disk
sdl 8:176 0 128M 0 disk
sdm 8:192 0 128M 0 disk
sdn 8:208 0 128M 0 disk
sdo 8:224 0 128M 0 disk
sdp 8:240 0 128M 0 disk
sdq 65:0 0 128M 0 disk
sdr 65:16 0 128M 0 disk
sds 65:32 0 128M 0 disk
sdt 65:48 0 128M 0 disk
sdu 65:64 0 128M 0 disk
sdv 65:80 0 128M 0 disk
sdw 65:96 0 128M 0 disk
sdx 65:112 0 128M 0 disk
sdy 65:128 0 128M 0 disk
sdz 65:144 0 128M 0 disk
vda1 252:1 0 512M 0 part /boot
`-vda 252:0 0 20G 0 disk
vda2 252:2 0 19G 0 part /
`-vda 252:0 0 20G 0 disk
vda3 252:3 0 512M 0 part [SWAP]
`-vda 252:0 0 20G 0 disk
============================================
(create the same stack again + added one more mapping "zero_one")
# lsblk -s
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 128M 0 disk
sdc 8:32 0 128M 0 disk
sde 8:64 0 128M 0 disk
sdk 8:160 0 128M 0 disk
sdl 8:176 0 128M 0 disk
sdm 8:192 0 128M 0 disk
sdn 8:208 0 128M 0 disk
sdo 8:224 0 128M 0 disk
sdp 8:240 0 128M 0 disk
sdq 65:0 0 128M 0 disk
sdr 65:16 0 128M 0 disk
sds 65:32 0 128M 0 disk
sdt 65:48 0 128M 0 disk
sdu 65:64 0 128M 0 disk
sdv 65:80 0 128M 0 disk
sdw 65:96 0 128M 0 disk
sdx 65:112 0 128M 0 disk
sdy 65:128 0 128M 0 disk
sdz 65:144 0 128M 0 disk
vda1 252:1 0 512M 0 part /boot
`-vda 252:0 0 20G 0 disk
vda2 252:2 0 19G 0 part /
`-vda 252:0 0 20G 0 disk
vda3 252:3 0 512M 0 part [SWAP]
`-vda 252:0 0 20G 0 disk
vg2-lvol0 (dm-6) 253:6 0 244M 0 lvm /mnt/temp
|-sdg 8:96 0 128M 0 disk
`-crypted_one (dm-5) 253:5 0 122M 0 crypt
`-vg1-lvol0 (dm-4) 253:4 0 124M 0 lvm
|-vg1-lvol0_mlog (dm-1) 253:1 0 4M 0 lvm
| `-vg-lvol0 (dm-0) 253:0 0 32M 0 lvm
| `-sda 8:0 0 128M 0 disk
|-vg1-lvol0_mimage_0 (dm-2) 253:2 0 124M 0 lvm
| `-sdd 8:48 0 128M 0 disk
`-vg1-lvol0_mimage_1 (dm-3) 253:3 0 124M 0 lvm
`-sdf 8:80 0 128M 0 disk
zero_one (dm-8) 253:8 0 32M 0 dm
`-sdj 8:144 0 128M 0 disk
vg3-lvol0 (dm-9) 253:9 0 28M 0 lvm /mnt/temp1
`-mpath_one (dm-7) 253:7 0 32M 0 dm
|-sdh 8:112 0 128M 0 disk
`-sdi 8:128 0 128M 0 disk
[0] rawhide/~ # blkdeactivate -u -l wholevg /dev/mapper/mpath_one /dev/mapper/zero_one
(specifying the devs to deactivate directly on the command line, if the dev is in the middle of the device stack, all holders of the dev will be deactivated first - like the /dev/mapper/mpath_one which has vg-lvol0 stacked on top)
Deactivating block devices:
UMOUNT: unmounting vg3-lvol0 (dm-7) mounted on /mnt/temp1
LVM: deactivating Volume Group vg3
0 logical volume(s) in volume group "vg3" now active
DM: deactivating dm device mpath_one (dm-8)
DM: deactivating dm device zero_one (dm-9)
# lsblk -s
(the vg2-lvol0 and downwards the stack stays untouched - no dev specified on the command line for blkdeactivate above is included in the stack)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 128M 0 disk
sdc 8:32 0 128M 0 disk
sde 8:64 0 128M 0 disk
sdh 8:112 0 128M 0 disk
sdi 8:128 0 128M 0 disk
sdj 8:144 0 128M 0 disk
sdk 8:160 0 128M 0 disk
sdl 8:176 0 128M 0 disk
sdm 8:192 0 128M 0 disk
sdn 8:208 0 128M 0 disk
sdo 8:224 0 128M 0 disk
sdp 8:240 0 128M 0 disk
sdq 65:0 0 128M 0 disk
sdr 65:16 0 128M 0 disk
sds 65:32 0 128M 0 disk
sdt 65:48 0 128M 0 disk
sdu 65:64 0 128M 0 disk
sdv 65:80 0 128M 0 disk
sdw 65:96 0 128M 0 disk
sdx 65:112 0 128M 0 disk
sdy 65:128 0 128M 0 disk
sdz 65:144 0 128M 0 disk
vda1 252:1 0 512M 0 part /boot
`-vda 252:0 0 20G 0 disk
vda2 252:2 0 19G 0 part /
`-vda 252:0 0 20G 0 disk
vda3 252:3 0 512M 0 part [SWAP]
`-vda 252:0 0 20G 0 disk
vg2-lvol0 (dm-6) 253:6 0 244M 0 lvm /mnt/temp
|-sdg 8:96 0 128M 0 disk
`-crypted_one (dm-5) 253:5 0 122M 0 crypt
`-vg1-lvol0 (dm-4) 253:4 0 124M 0 lvm
|-vg1-lvol0_mlog (dm-1) 253:1 0 4M 0 lvm
| `-vg-lvol0 (dm-0) 253:0 0 32M 0 lvm
| `-sda 8:0 0 128M 0 disk
|-vg1-lvol0_mimage_0 (dm-2) 253:2 0 124M 0 lvm
| `-sdd 8:48 0 128M 0 disk
`-vg1-lvol0_mimage_1 (dm-3) 253:3 0 124M 0 lvm
`-sdf 8:80 0 128M 0 disk
============================================
This should be all done automatically by initscript called on shutdown if blk-availability initscript is enabled (this is actually a "shutdown" only initscript). Also, it should be done before the iscsi service is disabled.
N.B.: the swap, /boot, /, /lib, /lib64, /bin, /sbin and /usr mount (including subdirs /bin, /sbin, /lib, /lib64) is *not* unmounted with the -u option!
(In reply to comment #20) > This should be all done automatically by initscript called on shutdown if > blk-availability initscript is enabled (this is actually a "shutdown" only > initscript). Also, it should be done before the iscsi service is disabled. Example snippet of shutdown (where there was one VG called "vg", just for an example...): ... Stopping HAL daemon: [ OK ] Stopping blk-availability Deactivating block devices: [SKIP]: unmount of vg_rhel6a-lv_root (dm-0) mounted on / LVM: deactivating Volume Group vg 0 logical volume(s) in volume group "vg" now active [ OK ] Stopping NetworkManager daemon: [ OK ] Stopping systemd message bus: [ OK ] Stopping rpcbind: [ OK ] Stopping iscsi: [ OK ] Shutting down system logger: [ OK ] ... (In reply to comment #20) > it should be done before the iscsi service is disabled. ..the same applies for fcoe service if enabled. *** Bug 892039 has been marked as a duplicate of this bug. *** Based on Comment #20 I created an arbitrarily stacked and mounted devices. But the reboot is not what you would call pretty and graceful, and blkdeactivate -u needs to be ran several times in order to unmount and deactivate all LVs/VGs. I think there is something to do with thin pool and thin LVs, and the order those are handled during this deactivation process. Setup (mountpoint did not really fit): (11:13:53) [root@r6-node02:~]$ lsblk -s NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda1 252:1 0 500M 0 part /boot └─vda 252:0 0 10G 0 disk VolGroup-lv_root (dm-0) 253:0 0 7.6G 0 lvm / └─vda2 252:2 0 9.5G 0 part └─vda 252:0 0 10G 0 disk VolGroup-lv_swap (dm-1) 253:1 0 2G 0 lvm [SWAP] └─vda2 252:2 0 9.5G 0 part └─vda 252:0 0 10G 0 disk sdb1 8:17 0 10G 0 part └─sdb 8:16 0 10G 0 disk sdc1 8:33 0 10G 0 part └─sdc 8:32 0 10G 0 disk sdf1 8:81 0 10G 0 part └─sdf 8:80 0 10G 0 disk sdi1 8:129 0 10G 0 part └─sdi 8:128 0 10G 0 disk thinny-pool (dm-5) 253:5 0 5G 0 lvm └─thinny-pool-tpool (dm-4) 253:4 0 5G 0 lvm ├─thinny-pool_tmeta (dm-2) 253:2 0 8M 0 lvm │ └─sdh1 8:113 0 10G 0 part │ └─sdh 8:112 0 10G 0 disk └─thinny-pool_tdata (dm-3) 253:3 0 5G 0 lvm └─sdh1 8:113 0 10G 0 part └─sdh 8:112 0 10G 0 disk thinny-lvolthin (dm-6) 253:6 0 5G 0 lvm /mnt/write_snap_1 └─thinny-pool-tpool (dm-4) 253:4 0 5G 0 lvm ├─thinny-pool_tmeta (dm-2) 253:2 0 8M 0 lvm │ └─sdh1 8:113 0 10G 0 part │ └─sdh 8:112 0 10G 0 disk └─thinny-pool_tdata (dm-3) 253:3 0 5G 0 lvm └─sdh1 8:113 0 10G 0 part └─sdh 8:112 0 10G 0 disk thinny-thinlv (dm-7) 253:7 0 5G 0 lvm /mnt/write_snap_1 └─thinny-pool-tpool (dm-4) 253:4 0 5G 0 lvm ├─thinny-pool_tmeta (dm-2) 253:2 0 8M 0 lvm │ └─sdh1 8:113 0 10G 0 part │ └─sdh 8:112 0 10G 0 disk └─thinny-pool_tdata (dm-3) 253:3 0 5G 0 lvm └─sdh1 8:113 0 10G 0 part └─sdh 8:112 0 10G 0 disk helter_skelter-syncd_secondary_core_4legs_1 (dm-14) 253:14 0 500M 0 lvm /mnt/syncd_primar └─helter_skelter-syncd_secondary_core_4legs_1-real (dm-13) 253:13 0 500M 0 lvm ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_0 (dm-10) 253:10 0 500M 0 lvm │ └─sdj1 8:145 0 10G 0 part │ └─sdj 8:144 0 10G 0 disk ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_2 (dm-11) 253:11 0 500M 0 lvm │ └─sdg1 8:97 0 10G 0 part │ └─sdg 8:96 0 10G 0 disk └─helter_skelter-syncd_secondary_core_4legs_1_mimage_3 (dm-12) 253:12 0 500M 0 lvm └─sde1 8:65 0 10G 0 part └─sde 8:64 0 10G 0 disk helter_skelter-hs_snap1 (dm-16) 253:16 0 500M 0 lvm ├─helter_skelter-syncd_secondary_core_4legs_1-real (dm-13) 253:13 0 500M 0 lvm │ ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_0 (dm-10) 253:10 0 500M 0 lvm │ │ └─sdj1 8:145 0 10G 0 part │ │ └─sdj 8:144 0 10G 0 disk │ ├─helter_skelter-syncd_secondary_core_4legs_1_mimage_2 (dm-11) 253:11 0 500M 0 lvm │ │ └─sdg1 8:97 0 10G 0 part │ │ └─sdg 8:96 0 10G 0 disk │ └─helter_skelter-syncd_secondary_core_4legs_1_mimage_3 (dm-12) 253:12 0 500M 0 lvm │ └─sde1 8:65 0 10G 0 part │ └─sde 8:64 0 10G 0 disk └─helter_skelter-hs_snap1-cow (dm-15) 253:15 0 252M 0 lvm └─sdj1 8:145 0 10G 0 part └─sdj 8:144 0 10G 0 disk raid-lvol0 (dm-21) 253:21 0 120M 0 lvm /mnt/write_snap_2 ├─raid-lvol0_rmeta_0 (dm-17) 253:17 0 4M 0 lvm │ └─sda1 8:1 0 10G 0 part │ └─sda 8:0 0 10G 0 disk ├─raid-lvol0_rimage_0 (dm-18) 253:18 0 120M 0 lvm │ └─sda1 8:1 0 10G 0 part │ └─sda 8:0 0 10G 0 disk ├─raid-lvol0_rmeta_1 (dm-19) 253:19 0 4M 0 lvm │ └─sdd1 8:49 0 10G 0 part │ └─sdd 8:48 0 10G 0 disk └─raid-lvol0_rimage_1 (dm-20) 253:20 0 120M 0 lvm └─sdd1 8:49 0 10G 0 part └─sdd 8:48 0 10G 0 disk stacked-lvol0 (dm-22) 253:22 0 200M 0 lvm /mnt/write_snap_1 ├─thinny-thpv1 (dm-8) 253:8 0 1G 0 lvm │ └─thinny-pool-tpool (dm-4) 253:4 0 5G 0 lvm │ ├─thinny-pool_tmeta (dm-2) 253:2 0 8M 0 lvm │ │ └─sdh1 8:113 0 10G 0 part │ │ └─sdh 8:112 0 10G 0 disk │ └─thinny-pool_tdata (dm-3) 253:3 0 5G 0 lvm │ └─sdh1 8:113 0 10G 0 part │ └─sdh 8:112 0 10G 0 disk └─thinny-thpv2 (dm-9) 253:9 0 1G 0 lvm └─thinny-pool-tpool (dm-4) 253:4 0 5G 0 lvm ├─thinny-pool_tmeta (dm-2) 253:2 0 8M 0 lvm │ └─sdh1 8:113 0 10G 0 part │ └─sdh 8:112 0 10G 0 disk └─thinny-pool_tdata (dm-3) 253:3 0 5G 0 lvm └─sdh1 8:113 0 10G 0 part └─sdh 8:112 0 10G 0 disk mounted as: /dev/mapper/helter_skelter-syncd_secondary_core_4legs_1 on /mnt/syncd_primary_2devices_1 type ext3 (rw) /dev/mapper/raid-lvol0 on /mnt/write_snap_2 type ext3 (rw) /dev/mapper/thinny-thinlv on /mnt/write_snap_1/test type ext4 (rw) /dev/mapper/stacked-lvol0 on /mnt/write_snap_1/test/again type ext4 (rw) /dev/mapper/thinny-lvolthin on /mnt/write_snap_1/test/again/deep type ext3 (rw) Nothing has been written into those locations - they were only mounted. Reboot: Stopping block device availability: Deactivating block devices: [SKIP]: unmount of VolGroup-lv_root (dm-0) mounted on / UMOUNT: unmounting thinny-lvolthin (dm-6) mounted on /mnt/write_snap_1/test/again/deep UMOUNT: unmounting thinny-thinlv (dm-7) mounted on /mnt/write_snap_1/test umount: /mnt/write_snap_1/test: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) UMOUNT: unmounting helter_skelter-syncd_secondary_core_4legs_1 (dm-14) mounted on /mnt/syncd_primary_2devices_1 LVM: deactivating Volume Group helter_skelter 0 logical volume(s) in volume group "helter_skelter" now active UMOUNT: unmounting raid-lvol0 (dm-21) mounted on /mnt/write_snap_2 LVM: deactivating Volume Group raid 0 logical volume(s) in volume group "raid" now active UMOUNT: unmounting stacked-lvol0 (dm-22) mounted on /mnt/write_snap_1/test/again LVM: deactivating Volume Group stacked 0 logical volume(s) in volume group "stacked" now active [ OK ] Stopping iscsi: device-mapper: thin: commit failed, error = -19 device-mapper: thin: switching pool to read-only mode device-mapper: thin: aborting transaction failed device-mapper: thin: switching pool to failure mode Buffer I/O error on device dm-7, logical block 0 lost page write due to I/O error on dm-7 Stopping monitoring for VG VolGroup: /dev/mapper/thinny-pool: read failed after 0 of 4096 at 5368643584: Input/output error /dev/mapper/thinny-pool: read failed after 0 of 4096 at 5368700928: Input/output error /dev/mapper/thinny-pool: read failed after 0 of 4096 at 0: Input/output error /dev/mapper/thinny-pool: read failed after 0 of 4096 at 4096: Input/output error /dev/thinny/lvolthin: read failed after 0 of 4096 at 5368643584: Input/output error /dev/thinny/lvolthin: read failed after 0 of 4096 at 5368700928: Input/output error /dev/thinny/lvolthin: read failed after 0 of 4096 at 0: Input/output error /dev/thinny/lvolthin: read failed after 0 of 4096 at 4096: Input/output error /dev/thinny/thinlv: read failed after 0 of 4096 at 5368643584: Input/output error /dev/thinny/thinlv: read failed after 0 of 4096 at 5368700928: Input/output error /dev/thinny/thinlv: read failed after 0 of 4096 at 0: Input/output error /dev/thinny/thinlv: read failed after 0 of 4096 at 4096: Input/output error /dev/thinny/thpv1: read failed after 0 of 4096 at 1073676288: Input/output error /dev/thinny/thpv1: read failed after 0 of 4096 at 1073733632: Input/output error /dev/thinny/thpv1: read failed after 0 of 4096 at 0: Input/output error /dev/thinny/thpv1: read failed after 0 of 4096 at 4096: Input/output error /dev/thinny/thpv2: read failed after 0 of 4096 at 1073676288: Input/output error /dev/thinny/thpv2: read failed after 0 of 4096 at 1073733632: Input/output error /dev/thinny/thpv2: read failed after 0 of 4096 at 0: Input/output error /dev/thinny/thpv2: read failed after 0 of 4096 at 4096: Input/output error 2 logical volume(s) in volume group "VolGroup" unmonitored [ OK ] Is there something I did too arbitrarily, or is this something that can be fixed in the way blkdeactivate is reading dependencies, mount order and umounting/deactivating things. (In reply to comment #26) > Is there something I did too arbitrarily, or is this something that can be > fixed in the way blkdeactivate is reading dependencies, mount order and > umounting/deactivating things. Your test is OK. Thanks a lot for spotting this! I've committed a fix for this (the problem was in nested mountpoints): http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=f7da1caf8deaf50e0597ec377fb035ea3de1ac36 (I've also found that mangled mount paths were not processed correctly, so this is fixed now as well.) The additional patch needs to be added to 6.4!
For example the simplest reproducer:
[root@rhel6-a ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 4G 0 disk
|-vg-lvol0 (dm-2) 253:2 0 32M 0 lvm /mnt/a
`-vg-lvol1 (dm-3) 253:3 0 32M 0 lvm /mnt/a/b
Before this patch:
[root@rhel6-a ~]# blkdeactivate -u
Deactivating block devices:
UMOUNT: unmounting vg-lvol0 (dm-2) mounted on /mnt/a
umount: /mnt/a: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
UMOUNT: unmounting vg-lvol1 (dm-3) mounted on /mnt/a/b
LVM: deactivating Logical Volume vg/lvol1
(deactivation of vg/lvol0 is skipped as /mnt/a that is on lvol0
can't be unmounted - it still has /mnt/a/b as nested mountpoint!)
With this patch applied:
[root@rhel6-a ~]# blkdeactivate -u
Deactivating block devices:
UMOUNT: unmounting vg-lvol1 (dm-3) mounted on /mnt/a/b
UMOUNT: unmounting vg-lvol0 (dm-2) mounted on /mnt/a
LVM: deactivating Logical Volume vg/lvol0
LVM: deactivating Logical Volume vg/lvol1
===
Also:
[root@rhel6-a ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 4G 0 disk
`-vg-lvol0 (dm-2) 253:2 0 32M 0 lvm /mnt/x y z
[root@rhel6-a ~]# lsblk -r
vg-lvol0 253:2 0 32M 0 lvm /mnt/x\x20y\x20z
(the mount point is mangled with \xNN that is visible in raw
lsblk output only and which is used in blkdeactive)
Before this patch:
[root@rhel6-a ~]# blkdeactivate -u
Deactivating block devices:
umount: /mnt/x\x20y\x20z: not found
After this patch applied:
[root@rhel6-a ~]# blkdeactivate -u
Deactivating block devices:
UMOUNT: unmounting vg-lvol0 (dm-2) mounted on /mnt/x\x20y\x20z
LVM: deactivating Logical Volume vg/lvol0
Tested with scratch build lvm2-2.02.98-9.el6.x86_64:
It works without issues and clean as expected, this is a paste from reboot:
Stopping block device availability: Deactivating block devices:
[SKIP]: unmount of VolGroup-lv_swap (dm-1) mounted on [SWAP]
UMOUNT: unmounting vg_thin-lvol1 (dm-27) mounted on /mnt/thin
UMOUNT: unmounting vg_thin-lvol0 (dm-26) mounted on /mnt/regular/skinny
UMOUNT: unmounting first-stripe (dm-2) mounted on /mnt/regular
UMOUNT: unmounting stacked-stripe (dm-28) mounted on /mnt/raid/stack
UMOUNT: unmounting raid-raid1_lv (dm-11) mounted on /mnt/raid
[SKIP]: unmount of VolGroup-lv_root (dm-0) mounted on /
LVM: deactivating Volume Group first
0 logical volume(s) in volume group "first" now active
LVM: deactivating Volume Group stacked
0 logical volume(s) in volume group "stacked" now active
LVM: deactivating Volume Group raid
0 logical volume(s) in volume group "raid" now active
LVM: deactivating Volume Group vg_thin
0 logical volume(s) in volume group "vg_thin" now active
[ OK ]
(In reply to comment #29) > Tested with scratch build lvm2-2.02.98-9.el6.x86_64: > (which includes this addendum too: http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=2be83f45431313947619ac57e9554c59621c371b) Marking verified based on my last comment. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0501.html === In Red Hat Customer Portal Case 00801196 ===
--- Comment by Zanoni, Vitor on 3/20/2013 6:32 PM ---
Hi
the message is the following:
Stopping block device availability: Deactivating block defices:
UMOUNT: umounting vg01-lv_audit (dm-5) mounted on /var/log/audit
umount: /var/log/audit:device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
UMOUNT: umounting vg01-lv_log (dm-6) mounted on /var/log
umount: /var/log:device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
UMOUNT: umounting vg01-lv_var (dm-1) mounted on /var
umount: /var:device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
[SKIP]: umount of vg01-lv_usr (dm-4) mounted on /usr
UMOUNT: umounting vg01-lv_tmp (dm-3) mounted on /tmp
umount: /tmp:device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
[SKIP]: umount of vg01-lv_swap (dm-0) mounted [SWAP]
UMOUNT: umounting vg01-lv_home (dm-3) mounted on /home
According with customer, is possible to shutdown this machine normally, however, these messages always are showed during this process.
This host is a VMware guest.
tks
Thanks for the feedback. The messages you see comes from the umount call that needs to be wrapped better. Anyway, this should not cause any harm, they're just messages that should have been better wrapped (and possibly the 'still-used' mountpoints better detected). I've opened a new bug #924137 for this. RHEL 6 The OS that cannot even shutdown cleanly. FYI I have a 6.4 system here which has just hung on this garbage script, "Stopping block device availability: Deactivating block devices:" Brilliant work guys. Please, attach the output of "lsblk", "lvs" and "dmsetup info -c" command and, if possible, the /var/log/messages content. Thanks. I am getting the same errors as Vitor and this is also a VM guest. Here is the output of the above requested commands. Running CentOS 6.4: [root@mdthlnvmwu036 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sda 8:0 0 35G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 34.5G 0 part ├─vg_mdthlnvmwu036-LVRoot (dm-0) 253:0 0 12G 0 lvm / ├─vg_mdthlnvmwu036-LVSwap (dm-1) 253:1 0 3.9G 0 lvm [SWAP] ├─vg_mdthlnvmwu036-LVVar (dm-2) 253:2 0 3.9G 0 lvm /var ├─vg_mdthlnvmwu036-LVVarLogAudit (dm-3) 253:3 0 3.9G 0 lvm /var/log/audit ├─vg_mdthlnvmwu036-LVTmp (dm-4) 253:4 0 6.9G 0 lvm /tmp └─vg_mdthlnvmwu036-LVVarLog (dm-5) 253:5 0 3.9G 0 lvm /var/log [root@mdthlnvmwu036 ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert LVRoot vg_mdthlnvmwu036 -wi-ao--- 12.05g LVSwap vg_mdthlnvmwu036 -wi-ao--- 3.91g LVTmp vg_mdthlnvmwu036 -wi-ao--- 6.84g LVVar vg_mdthlnvmwu036 -wi-ao--- 3.91g LVVarLog vg_mdthlnvmwu036 -wi-ao--- 3.91g LVVarLogAudit vg_mdthlnvmwu036 -wi-ao--- 3.91g [root@mdthlnvmwu036 ~]# dmsetup info -c Name Maj Min Stat Open Targ Event UUID vg_mdthlnvmwu036-LVSwap 253 1 L--w 1 1 0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpS3qzq1yQJ0vPk905PxiQFP9Kj2UR39Z0V vg_mdthlnvmwu036-LVVarLogAudit 253 3 L--w 1 1 0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpS98soFjJVqv2Afre62T48x9MWtuItS4Qn vg_mdthlnvmwu036-LVRoot 253 0 L--w 1 1 0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpSYAhfzybVBPU2Hd4pRuavqK2V6J8BRKHK vg_mdthlnvmwu036-LVTmp 253 4 L--w 1 1 0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpShPKhLdtDNogjoiJHIZpc3SSDB1bsTPGt vg_mdthlnvmwu036-LVVar 253 2 L--w 1 1 0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpSbyCKBR35cYPBtbgdkZhDlWHwn00vAi1h vg_mdthlnvmwu036-LVVarLog 253 5 L--w 1 1 0 LVM-gRbqwjztgbGDeJ9LtWU8VJMVaT1nsLpSw28SI5M4YC0hC2ZPG7QtCoPBFV2VTMHp Any progress on this issue? Thanks. Mike There's a patch added for blkdeactivate in RHEL 6.5 which changes the way the tool reports messages from external tools. Also, the /var and /var/log is not unmounted as other tools make use of it during shutdown (see also bug #924137 comment 10). |