Bug 1086426

Summary: Problems attempting to remove cache pool before cache origin
Product: Red Hat Enterprise Linux 7 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Jonathan Earl Brassow <jbrassow>
lvm2 sub component: Cache Logical Volumes QA Contact: Cluster QE <mspqa-list>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified CC: agk, cmarthal, heinzm, jbrassow, jharriga, msnitzer, prajnoha, prockai, zkabelac
Version: 7.0   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.112-1.el7 Doc Type: Bug Fix
Doc Text:
No Documentation Needed. Cache was in tech preview previously.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 13:08:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1119326    

Description Corey Marthaler 2014-04-10 21:06:10 UTC
Description of problem:
## With VG still active

[root@harding-02 ~]# lvs -a -o +devices
  LV                    VG           Attr       LSize  Pool          Origin         Devices
  display_cache         cache_sanity Cwi-a-C---  1.00g                              display_cache_cdata(0)
  [display_cache_cdata] cache_sanity Cwi-aoC---  1.00g                              /dev/sdb2(0)
  [display_cache_cmeta] cache_sanity ewi-aoC---  8.00m                              /dev/sdb2(256)
  [lvol0_pmspare]       cache_sanity ewi-------  8.00m                              /dev/sdc2(0)
  origin                cache_sanity Cwi-a-C---  1.00g display_cache [origin_corig] origin_corig(0)
  [origin_corig]        cache_sanity -wi-ao----  1.00g                              /dev/sdb1(0)
[root@harding-02 ~]# lvremove cache_sanity/display_cache
  Flushing cache for origin
  6 blocks must still be flushed.
  0 blocks must still be flushed.
Do you really want to remove active logical volume display_cache? [y/n]: y
  Logical volume "display_cache" successfully removed

Apr 10 15:52:53 harding-02 kernel: [1308393.923225] quiet_error: 146 callbacks suppressed
Apr 10 15:52:53 harding-02 kernel: [1308393.928688] Buffer I/O error on device dm-6, logical block 262128
Apr 10 15:52:53 harding-02 kernel: [1308393.928750] Buffer I/O error on device dm-3, logical block 262128
Apr 10 15:52:53 harding-02 kernel: [1308393.928759] Buffer I/O error on device dm-3, logical block 262128
Apr 10 15:52:53 harding-02 kernel: [1308393.928769] Buffer I/O error on device dm-3, logical block 262142
Apr 10 15:52:53 harding-02 kernel: [1308393.928773] Buffer I/O error on device dm-3, logical block 262142
Apr 10 15:52:53 harding-02 kernel: [1308393.928779] Buffer I/O error on device dm-3, logical block 0
Apr 10 15:52:53 harding-02 kernel: [1308393.928783] Buffer I/O error on device dm-3, logical block 0
Apr 10 15:52:53 harding-02 kernel: [1308393.928791] Buffer I/O error on device dm-3, logical block 1
Apr 10 15:52:53 harding-02 kernel: [1308393.928803] Buffer I/O error on device dm-3, logical block 262143
Apr 10 15:52:53 harding-02 kernel: [1308393.928810] Buffer I/O error on device dm-3, logical block 262143


## With VG deactivated

[root@harding-02 ~]# lvs -a -o +devices
  LV                    VG           Attr       LSize  Pool          Origin         Devices
  display_cache         cache_sanity Cwi-a-C---  1.00g                              display_cache_cdata(0)
  [display_cache_cdata] cache_sanity Cwi-aoC---  1.00g                              /dev/sdb2(0)
  [display_cache_cmeta] cache_sanity ewi-aoC---  8.00m                              /dev/sdb2(256)
  [lvol0_pmspare]       cache_sanity ewi-------  8.00m                              /dev/sdc2(0)
  origin                cache_sanity Cwi-a-C---  1.00g display_cache [origin_corig] origin_corig(0)
  [origin_corig]        cache_sanity -wi-ao----  1.00g                              /dev/sdc1(0)
[root@harding-02 ~]# vgchange -an cache_sanity
  0 logical volume(s) in volume group "cache_sanity" now active
[root@harding-02 ~]# lvremove cache_sanity/display_cache
  Command failed with status code 5.


## Now remove origin before the cache pool

[root@harding-02 ~]# lvremove cache_sanity/origin
  Logical volume "origin" successfully removed
[root@harding-02 ~]# lvremove cache_sanity/display_cache
  Logical volume "display_cache" successfully removed



Version-Release number of selected component (if applicable):
3.10.0-110.el7.x86_64
lvm2-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
lvm2-libs-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
lvm2-cluster-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-libs-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-event-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-event-libs-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-persistent-data-0.2.8-4.el7    BUILT: Fri Jan 24 14:28:55 CST 2014
cmirror-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014


How reproducible:
Everytime

Comment 1 Corey Marthaler 2014-04-11 20:45:25 UTC
I assume this is the same issue, but renaming cache pools and cache origin volumes also results in Buffer I/O errors.

[root@host-029 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool          Origin            Devices               
  coriginB              cache_sanity  Cwi-a-C---   4.00g renameB_cache [coriginB_corig]  coriginB_corig(0)     
  [coriginB_corig]      cache_sanity  -wi-ao----   4.00g                                 /dev/sdg1(0)
  [lvol0_pmspare]       cache_sanity  ewi-------   8.00m                                 /dev/sdc1(0)
  renameB_cache         cache_sanity  Cwi-a-C---   2.00g                                 renameB_cache_cdata(0)
  [renameB_cache_cdata] cache_sanity  Cwi-aoC---   2.00g                                 /dev/sde1(0)
  [renameB_cache_cmeta] cache_sanity  ewi-aoC---   8.00m                                 /dev/sde1(512)

[root@host-029 ~]# lvrename cache_sanity/coriginB cache_sanity/coriginC
  Renamed "coriginB" to "coriginC" in volume group "cache_sanity"

Apr 11 15:39:13 host-029 kernel: Buffer I/O error on device dm-3, logical block 524272
Apr 11 15:39:13 host-029 kernel: Buffer I/O error on device dm-3, logical block 524272

[root@host-029 ~]# lvrename cache_sanity/renameB_cache cache_sanity/renameC_cache
  Renamed "renameB_cache" to "renameC_cache" in volume group "cache_sanity"

Apr 11 15:39:32 host-029 kernel: Buffer I/O error on device dm-3, logical block 524272
Apr 11 15:39:32 host-029 kernel: Buffer I/O error on device dm-3, logical block 524272
Apr 11 15:39:32 host-029 kernel: Buffer I/O error on device dm-3, logical block 524272
Apr 11 15:39:32 host-029 kernel: Buffer I/O error on device dm-3, logical block 524272

[root@host-029 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool          Origin            Devices               
  coriginC              cache_sanity  Cwi-a-C---   4.00g renameC_cache [coriginC_corig]  coriginC_corig(0)     
  [coriginC_corig]      cache_sanity  -wi-ao----   4.00g                                 /dev/sdg1(0)
  [lvol0_pmspare]       cache_sanity  ewi-------   8.00m                                 /dev/sdc1(0)
  renameC_cache         cache_sanity  Cwi-a-C---   2.00g                                 renameC_cache_cdata(0)
  [renameC_cache_cdata] cache_sanity  Cwi-aoC---   2.00g                                 /dev/sde1(0)
  [renameC_cache_cmeta] cache_sanity  ewi-aoC---   8.00m                                 /dev/sde1(512)

Comment 2 Zdenek Kabelac 2014-09-25 08:51:29 UTC
Should be fixed, please retest.

Comment 3 Jonathan Earl Brassow 2014-09-30 14:26:03 UTC
I was able to reproduce this with the older code.  With the latest code, I get:

ACTIVE:
[root@bp-01 ~]# cp -r linux-rhel6/ /mnt &
[root@bp-01 ~]# lvremove -ff vg/cachepool
  Flushing cache for lv.
  12840 blocks must still be flushed.
...
  109 blocks must still be flushed.
  0 blocks must still be flushed.
  Logical volume "cachepool" successfully removed

There are no I/O errors in the log.

INACTIVE:
[root@bp-01 ~]# vgchange -an vg
  0 logical volume(s) in volume group "vg" now active
[root@bp-01 ~]# lvremove vg/cachepool
  Flushing cache for lv.
  0 blocks must still be flushed.
  Logical volume "cachepool" successfully removed

Comment 6 Corey Marthaler 2014-11-05 23:07:50 UTC
This does appear fixed in the latest rpms.

lvm2-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
lvm2-libs-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
lvm2-cluster-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-libs-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-event-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-event-libs-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-persistent-data-0.3.2-1.el7    BUILT: Thu Apr  3 09:58:51 CDT 2014
cmirror-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014

Comment 8 Corey Marthaler 2014-12-04 22:50:13 UTC
Marking verified in the latest rpms. None of the issues in comment #0 appear any longer and cache pool renaming is no longer allowed.

[root@host-119 ~]# lvrename cache_sanity/rename_pool_A cache_sanity/rename_pool_B
  Cannot rename internal LV "rename_pool_A".


3.10.0-206.el7.x86_64
lvm2-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
lvm2-libs-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
lvm2-cluster-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-libs-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-event-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-event-libs-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-persistent-data-0.4.1-2.el7    BUILT: Wed Nov 12 12:39:46 CST 2014
cmirror-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014

Comment 10 errata-xmlrpc 2015-03-05 13:08:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0513.html