Bug 1160903

Summary: RAID cache: Removing cache logical volume prints stray error message
Product: Red Hat Enterprise Linux 7 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
lvm2 sub component: Cache Logical Volumes QA Contact: Cluster QE <mspqa-list>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: agk, cmarthal, heinzm, jbrassow, lvm-team, msnitzer, mspqa-list, prajnoha, prockai, zkabelac
Version: 7.1   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: lvm2-2.02.113-1.el7 Doc Type: Bug Fix
Doc Text:
Removal of cache pool incorrectly released data and metadata if they were raid volumes. This has caused stray error messages to be printed. Order of removal of such raid volumes has been fixed.
Story Points: ---
Clone Of: 1104381 Environment:
Last Closed: 2015-03-05 13:10:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1104381    
Bug Blocks: 1119326    

Comment 1 Corey Marthaler 2014-11-05 23:21:31 UTC
The fix for this still needs to get pulled into the 7.1 build.

[root@host-110 ~]# lvs -a -o +devices
  LV                   VG  Attr       LSize   Pool         Origin      Devices
  lv                   vg  Cwi-a-C---  10.00g lv_cachepool [lv_corig]  lv_corig(0)
  lv_cachepool         vg  Cwi---C---   1.00g                          lv_cachepool_cdata(0)
  [lv_cachepool_cdata] vg  Cwi-------   1.00g                          /dev/sda1(2563)
  [lv_cachepool_cmeta] vg  ewi-------   8.00m                          /dev/sda1(2561)
  [lv_corig]           vg  rwi---C---  10.00g                          lv_corig_rimage_0(0),lv_corig_rimage_1(0)
  [lv_corig_rimage_0]  vg  Iwi-aor-r-  10.00g                          /dev/sda1(1)
  [lv_corig_rimage_1]  vg  Iwi-aor-r-  10.00g                          /dev/sdb1(1)
  [lv_corig_rmeta_0]   vg  ewi-aor-r-   4.00m                          /dev/sda1(0)
  [lv_corig_rmeta_1]   vg  ewi-aor-r-   4.00m                          /dev/sdb1(0)

[root@host-110 ~]# lvremove -ff vg
  _get_device_info: LVM-T0YHV5t4FvJzLwzsFNhha094P5k4KHUKHY8wA9fwed3G2afWnK1JxaBfed0nge6e: device not found
  WARNING: Integrity check of metadata for pool vg/lv_cachepool failed.
  Logical volume "lv" successfully removed
  Logical volume "lv_cachepool" successfully removed


lvm2-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
lvm2-libs-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
lvm2-cluster-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-libs-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-event-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-event-libs-1.02.90-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014
device-mapper-persistent-data-0.3.2-1.el7    BUILT: Thu Apr  3 09:58:51 CDT 2014
cmirror-2.02.111-1.el7    BUILT: Mon Sep 29 09:18:07 CDT 2014

Comment 3 Zdenek Kabelac 2014-11-10 21:53:26 UTC
Patch fixes the removal path has been pushed upstream as part of:
https://www.redhat.com/archives/lvm-devel/2014-November/msg00059.html
and related commits.

Comment 6 Corey Marthaler 2014-11-13 20:36:19 UTC
This doesn't appear to have made the .112-1 build.

3.10.0-163.el7.x86_64
lvm2-2.02.112-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014
lvm2-libs-2.02.112-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014
lvm2-cluster-2.02.112-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014
device-mapper-1.02.91-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014
device-mapper-libs-1.02.91-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014
device-mapper-event-1.02.91-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014
device-mapper-event-libs-1.02.91-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014
device-mapper-persistent-data-0.3.2-1.el7    BUILT: Thu Apr  3 09:58:51 CDT 2014
cmirror-2.02.112-1.el7    BUILT: Tue Nov 11 09:39:35 CST 2014

[root@host-109 ~]# lvs -a -o +devices
  LV                        Attr       LSize   Pool   Origin          Data%  Meta%  Cpy%Sync Devices
  corigin                   Cwi-a-C---   4.00g [pool] [corigin_corig] 15.57  3.47   0.00     corigin_corig(0)
  [corigin_corig]           rwi-aoC---   4.00g                                      100.00   corigin_corig_rimage_0(0),corigin_corig_rimage_1(0)
  [corigin_corig_rimage_0]  iwi-aor---   4.00g                                               /dev/sdb2(1)
  [corigin_corig_rimage_1]  iwi-aor---   4.00g                                               /dev/sda1(1)
  [corigin_corig_rmeta_0]   ewi-aor---   4.00m                                               /dev/sdb2(0)
  [corigin_corig_rmeta_1]   ewi-aor---   4.00m                                               /dev/sda1(0)
  [lvol0_pmspare]           ewi-------   8.00m                                               /dev/sdd2(516)
  [pool]                    Cwi---C---   2.00g                        15.57  3.47   0.00     pool_cdata(0)
  [pool_cdata]              Cwi-aor---   2.00g                                      100.00   pool_cdata_rimage_0(0),pool_cdata_rimage_1(0)      
  [pool_cdata_rimage_0]     iwi-aor---   2.00g                                               /dev/sdd2(1)
  [pool_cdata_rimage_1]     iwi-aor---   2.00g                                               /dev/sda2(1)
  [pool_cdata_rmeta_0]      ewi-aor---   4.00m                                               /dev/sdd2(0)
  [pool_cdata_rmeta_1]      ewi-aor---   4.00m                                               /dev/sda2(0)
  [pool_cmeta]              ewi-aor---   8.00m                                      100.00   pool_cmeta_rimage_0(0),pool_cmeta_rimage_1(0)      
  [pool_cmeta_rimage_0]     iwi-aor---   8.00m                                               /dev/sdd2(514)
  [pool_cmeta_rimage_1]     iwi-aor---   8.00m                                               /dev/sda2(514)
  [pool_cmeta_rmeta_0]      ewi-aor---   4.00m                                               /dev/sdd2(513)
  [pool_cmeta_rmeta_1]      ewi-aor---   4.00m                                               /dev/sda2(513)

[root@host-109 ~]# lvremove -f /dev/cache_sanity/corigin
  _get_device_info: LVM-GnqorGXMCg8OWRYbzjjkUjSpPovHL2oL067bt0p1PpwrCiMwNIqByFesqWwbztFX: device not found
  _get_device_info: LVM-GnqorGXMCg8OWRYbzjjkUjSpPovHL2oL067bt0p1PpwrCiMwNIqByFesqWwbztFX: device not found
  cache_sanity/corigin: raid1 segment monitoring function failed.
  Logical volume "pool" successfully removed
  Logical volume "corigin" successfully removed

Comment 7 Zdenek Kabelac 2014-11-25 10:47:29 UTC
Yes - 112 was buggy release for cache support since before the final freeze some last minute patches introduced more problems.

Should be fixed with 113:

[root@linux ~]# lvs -a
  LV                     VG   Attr       LSize  Pool    Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0]                vg   Cwi---C--- 10,00m                       1,25   0,27            0,00            
  [lvol0_cdata]          vg   Cwi-aor--- 10,00m                                              100,00          
  [lvol0_cdata_rimage_0] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cdata_rimage_1] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cdata_rmeta_0]  vg   ewi-aor--- 32,00k                                                              
  [lvol0_cdata_rmeta_1]  vg   ewi-aor--- 32,00k                                                              
  [lvol0_cmeta]          vg   ewi-aor--- 10,00m                                              100,00          
  [lvol0_cmeta_rimage_0] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cmeta_rimage_1] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cmeta_rmeta_0]  vg   ewi-aor--- 32,00k                                                              
  [lvol0_cmeta_rmeta_1]  vg   ewi-aor--- 32,00k                                                              
  [lvol1_pmspare]        vg   ewi------- 10,00m                                                              
  lvol2                  vg   Cwi-a-C--- 10,00m [lvol0] [lvol2_corig] 1,25   0,27            0,00            
  [lvol2_corig]          vg   owi-aoC--- 10,00m                                                              
[root@linux ~]# lvremove  -f vg
  Logical volume "lvol0" successfully removed
  Logical volume "lvol2" successfully removed
[root@linux ~]# lvs -a
[root@linux ~]#

Comment 8 Zdenek Kabelac 2014-11-25 10:50:06 UTC
[root@linux ~]# lvs -a
  LV                     VG   Attr       LSize  Pool    Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  [lvol0]                vg   Cwi---C--- 10,00m                       1,25   0,27            0,00            
  [lvol0_cdata]          vg   Cwi-aor--- 10,00m                                              100,00          
  [lvol0_cdata_rimage_0] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cdata_rimage_1] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cdata_rmeta_0]  vg   ewi-aor--- 32,00k                                                              
  [lvol0_cdata_rmeta_1]  vg   ewi-aor--- 32,00k                                                              
  [lvol0_cmeta]          vg   ewi-aor--- 10,00m                                              100,00          
  [lvol0_cmeta_rimage_0] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cmeta_rimage_1] vg   iwi-aor--- 10,00m                                                              
  [lvol0_cmeta_rmeta_0]  vg   ewi-aor--- 32,00k                                                              
  [lvol0_cmeta_rmeta_1]  vg   ewi-aor--- 32,00k                                                              
  lvol2                  vg   Cwi-a-C--- 10,00m [lvol0] [lvol2_corig] 1,25   0,27            0,00            
  [lvol2_corig]          vg   rwi-aoC--- 10,00m                                              100,00          
  [lvol2_corig_rimage_0] vg   iwi-aor--- 10,00m                                                              
  [lvol2_corig_rimage_1] vg   iwi-aor--- 10,00m                                                              
  [lvol2_corig_rmeta_0]  vg   ewi-aor--- 32,00k                                                              
  [lvol2_corig_rmeta_1]  vg   ewi-aor--- 32,00k                                                              
  [lvol3_pmspare]        vg   ewi------- 10,00m                                                              
[root@linux ~]# lvremove -f vg
  Logical volume "lvol0" successfully removed
  Logical volume "lvol2" successfully removed

Comment 10 Corey Marthaler 2014-12-04 00:07:00 UTC
Fix verified in the latest rpms.

3.10.0-206.el7.x86_64
lvm2-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
lvm2-libs-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
lvm2-cluster-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-libs-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-event-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-event-libs-1.02.92-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014
device-mapper-persistent-data-0.4.1-2.el7    BUILT: Wed Nov 12 12:39:46 CST 2014
cmirror-2.02.114-2.el7    BUILT: Mon Dec  1 10:57:14 CST 2014


[root@host-117 ~]# lvs -a -o +devices
  LV                             Attr       LSize   Pool            Origin          Data%  Meta% Cpy%Sync Devices
  corigin                        Cwi-a-C---   4.00g [display_cache] [corigin_corig] 0.02   2.31  0.00     corigin_corig(0)
  [corigin_corig]                rwi-aoC---   4.00g                                              100.00   corigin_corig_rimage_0(0),corigin_corig_rimage_1(0)            
  [corigin_corig_rimage_0]       iwi-aor---   4.00g                                                       /dev/sdd2(1)
  [corigin_corig_rimage_1]       iwi-aor---   4.00g                                                       /dev/sda1(1)
  [corigin_corig_rmeta_0]        ewi-aor---   4.00m                                                       /dev/sdd2(0)
  [corigin_corig_rmeta_1]        ewi-aor---   4.00m                                                       /dev/sda1(0)
  [display_cache]                Cwi---C---   2.00g                                 0.02   2.31  0.00     display_cache_cdata(0)
  [display_cache_cdata]          Cwi-aor---   2.00g                                              100.00   display_cache_cdata_rimage_0(0),display_cache_cdata_rimage_1(0)
  [display_cache_cdata_rimage_0] iwi-aor---   2.00g                                                       /dev/sde2(1)
  [display_cache_cdata_rimage_1] iwi-aor---   2.00g                                                       /dev/sdf2(1)
  [display_cache_cdata_rmeta_0]  ewi-aor---   4.00m                                                       /dev/sde2(0)
  [display_cache_cdata_rmeta_1]  ewi-aor---   4.00m                                                       /dev/sdf2(0)
  [display_cache_cmeta]          ewi-aor---  12.00m                                              100.00   display_cache_cmeta_rimage_0(0),display_cache_cmeta_rimage_1(0)
  [display_cache_cmeta_rimage_0] iwi-aor---  12.00m                                                       /dev/sde2(514)
  [display_cache_cmeta_rimage_1] iwi-aor---  12.00m                                                       /dev/sdf2(514)
  [display_cache_cmeta_rmeta_0]  ewi-aor---   4.00m                                                       /dev/sde2(513)
  [display_cache_cmeta_rmeta_1]  ewi-aor---   4.00m                                                       /dev/sdf2(513)
  [lvol0_pmspare]                ewi-------  12.00m                                                       /dev/sda2(0)

[root@host-117 ~]# lvremove -f /dev/cache_sanity/corigin
  Flushing cache for corigin.
  Logical volume "display_cache" successfully removed
  Logical volume "corigin" successfully removed

Comment 12 errata-xmlrpc 2015-03-05 13:10:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0513.html