Bug 1104821

Summary: Buffer I/O errors when configuring cache volumes
Product: Red Hat Enterprise Linux 7 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Zdenek Kabelac <zkabelac>
lvm2 sub component: Cache Logical Volumes QA Contact: Cluster QE <mspqa-list>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: agk, heinzm, jbrassow, jharriga, msnitzer, nperic, prajnoha, zkabelac
Version: 7.0   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.111-1.el7 Doc Type: Bug Fix
Doc Text:
Initial prove-of-concept cache implementation was not correctly handling numerous code paths and may have resulted into incorrectly loaded dm-table producing buffer-io errors. This has been fixed with newer release of lvm2.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 13:08:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1119326    
Attachments:
Description Flags
-vvvv of the lvconvert none

Description Corey Marthaler 2014-06-04 18:28:19 UTC
Description of problem:
This may be related to bug 1086426 and bug 1091553.

[root@host-073 ~]# pvscan
  PV /dev/sda1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]
  PV /dev/sdb1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]
  PV /dev/sdg1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]
  PV /dev/sdf1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]

[root@host-073 ~]# lvcreate -L 4G -n rename_orig_A cache_sanity /dev/sda1
  Logical volume "rename_orig_A" created
[root@host-073 ~]# lvcreate -L 4G -n rename_pool_A cache_sanity /dev/sdg1
  Logical volume "rename_pool_A" created
[root@host-073 ~]# lvcreate -L 8M -n rename_pool_A_meta cache_sanity /dev/sdg1
  Logical volume "rename_pool_A_meta" created


[root@host-073 ~]# lvconvert --type cache-pool --cachemode writeback --poolmetadata cache_sanity/rename_pool_A_meta cache_sanity/rename_pool_A
  Logical volume "lvol0" created
  Converted cache_sanity/rename_pool_A to cache pool.


Jun  4 12:37:08 host-073 kernel: Buffer I/O error on device dm-3, logical block 1048560
Jun  4 12:37:08 host-073 kernel: Buffer I/O error on device dm-3, logical block 1048560

[root@host-073 ~]# lvconvert --type cache --cachepool cache_sanity/rename_pool_A cache_sanity/rename_orig_A
  cache_sanity/rename_orig_A is now cached.


Jun  4 12:37:17 host-073 kernel: device-mapper: cache-policy-mq: version 1.2.0 loaded
Jun  4 12:37:17 host-073 kernel: bio: create slab <bio-0> at 0
Jun  4 12:37:17 host-073 kernel: TECH PREVIEW: DM cache may not be fully supported.
Jun  4 12:37:17 host-073 kernel: bio: create slab <bio-2> at 2

[root@host-073 ~]# lvcreate -s /dev/cache_sanity/rename_orig_A -c 64 -n snap1 -L 500M
  Logical volume "snap1" created

Jun  4 12:37:47 host-073 kernel: bio: create slab <bio-3> at 3
Jun  4 12:37:48 host-073 systemd: Starting Device-mapper event daemon...
Jun  4 12:37:48 host-073 systemd: Started Device-mapper event daemon.
Jun  4 12:37:48 host-073 kernel: Buffer I/O error on device dm-3, logical block 1048560
Jun  4 12:37:48 host-073 kernel: Buffer I/O error on device dm-3, logical block 1048560
Jun  4 12:37:48 host-073 dmeventd[3121]: dmeventd ready for processing.
Jun  4 12:37:48 host-073 lvm[3121]: Monitoring snapshot cache_sanity-snap1


[root@host-073 ~]# lvs -a -o +devices
  LV                    Attr       LSize   Pool          Origin                Data%   Devices
  [lvol0_pmspare]       ewi-------   8.00m                                             /dev/sda1(1024)
  rename_orig_A         owi-a-C---   4.00g rename_pool_A [rename_orig_A_corig]         rename_orig_A_corig(0)
  [rename_orig_A_corig] -wi-ao----   4.00g                                             /dev/sda1(0)
  rename_pool_A         Cwi-a-C---   4.00g                                             rename_pool_A_cdata(0)
  [rename_pool_A_cdata] Cwi-aoC---   4.00g                                             /dev/sdg1(0)
  [rename_pool_A_cmeta] ewi-aoC---   8.00m                                             /dev/sdg1(1024)
  snap1                 swi-a-s--- 500.00m               rename_orig_A           0.00  /dev/sda1(1026)


Version-Release number of selected component (if applicable):
3.10.0-123.el7.x86_64
lvm2-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
lvm2-libs-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
lvm2-cluster-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-libs-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-event-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-event-libs-1.02.84-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014
device-mapper-persistent-data-0.3.2-1.el7    BUILT: Thu Apr  3 09:58:51 CDT 2014
cmirror-2.02.105-14.el7    BUILT: Wed Mar 26 08:29:41 CDT 2014


How reproducible:
Everytime

Comment 2 Corey Marthaler 2014-06-04 19:56:46 UTC
This appears to be the easiest way to repo this issue:

[root@host-073 ~]# pvscan
  PV /dev/sda1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]
  PV /dev/sdb1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]
  PV /dev/sdg1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]
  PV /dev/sdf1   VG cache_sanity    lvm2 [15.00 GiB / 15.00 GiB free]
  PV /dev/vda2   VG rhel_host-073   lvm2 [7.51 GiB / 0    free]
  Total: 5 [67.49 GiB] / in use: 5 [67.49 GiB] / in no VG: 0 [0   ]

[root@host-073 ~]# lvcreate -L 4G -n corigin cache_sanity /dev/sda1
  Logical volume "corigin" created

[root@host-073 ~]# lvcreate -L 2G -n display_cache cache_sanity /dev/sdg1
  Logical volume "display_cache" created

[root@host-073 ~]# lvcreate -L 8M -n display_cache_meta cache_sanity /dev/sdg1
  Logical volume "display_cache_meta" created

# This command is the first to cause the IO errors
[root@host-073 ~]# lvconvert -vvvv --type cache-pool --cachemode writeback --poolmetadata cache_sanity/display_cache_meta cache_sanity/display_cache > /tmp/lvconvert 2>&1

Jun  4 14:46:11 host-073 kernel: bio: create slab <bio-0> at 0
Jun  4 14:47:41 host-073 kernel: Buffer I/O error on device dm-3, logical block 524272
Jun  4 14:47:41 host-073 kernel: Buffer I/O error on device dm-3, logical block 524272

[root@host-073 ~]# dmsetup info -c
Name                             Maj Min Stat Open Targ Event  UUID                                                                
cache_sanity-display_cache_cdata 253   5 L--w    0    1      0 LVM-QIgxlKHIAc56ZTD88xAnpG9LyyHrI38K7ME5rOK28epIOsaauqPBFdxcS2Umuk7N
cache_sanity-display_cache_cmeta 253   4 L--w    0    1      0 LVM-QIgxlKHIAc56ZTD88xAnpG9LyyHrI38KCXYQ1smXUF8E8PSHZvm6fMgUibfOD7eO
cache_sanity-display_cache       253   3 L--w    0    1      0 LVM-QIgxlKHIAc56ZTD88xAnpG9LyyHrI38KqLk2PGyBzWd7vXzDgx0K5pewciooc28V
cache_sanity-corigin             253   2 L--w    0    1      0 LVM-QIgxlKHIAc56ZTD88xAnpG9LyyHrI38KbGDoebUZDWWTf1mEFrOKw643IxWGrZMa
rhel_host--073-swap              253   1 L--w    2    1      0 LVM-pJyq8YF6yCLhwVVljtEDE79H2YAGBoYdMY6FeFiJST2GfHQ6V2fRkjPXi4oeTk3U
rhel_host--073-root              253   0 L--w    1    1      0 LVM-pJyq8YF6yCLhwVVljtEDE79H2YAGBoYdj1PeSNEOjGv8FFPTnYzxf9fKCJKahTAC

[root@host-073 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool Origin  Devices
  corigin               cache_sanity  -wi-a-----   4.00g              /dev/sda1(0)
  display_cache         cache_sanity  Cwi-a-C---   2.00g              display_cache_cdata(0)
  [display_cache_cdata] cache_sanity  Cwi-a-C---   2.00g              /dev/sdg1(0)
  [display_cache_cmeta] cache_sanity  ewi-a-C---   8.00m              /dev/sdg1(512)
  [lvol0_pmspare]       cache_sanity  ewi-------   8.00m              /dev/sda1(1024)
  root                  rhel_host-073 -wi-ao----   6.71g              /dev/vda2(205)
  swap                  rhel_host-073 -wi-ao---- 820.00m              /dev/vda2(0)

Comment 3 Corey Marthaler 2014-06-04 19:57:40 UTC
Created attachment 902313 [details]
-vvvv of the lvconvert

Comment 4 Zdenek Kabelac 2014-09-25 09:11:08 UTC
I expect this bug to be already fixed, please retest.

Comment 8 Nenad Peric 2015-01-16 16:48:14 UTC
Cannot find any weird messages during the cache creation/conversion.

it was successfully converted

[root@tardis-03 log]# lvs -a
  LV                    VG             Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  corigin               cache_sanity   -wi-a-----   4.00g                                                    
  display_cache         cache_sanity   Cwi---C---   2.00g                                                    
  [display_cache_cdata] cache_sanity   Cwi-------   2.00g                                                    
  [display_cache_cmeta] cache_sanity   ewi-------   8.00m                                                    
  [lvol0_pmspare]       cache_sanity   ewi-------   8.00m                                                    
  home                  rhel_tardis-03 -wi-ao---- 224.82g                                                    
  root                  rhel_tardis-03 -wi-ao----  50.00g                                                    
  swap                  rhel_tardis-03 -wi-ao----   4.00g                    

Marking VERIFIEd with:

3.10.0-223.el7.x86_64

lvm2-2.02.114-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015
lvm2-libs-2.02.114-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015
lvm2-cluster-2.02.114-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015
device-mapper-1.02.92-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015
device-mapper-libs-1.02.92-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015
device-mapper-event-1.02.92-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015
device-mapper-event-libs-1.02.92-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015
device-mapper-persistent-data-0.4.1-2.el7    BUILT: Wed Nov 12 19:39:46 CET 2014
cmirror-2.02.114-5.el7    BUILT: Wed Jan 14 15:42:28 CET 2015

Comment 10 errata-xmlrpc 2015-03-05 13:08:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0513.html