Bug 1711327

Summary: Repairing thin metadata is suggesting to use repair
Product: Red Hat Enterprise Linux 7 Reporter: Jakub Krysl <jkrysl>
Component: device-mapper-persistent-dataAssignee: Joe Thornber <thornber>
Status: CLOSED ERRATA QA Contact: Jakub Krysl <jkrysl>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.7CC: agk, heinzm, jbrassow, lvm-team, mcsontos, msnitzer, thornber
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: device-mapper-persistent-data-0.8.5-1.el7 Doc Type: No Doc Update
Doc Text:
In release only
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 13:17:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Complete setup log
none
dd'd metadata just before running thin_repair
none
dd'd metadata just after running thin_repair none

Description Jakub Krysl 2019-05-17 14:10:40 UTC
Created attachment 1570118 [details]
Complete setup log

Description of problem:
When using 'thin_repair' or 'thin_dump --repair' on corrupted swapped metadata, the output suggests to use '--repair'.

# lvs
  LV       VG     Attr       LSize   Pool     Origin Data%  Meta%  Move Log Cpy%Sync Convert
  swapvol  vgtest -wi-a-----   4.00m                                                        
  thinpool vgtest twi---tz-- 100.00m                                                        
  thinvol0 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol1 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol2 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol3 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol4 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol5 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol6 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol7 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol8 vgtest Vwi---tz--  12.00m thinpool                                               
  thinvol9 vgtest Vwi---tz--  12.00m thinpool  
# thin_check /dev/mapper/vgtest-swapvol 
examining superblock
TRANSACTION_ID=10
METADATA_FREE_BLOCKS=1023
examining devices tree
examining mapping tree
checking space map counts

### Breaking the metadata here ###
# thin_repair -i nonexistent -o /dev/mapper/vgtest-swapvol 
Couldn't stat path
# thin_check /dev/mapper/vgtest-swapvol
examining superblock
TRANSACTION_ID=10
METADATA_FREE_BLOCKS=1023
examining devices tree
examining mapping tree
checking space map counts
bad checksum in superblock, wanted 2711534584

# fallocate -l 5M /tmp/metadata
# thin_dump /dev/mapper/vgtest-swapvol -o /tmp/metadata --repair --format xml
metadata contains errors (run thin_check for details).
perhaps you wanted to run with --repair

# fallocate -l 5M /tmp/metadata
# thin_repair -i /dev/mapper/vgtest-swapvol -o /tmp/metadata 
metadata contains errors (run thin_check for details).
perhaps you wanted to run with --repair

Version-Release number of selected component (if applicable):
kernel-3.10.0-1048.el7.x86_64
device-mapper-persistent-data-0.8.1-1.el7.x86_64
device-mapper-1.02.158-1.el7.x86_64
lvm2-2.02.185-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. thin_repair -i nonexistent -o /dev/mapper/vgtest-swapvol
2a. fallocate -l 5M /tmp/metadata
3a. thin_dump /dev/mapper/vgtest-swapvol -o /tmp/metadata --repair --format xml

2b. fallocate -l 5M /tmp/metadata
3b. thin_repair -i /dev/mapper/vgtest-swapvol -o /tmp/metadata 

Actual results:
metadata contains errors (run thin_check for details).
perhaps you wanted to run with --repair

Expected results:
no message pointing to run --repair when already rea

Additional info:

Comment 3 Jakub Krysl 2019-05-27 13:25:19 UTC
Tested with device-mapper-persistent-data-0.8.2-1.el7.x86_64.rpm.

I cannot see "bad checksum in superblock" in thin_check, but 'thin_repair -i nonexistent' still does something to the metadata, as the output of 'thin_dump --repair' still fails after running it. It works before running 'thin_repair -i nonexistent'.

# thin_dump /dev/mapper/vgtest-swapvol -o /tmp/metadata --repair --format xml

# thin_check /dev/mapper/vgtest-swapvol 
examining superblock
TRANSACTION_ID=10
METADATA_FREE_BLOCKS=1023
examining devices tree
examining mapping tree
checking space map counts

# thin_repair -i nonexistent -o /dev/mapper/vgtest-swapvol
Couldn't stat path

# thin_check /dev/mapper/vgtest-swapvol 
examining superblock
TRANSACTION_ID=10
METADATA_FREE_BLOCKS=1023
examining devices tree
examining mapping tree
checking space map counts

# thin_dump /dev/mapper/vgtest-swapvol -o /tmp/metadata --repair --format xml
metadata contains errors (run thin_check for details).
perhaps you wanted to run with --repair


Attaching binary dump of metadata (using dd) before and after running 'thin_repair -i nonexistent -o /dev/mapper/vgtest-swapvol' in following comments.

Returning back after discussion with Marian.

Comment 4 Jakub Krysl 2019-05-27 13:26:31 UTC
Created attachment 1573927 [details]
dd'd metadata just before running thin_repair

Comment 5 Jakub Krysl 2019-05-27 13:27:09 UTC
Created attachment 1573928 [details]
dd'd metadata just after running thin_repair

Comment 6 Jakub Krysl 2019-05-27 15:15:25 UTC
Compared to device-mapper-persistent-data-0.7.3-3.el7.x86_64 using another reproducer:
# echo 'nothing' >> /dev/mapper/vgtest-swapvol
# thin_check /dev/mapper/vgtest-swapvol                                                                                                                                 
examining superblock
  superblock is corrupt
    bad checksum in superblock, wanted 1684593029
# thin_repair -i /dev/mapper/vgtest-swapvol -o /tmp/metadata                                                                                                           
bad checksum in superblock, wanted 1684593029
# thin_repair -i /tmp/metadata -o /dev/mapper/vgtest-swapvol                                                                                                          
metadata contains errors (run thin_check for details).
perhaps you wanted to run with --repair


Before the output looked like this:
# echo 'nothing' >> /dev/mapper/vgtest-swapvol                                                                                        
# thin_check /dev/mapper/vgtest-swapvol                                                                                               
examining superblock
  superblock is corrupt
    bad checksum in superblock, wanted 3627560579
# thin_repair -i /dev/mapper/vgtest-swapvol -o /tmp/metadata
bad checksum in superblock, wanted 3627560579
# thin_repair -i /tmp/metadata -o /dev/mapper/vgtest-swapvol
bad checksum in superblock, wanted 1490015127

Comment 8 Jakub Krysl 2019-06-27 14:14:59 UTC
The issue is no longer present in device-mapper-persistent-data-0.8.5-1.el7.x86_64:

# fallocate -l 10M /tmp/metadata
# thin_dump /dev/mapper/vgtest-swapvol -o /tmp/metadata --repair --format xml
# thin_check /dev/mapper/vgtest-swapvol
examining superblock
TRANSACTION_ID=10
METADATA_FREE_BLOCKS=1023
examining devices tree
examining mapping tree
checking space map counts
# thin_repair -i nonexistent -o /dev/mapper/vgtest-swapvol
Couldn't stat path
# thin_check /dev/mapper/vgtest-swapvol
examining superblock
TRANSACTION_ID=10
METADATA_FREE_BLOCKS=1023
examining devices tree
examining mapping tree
checking space map counts
# thin_dump /dev/mapper/vgtest-swapvol -o /tmp/metadata --repair --format xml
#


# echo 'nothing' >> /dev/mapper/vgtest-swapvol
# thin_check /dev/mapper/vgtest-swapvol
examining superblock
  superblock is corrupt
    bad checksum in superblock, wanted 3208244287
# fallocate -l 50M /tmp/metadata
# thin_repair -i /dev/mapper/vgtest-swapvol -o /tmp/metadata
bad checksum in superblock, wanted 3208244287

Comment 10 errata-xmlrpc 2019-08-06 13:17:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2320