Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: 1. It was power shortage 2. And probably the LV space is 0 3. After power recovery LVM volume pve/data is not recovered. 4. I've updated thin-provisioning-tools to version from github 0.9.0 5. Trying to recover system with lvconvert --repair -vv /dev/pve/data 5.1 no space on vg 5.2 my junior it speciallist delete couple of lv. Dumpest idea, and not helped 6. Add additional USB drive to VG for space 7. Trying to recover system with lvconvert --repair -vv /dev/pve/data it takes more than 8 hours and accidentally was stopped by my junior IT specialist 8. If i'm trying to recover LVM again I got nothing 9. With help from Zdenek Kabelec we managed to thin recover proper meta volume, it takes some time, but got the following # thin_dump /dev/pve/recov <superblock uuid="" time="1" transaction="56" flags="0" version="2" data_block_size="128" nr_data_blocks="58531776"> </superblock> Server failure happened later this day /var/log/syslog.2.gz May 3 15:22:27 pve01 kernel: [ 77.720453] device-mapper: space map common: index_check failed: csum 3350383422 != wanted 3 350290675 May 3 15:22:27 pve01 kernel: [ 77.720540] device-mapper: block manager: index validator check failed for block 1362901 May 3 15:22:27 pve01 kernel: [ 77.720618] device-mapper: transaction manager: couldn't open metadata space map May 3 15:22:27 pve01 kernel: [ 77.720691] device-mapper: thin metadata: tm_open_with_sm failed May 3 15:22:27 pve01 kernel: [ 77.743305] device-mapper: table: 253:4: thin-pool: Error creating metadata object May 3 15:22:27 pve01 kernel: [ 77.743392] device-mapper: ioctl: error adding target to table Lvm on raid array meta pool dump in here - https://agregator.kz/tmeta.bin.bz2 root@pve01:~# lvs -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data pve twi---tz-- <3.49t data_meta0 pve -wi-a----- 15.81g data_meta1 pve -wi-a----- 15.81g data_meta10 pve -wi-a----- 15.81g data_meta2 pve -wi-a----- 15.81g data_meta3 pve -wi-a----- 15.81g data_meta4 pve -wi-a----- 15.81g data_meta5 pve -wi-a----- 15.81g data_meta6 pve -wi-a----- 15.81g data_meta7 pve -wi-a----- 15.81g data_meta8 pve -wi-a----- 15.81g data_meta9 pve -wi-a----- 15.81g [data_tdata] pve Twi-a----- <3.49t [data_tmeta] pve ewi-a----- 16.00g dump pve Vwi---tz-- 400.00g data [lvol7_pmspare] pve ewi------- 15.81g recov pve -wi-a----- 16.00g repaired_01 pve -wi-a----- 15.81g root pve -wi-ao---- 96.00g snap_vm-104-disk-0_init pve Vri---tz-k 100.00g data vm-104-disk-0 swap pve -wi-ao---- 8.00g vm-100-disk-0 pve Vwi---tz-- 50.00g data vm-101-disk-0 pve Vwi---tz-- 100.00g data vm-102-disk-0 pve Vwi---tz-- 100.00g data vm-102-disk-1 pve Vwi---tz-- 1.00t data vm-103-disk-0 pve Vwi---tz-- 100.00g data vm-104-disk-0 pve Vwi---tz-- 100.00g data vm-105-disk-0 pve Vwi---tz-- 100.00g data vm-107-disk-0 pve Vwi---tz-- 500.00g data vm-107-disk-1 pve Vwi---tz-- 500.00g data vm-107-disk-2 pve Vwi---tz-- 500.00g data vm-108-disk-0 pve Vwi---tz-- 200.00g data vm-110-disk-0 pve Vwi---tz-- 150.00g data vm-110-disk-1 pve Vwi---tz-- 100.00g data vm-112-disk-0 pve Vwi---tz-- 120.00g data deleted volumes vm-131-disk-0 vm-131-disk-1 Version-Release number of selected component (if applicable): LVM version: 2.03.02(2) (2018-12-18) Library version: 1.02.155 (2018-12-18) Driver version: 4.41.0 thin-provisioning-tools to version from github 0.9.0 How reproducible: Have no idea Expected results: Please help to restore data Additional info:
I'd be far from saying 'proper' - since recovered result is really just 'empty' set. But please also attach 'vgcfgbackup' of existing lvm2 volume group - so we could eventually see pairing between LVs & kernel metadata - if there will be anything recoverable. Also - has the 'power outage' happened in the moment of thin-pool being out-of-space ? Was it out of data or metadata ? Was this before this date: May 3 15:22:27 ? (Is thin_checking disabled ? - please attach also lvm.conf)
Created attachment 1877391 [details] vgcfgbackup of lvm2 pool
Created attachment 1877392 [details] lvm.conf
>Also - has the 'power outage' happened in the moment of thin-pool being out-of-space ? No, there was no space for recover process. >Was it out of data or metadata ? Data and metadata has spare space before power outage I'm not shure for 100%
I'm checking the metadata to see if there's any chance to repair it in a shorter time.
Renat: Could you help provide the lvm archive please? (files under /etc/lvm/)
Created attachment 1877635 [details] /etc/lvm directory
Created attachment 1877999 [details] Part1 of repaired metadata dump
Created attachment 1878002 [details] Part2 of repaired metadata dump
Renat: I'm able to dump the metadata using the new Rust version of thin_dump, so your data should be fine. For your convenience I uploaded the metadata dump as shown in attachments. Download and merge them first: # cat thin_dump_bz2082197.tar.xz.00* > thin_dump_bz2082197.tar.xz The merged xz file has md5sum 42c7192fe0069854d6aec449f455dce5 Decompress the file, thin_restore the text file into a new metadata, then swap the pool metadata, and you should be able to activate the pool. # tar -xf thin_dump_bz2082197.tar.xz # lvcreate pve --name repaired --size 15g # thin_restore -i thin_dump.txt -o /dev/pve/repaired # lvconvert --thinpool pve/data --poolmetadata pve/repaired Please note that the metadata comes with the 4 recent deleted voluems, i.e., vm-131-disk-0 (15), vm-131-disk-1 (16), vm-111-disk-0 (29), and vm-111-disk-1 (30) are still available. Delete them manually if they are no longer used: # dmsetup message pve-data-tpool 0 "delete 15" # dmsetup message pve-data-tpool 0 "delete 16" # dmsetup message pve-data-tpool 0 "delete 29" # dmsetup message pve-data-tpool 0 "delete 30"
Good day Im try the following root@pve01:/# lvcreate pve --name repaired --size 15g File descriptor 7 (pipe:[10134482]) leaked on lvcreate invocation. Parent PID 10417: bash WARNING: Sum of all thin volume sizes (<4.05 TiB) exceeds the size of thin pools and the amount of free space in volume group (<3.47 TiB). WARNING: You have not turned on protection against thin pools running out of space. WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full. Logical volume "repaired" created. root@pve01:/# thin_restore -i thin_dump.txt -o /dev/pve/repaired Restoring: [==================================================] 100% root@pve01:/# lvconvert --thinpool pve/data --poolmetadata pve/repaired File descriptor 7 (pipe:[10134482]) leaked on lvconvert invocation. Parent PID 10417: bash Do you want to swap metadata of pve/data pool with metadata volume pve/repaired? [y/n]: y root@pve01:/# lvs -a File descriptor 7 (pipe:[10134482]) leaked on lvs invocation. Parent PID 10417: LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data pve twi---tz-- <3.49t data_meta0 pve -wi-a----- 15.81g data_meta1 pve -wi-a----- 15.81g data_meta10 pve -wi-a----- 15.81g data_meta2 pve -wi-a----- 15.81g data_meta3 pve -wi-a----- 15.81g data_meta4 pve -wi-a----- 15.81g data_meta5 pve -wi-a----- 15.81g data_meta6 pve -wi-a----- 15.81g data_meta7 pve -wi-a----- 15.81g data_meta8 pve -wi-a----- 15.81g data_meta9 pve -wi-a----- 15.81g [data_tdata] pve Twi-a----- <3.49t [data_tmeta] pve ewi------- 15.00g dump pve Vwi---tz-- 400.00g data [lvol7_pmspare] pve ewi------- 15.81g recov pve -wi-a----- 16.00g repaired pve -wi------- 16.00g repaired_01 pve -wi-a----- 15.81g root pve -wi-ao---- 96.00g snap_vm-104-disk-0_init pve Vri---tz-k 100.00g data vm-104-disk-0 swap pve -wi-ao---- 8.00g vm-100-disk-0 pve Vwi---tz-- 50.00g data vm-101-disk-0 pve Vwi---tz-- 100.00g data vm-102-disk-0 pve Vwi---tz-- 100.00g data vm-102-disk-1 pve Vwi---tz-- 1.00t data vm-103-disk-0 pve Vwi---tz-- 100.00g data vm-104-disk-0 pve Vwi---tz-- 100.00g data vm-105-disk-0 pve Vwi---tz-- 100.00g data vm-107-disk-0 pve Vwi---tz-- 500.00g data vm-107-disk-1 pve Vwi---tz-- 500.00g data vm-107-disk-2 pve Vwi---tz-- 500.00g data vm-108-disk-0 pve Vwi---tz-- 200.00g data vm-110-disk-0 pve Vwi---tz-- 150.00g data vm-110-disk-1 pve Vwi---tz-- 100.00g data vm-112-disk-0 pve Vwi---tz-- 120.00g data root@pve01:/# lvchange -ay pve/data File descriptor 7 (pipe:[10134482]) leaked on lvchange invocation. Parent PID 10417: bash Activation of logical volume pve/data is prohibited while logical volume pve/data_tdata is active. root@pve01:/# lvchange -an pve/data_tdata File descriptor 7 (pipe:[10134482]) leaked on lvchange invocation. Parent PID 10417: bash root@pve01:/# lvchange -ay pve/data File descriptor 7 (pipe:[10134482]) leaked on lvchange invocation. Parent PID 10417: bash device-mapper: create ioctl on pve-data_tmeta LVM-UyQiQTeLIPZImpnZ6peOr2OV6cx4qnO2crKJO5DKnBcWLal6A5SQRQWFXTlkaNS1-tmeta failed: Device or resource busy root@pve01:/# init 6 System had a long time to restart and root@pve01:~# lvs -a ^C Interrupted... Giving up waiting for lock. Can't get lock for pve Cannot process volume group pve root@pve01:~# top top - 09:35:00 up 31 min, 1 user, load average: 0.37, 0.43, 0.28 Tasks: 545 total, 2 running, 543 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.4 us, 0.1 sy, 0.0 ni, 99.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 128583.3 total, 126101.3 free, 2158.8 used, 323.2 buff/cache MiB Swap: 8192.0 total, 8192.0 free, 0.0 used. 125615.4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2827 root 2 -18 47808 44272 4392 R 18.3 0.0 0:15.62 thin_check [ 1801.166113] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 1905.268175] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2009.206005] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2113.135317] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2217.105121] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2321.807378] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2428.581006] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2532.762836] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2636.392753] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2740.191360] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2844.113516] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 2948.174878] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3054.802068] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3158.855834] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3270.008936] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3373.549868] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3478.099934] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3582.197472] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3692.736574] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3796.689358] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 3900.706434] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4004.592602] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4109.102283] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4213.180140] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4320.189620] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4424.227704] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4528.252644] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4632.050268] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4735.855582] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4840.677899] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 4947.710789] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. [ 5057.816641] device-mapper: thin: Data device (dm-17) discard unsupported: Disabling discard passdown. root@pve01:~# lvchange -ay pve/data Thin pool pve-data-tpool (253:18) transaction_id is 56, while expected 60. Any advice?
The transaction id is not syned due to previous errors. It's suggested to advance the tid in pool metadata in this situation. Open the thin_dump.txt, modify the first line from: <superblock uuid="" time="1" transaction="56" flags="0" version="2" data_block_size="128" nr_data_blocks="58531776"> to <superblock uuid="" time="1" transaction="60" flags="0" version="2" data_block_size="128" nr_data_blocks="58531776"> And redo the steps above (thin_restore & lvconvert)
Another optional way could be to just update lvm2 metadata to the expected transaction id. vgcfgbackup/ fix by vi editor / vgcfgrestore --force Also maybe directly the largest possible size should have been used - but if the 15G is ATM enough to fit in metadata - it's not a big issue (and there is still some 'space' to grow-up metadata in size). Also - don't forget to remove all useless data_meta{1..10} metadata volumes from your repair trials - they are useless and just take up space - the only useful one was data_meta0 used for repair.
(In reply to Zdenek Kabelac from comment #13) > Another optional way could be to just update lvm2 metadata to the expected > transaction id. > > vgcfgbackup/ fix by vi editor / vgcfgrestore --force y, that's the faster way, and should be safe since all the thins are created before transaction #56. > Also maybe directly the largest possible size should have been used - but if > the 15G is ATM enough to fit in metadata - it's not a big issue (and there > is still some 'space' to grow-up metadata in size). 15G is enough for the pve pool. It just consumes 1.3G.
(In reply to Ming-Hung Tsai from comment #12) > The transaction id is not syned due to previous errors. It's suggested to > advance the tid in pool metadata in this situation. Open the thin_dump.txt, > modify the first line from: > > <superblock uuid="" time="1" transaction="56" flags="0" version="2" > data_block_size="128" nr_data_blocks="58531776"> > > to > > <superblock uuid="" time="1" transaction="60" flags="0" version="2" > data_block_size="128" nr_data_blocks="58531776"> > > And redo the steps above (thin_restore & lvconvert) Done. Its working again. Thank you guys!!!
Closing - although there should be made some progress with public release of new tooling.
Follow up information: The v0.9.0 release does not support metadata with empty devices (e.g., devices 13 and 29 in this case). Moreover, this version does not skip unused nodes while finding the root pairs, that's why thin_dump spent so much time in this case but output nothing. These two issues had been addressed by upstream commits d1e8168f and e6f17d4b. The first one had been backported to RHEL9 in bug 2020660. I'll open another ticket for the second issue.