Red Hat Bugzilla – Bug 211235
apparent file system corruption of snapshot fs
Last modified: 2012-03-22 08:09:48 EDT
Description of problem: I created an origin volume, put an ext filesystem on it, used 'check-it' to write random files to that fs and then verify them, created a snapshot of that volume, and then attempted to verify the snapshots data with 'check-it' as well. That verify failed. I'll try to reproduce this and gather more information. SCENARIO - [fs_io_to_snaps] Create snapshots of origin with fs data and verify that data on snapshot Making origin volume Placing an ext filesystem on origin volume mke2fs 1.35 (28-Feb-2004) Mounting origin volume Writing files to /mnt/origin checkit starting with: CREATE Num files: 500 Random Seed: 4321 Verify XIOR Stream: /tmp/checkit_origin_1 Working dir: /mnt/origin Checking files on /mnt/origin checkit starting with: VERIFY Verify XIOR Stream: /tmp/checkit_origin_1 Working dir: /mnt/origin Making first snapshot of origin volume Mounting 1st snap volume Checking files on /mnt/fs_snap1 checkit starting with: VERIFY Verify XIOR Stream: /tmp/checkit_origin_1 Working dir: /mnt/fs_snap1 Can not stat ycvbn: Input/output error checkit verifiy failed [root@link-08 ~]# ls -l /mnt/fs_snap1/ total 16 ?--------- ? ? ? ? ? asevvlghcnttmnrqrndk ?--------- ? ? ? ? ? asqgwgpnbycmmbicqwotwfiyngsvbheb ?--------- ? ? ? ? ? ax ?--------- ? ? ? ? ? bdojjdapcvsntyniqsenxhmwgnactfawkpgtujlxhhnbgbmyv ?--------- ? ? ? ? ? bfislrhxncclnjodc ?--------- ? ? ? ? ? bgjifdisttoxiefdbylqfsotmneyugjxnt ?--------- ? ? ? ? ? bjcxstgmgmivtr ?--------- ? ? ? ? ? bqlceahjrumfearrnppswtfmwlxpb ?--------- ? ? ? ? ? cpmigrpmdieafsjmrvbbjqovfinkdlghdvqjngxsrduwxfl ?--------- ? ? ? ? ? cwileceuhniovnakcjyean ?--------- ? ? ? ? ? djmpvgirohiduwcitaoaukqindbiitblepdcvltmscpp ?--------- ? ? ? ? ? djpjqjisemqmdekowjmyiypimyvflbbqmscddkxhwovbsip ?--------- ? ? ? ? ? dwrc ?--------- ? ? ? ? ? egeummgwxnjsjcrjnykmitsjykcu ?--------- ? ? ? ? ? emmkseyrpjieplqtrnqaqttwmxbcur ?--------- ? ? ? ? ? eqbbcdmygnmiwoidgsrjocqpewnbrcqyvraxxoy ?--------- ? ? ? ? ? eubrlxolbtrrfuobgwljeegmewrcn ?--------- ? ? ? ? ? ffvipfvbeeffhyvyguymmtkyyxpiywdeecpthluopdty ?--------- ? ? ? ? ? fijbdkl ?--------- ? ? ? ? ? fjepvnbdofbmkcxcmp ?--------- ? ? ? ? ? fkcenwrwjfqvnbigbkcwfebxmoguaiffuh ?--------- ? ? ? ? ? fpokbdgaptgtqbilgcsbnfihnomsjxgqnvbry ?--------- ? ? ? ? ? fuakpcwnugkepqmtsegssniebjffjyqowqbo ?--------- ? ? ? ? ? fysavfbglfsyjxllcnrtdrmekkqccrqhtkkprl ?--------- ? ? ? ? ? gaojhsvbowrdfsa ?--------- ? ? ? ? ? gecnoaqwuprarlwavketdghiled ?--------- ? ? ? ? ? gndkjefqprkhpbgp ?--------- ? ? ? ? ? gvcrsrdnrtikqmgmhqugpjkqdusakolqknk ?--------- ? ? ? ? ? gwbki ?--------- ? ? ? ? ? gwlceuptmudnublqclwgxskhnwegcngkmtmtodpdassxugqys ?--------- ? ? ? ? ? hi ?--------- ? ? ? ? ? htkaxsrvdauvufomjrsqa ?--------- ? ? ? ? ? htnfxfiqgdyokusjkjavhfmhufruhrlqnbvlggcnl ?--------- ? ? ? ? ? hulfwxxckgiwdjipjssntwabahfl ?--------- ? ? ? ? ? ikiyajvfgnkatbkonc ?--------- ? ? ? ? ? injvprblmnaircedmgwbpfeeaowdsmadcjytdbipqixime ?--------- ? ? ? ? ? iorvyv ?--------- ? ? ? ? ? ivbslobgvspayjskteajandaetbtponbkovyfyfbrv ?--------- ? ? ? ? ? j ?--------- ? ? ? ? ? jipqbhflkgnejvttnlxqqnsrnuwbsmtcvkuytbkgjakt ?--------- ? ? ? ? ? jmrlduqinlfewvqaikvvgxkbsqxgypoj ?--------- ? ? ? ? ? khkbcrbxvguwpctubfeukvxuoajsswcfe ?--------- ? ? ? ? ? klyfeknnfnj ?--------- ? ? ? ? ? kxglxjrfvedps ?--------- ? ? ? ? ? lbffggaawkcvjbplgimjgno ?--------- ? ? ? ? ? lefmugidccthwxgoamjyfegfh drwx------ 2 root root 16384 Oct 17 13:23 lost+found ?--------- ? ? ? ? ? lupcrfuftujacskjjvrcfsfojysod ?--------- ? ? ? ? ? lyhhglxgsctdjsylnmmbsababxpwmo ?--------- ? ? ? ? ? majiugwhce ?--------- ? ? ? ? ? mdfxmgjgiviurpwujnjuwdfrhs ?--------- ? ? ? ? ? mlrhqsgvuqkcf ?--------- ? ? ? ? ? mxehliiablptgviwp ?--------- ? ? ? ? ? nabynccxfnktdumsjymlvhuomawdmaja ?--------- ? ? ? ? ? nlahgphipvqfxrfwnxbapddmqmlrcotpcvaimhsefkjfcqcrpe ?--------- ? ? ? ? ? ntvhqaoxpgkupjowxoiegifitixqtrdjlyqe ?--------- ? ? ? ? ? nxhkiwmrvxxbaealrbqpxntnpnafdrorqye ?--------- ? ? ? ? ? ojjkn ?--------- ? ? ? ? ? oqefbntqunnnxkfnhevitbkcjbhclgrcavhdmdth ?--------- ? ? ? ? ? oysfyhtnsojqkmlnbhphrhmhxdfaajxplrwkbrbuiknsyaiah ?--------- ? ? ? ? ? paidaouhvrxooobcmqidqfggwnvvasppsbuuppelkcdbs ?--------- ? ? ? ? ? persaodgfivxgksywxfka ?--------- ? ? ? ? ? phfoastdvfbgjxijergwbnvposchqgdiqkwqdswbyajix ?--------- ? ? ? ? ? piysthagxsrdgqwrfqryvjmggfxofhmurlnnvqttjoyshxkmp ?--------- ? ? ? ? ? puffhmubwrjgdawglpkwnfcu ?--------- ? ? ? ? ? qmjeuxtmyweikxse ?--------- ? ? ? ? ? qwnpaynvejrvgyyxsohkycuoulacwkh ?--------- ? ? ? ? ? rgfvflmhskusduiouvlqgpecjsbfvmtqsymxmyh ?--------- ? ? ? ? ? rgkuxklvpsdgonftrkihmvmesxwhwcyqilmjwahosku ?--------- ? ? ? ? ? rhqgfnuohumwxi ?--------- ? ? ? ? ? rjjcfaqjgrupfjmgemkwqnwxollactqwedylfswl ?--------- ? ? ? ? ? rjprabbogumxmowuaifpeohx ?--------- ? ? ? ? ? rsrxtgabywhbtrlvxdjtunwlsqpwpk ?--------- ? ? ? ? ? segfs ?--------- ? ? ? ? ? sidnsyqhpkxutcdkk ?--------- ? ? ? ? ? sifxflsahd ?--------- ? ? ? ? ? swkowtcyudmnbufmymt ?--------- ? ? ? ? ? tcj ?--------- ? ? ? ? ? tcqmaatakfsqlhdnx ?--------- ? ? ? ? ? tdocrdno ?--------- ? ? ? ? ? tgjagyyujyxdhsbptmmoyfsltvecdwqwdawmcx ?--------- ? ? ? ? ? tqrxxisssctejgjrvygsiwodcala ?--------- ? ? ? ? ? ttqefpadgacsjplyavyvcvyqmbxjdeswald ?--------- ? ? ? ? ? txerofnemqugjmqekmiouqemayvvgrqarurgchlqaiwmxp ?--------- ? ? ? ? ? ubdwjbtgqqkhibfqtpvcntbbcnrsouhlxmjjnf ?--------- ? ? ? ? ? ugmjeioqgcwqitkqktbueyeqorywwr ?--------- ? ? ? ? ? uikispa ?--------- ? ? ? ? ? uknmjbylujgkncuayvaexlaeodtvkuuhhktqltegdmqtpntq ?--------- ? ? ? ? ? uooubjqgwdqelldaluotwnfjlfc ?--------- ? ? ? ? ? utcyymeghhuxv ?--------- ? ? ? ? ? vcdknbwocittfenjhnitnavxberpee ?--------- ? ? ? ? ? vlqxuafkqdteihunkeuxvivyqfnvdmjyxaysa ?--------- ? ? ? ? ? vpjugthynbbsynxlflliibsjmbkltkgpcpkjktkavlsubs ?--------- ? ? ? ? ? vupqbjqfbdwux ?--------- ? ? ? ? ? vuxbkpybrsyesdevbowsmleyioujkrhgmhkyxjapcctwfy ?--------- ? ? ? ? ? wysttktjaibiuarshsmhehskwmprtpdqqvkn ?--------- ? ? ? ? ? xhaxehwjye ?--------- ? ? ? ? ? xhidlpcnsucketafrmlmpfdsavuwawxyeiesaiitdmgkgjpav ?--------- ? ? ? ? ? xkvecqkticwlcticqarmrnfmawwxeihevd ?--------- ? ? ? ? ? xpbroguntoqttupkrbbemydikaymsit ?--------- ? ? ? ? ? xtyhdlmkuqw ?--------- ? ? ? ? ? xuktqllvkaqxtylojboab ?--------- ? ? ? ? ? xxmslarihkntrjumangrvafaxqidtu ?--------- ? ? ? ? ? xyrtfdchgewtmymn ?--------- ? ? ? ? ? ycvbn ?--------- ? ? ? ? ? yddoabxiwuq ?--------- ? ? ? ? ? yfhbovvqqn ?--------- ? ? ? ? ? yic ?--------- ? ? ? ? ? yjcfqqaxjisksnrakdjy ?--------- ? ? ? ? ? yjk ?--------- ? ? ? ? ? yosecmxlaxq ?--------- ? ? ? ? ? ypbeffunyhylkpghbwncfxjnpcdyjvhilk ?--------- ? ? ? ? ? yrbtywmjweqgbuthcaxyajg ?--------- ? ? ? ? ? yxnuyragykigkwyp [root@link-08 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 72G 2.2G 66G 4% / /dev/hda1 99M 45M 49M 48% /boot none 500M 0 500M 0% /dev/shm /dev/mapper/snapper-origin 4.0G 156M 3.6G 5% /mnt/origin /dev/mapper/snapper-fs_snap1 4.0G 8.0M 3.8G 1% /mnt/fs_snap1 [root@link-08 ~]# lvs -a -o+devices LV VG Attr LSize Origin Snap% Move Log Copy% Devices fs_snap1 snapper swi-ao 2.50G origin 3.94 /dev/sdc1(0) origin snapper owi-ao 4.00G /dev/sdb1(0) [root@link-08 ~]# dmsetup ls snapper-origin-real (253, 3) VolGroup00-LogVol01 (253, 1) snapper-fs_snap1-cow (253, 4) VolGroup00-LogVol00 (253, 0) snapper-fs_snap1 (253, 5) snapper-origin (253, 2) Version-Release number of selected component (if applicable): [root@link-08 ~]# rpm -q lvm2 lvm2-2.02.06-6.0.RHEL4 [root@link-08 ~]# rpm -q lvm2-cluster lvm2-cluster-2.02.06-7.0.RHEL4
Reproduced this many times, although not everytime. The corruption is seen as soon as I attempt to stat the first file in the mounted snapshot.
No more reports, bug 175830 is already fixed and closed, thus I'm closing this too.
I just reproduced this while running the standard regression tests for the 4.9 errata. Test Output: SCENARIO - [fs_io_A] Create snapshots of origin with fs data, and then verify that data on snapshots Making origin volume Placing an ext filesystem on origin volume mke2fs 1.35 (28-Feb-2004) Mounting origin volume Writing files to /mnt/origin checkit starting with: CREATE Num files: 500 Random Seed: 17762 Verify XIOR Stream: /tmp/checkit_origin_1 Working dir: /mnt/origin Checking files on /mnt/origin checkit starting with: VERIFY Verify XIOR Stream: /tmp/checkit_origin_1 Working dir: /mnt/origin Making 1st snapshot of origin volume Mounting 1st snap volume Checking files on /mnt/fs_snap1 checkit starting with: VERIFY Verify XIOR Stream: /tmp/checkit_origin_1 Working dir: /mnt/fs_snap1 Can not stat mlcecjdkldqxxocapfro: Stale NFS file handle checkit verify failed [root@grant-01 ~]# lvs -a -o +devices LV VG Attr LSize Origin Snap% Devices fs_snap1 snapper swi-ao 3.50G origin 0.00 /dev/sdc8(0) origin snapper owi-ao 4.00G /dev/sdc7(0) [root@grant-01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/snapper-origin 4.0G 160M 3.6G 5% /mnt/origin /dev/mapper/snapper-fs_snap1 4.0G 160M 3.6G 5% /mnt/fs_snap1 [root@grant-01 ~]# ls -l /mnt/fs_snap1 total 16 ?--------- ? ? ? ? ? aafmp ?--------- ? ? ? ? ? aakclf ?--------- ? ? ? ? ? abirsgeqstl ?--------- ? ? ? ? ? acfjiktdjar ?--------- ? ? ? ? ? achhoynttfrclmgrhteoccnekdiumlco ?--------- ? ? ? ? ? acymmtktjtmuimcnkrefoigyvxdplilnkmaxgmrqjhnrt ?--------- ? ? ? ? ? affbbgahdpkpvyvcxwusppjorkjpqraryhs [...]
This is easy to hit now. Is this a possible regression or was this bug just not that important in prior rhel4.* releases? SCENARIO - [fs_io_A] Create snapshots of origin with fs data, and then verify that data on snapshots Making origin volume Placing an ext filesystem on origin volume mke2fs 1.35 (28-Feb-2004) Mounting origin volume Writing files to /mnt/origin grant-01: /usr/tests/sts-rhel4.9/bin/checkit -w /mnt/origin -f /tmp/checkit_origin_1 -n 500 Checking files on /mnt/origin grant-01: /usr/tests/sts-rhel4.9/bin/checkit -w /mnt/origin -f /tmp/checkit_origin_1 -v Making 1st snapshot of origin volume grant-01: lvcreate -s /dev/snapper/origin -c 32 -n fs_snap1 -L 3.5G [Corruption] Version: 2.6.9-94.ELsmp lvm2-2.02.42-9.el4 BUILT: Thu Oct 21 15:49:57 CDT 2010 lvm2-cluster-2.02.42-9.el4 BUILT: Thu Oct 21 15:46:55 CDT 2010 device-mapper-1.02.28-3.el4 BUILT: Thu Mar 4 14:48:16 CST 2010 cmirror-1.0.1-1 BUILT: Tue Jan 30 17:28:02 CST 2007 cmirror-kernel-2.6.9-43.14.el4 BUILT: Wed Dec 22 16:24:19 CST 2010
If there is going to be a 4.9.z or a 4.10, I'll propose this bug for that release...
What is the physical device and driver you are using? Could you please recheck with different drivers to rule out driver bugs? Is the data corruption permanent or temporary? If you unmount the snapshot and mount it again, is it still corrupted the same way? If you deactivate the volume group with vgchange -an and reactivate it with -ay, is it still corrupted in the same way?
Is it ext2 or ext3 filesystem?
I don't think this is a device/driver issue, since this same test case works in rhel5 and rhel6 using the same device/driver. This is an ext2 file system.
This corruption appears permanent. Unmounting and remounting, as well as deactivating and reactivating, has no affect on the state of the snapshot.
I tried this script, but couldn't reproduce the data corruption. #!/bin/sh umount /mnt/test umount /mnt/test2 lvremove -f vg1/m1_snap1 lvremove -f vg1/m1 lvcreate -L 1G -n m1 vg1 || exit mkfs.ext2 /dev/vg1/m1 || exit mount -t ext2 /dev/vg1/m1 /mnt/test || exit ./checkit -w /mnt/test -f /tmp/checkit_origin_1 -n 500 || exit ./checkit -w /mnt/test -f /tmp/checkit_origin_1 -v || exit lvcreate -s -L 500M -n m1_snap1 vg1/m1 || exit mount -t ext2 /dev/vg1/m1_snap1 /mnt/test2 || exit ./checkit -w /mnt/test2 -f /tmp/checkit_origin_1 -v || exit Please send an exact script that triggers it.
RHEL4 is no longer actively developed and also this bug is missing needinfo for months, closing.