Bug 211235 - apparent file system corruption of snapshot fs
apparent file system corruption of snapshot fs
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: lvm2 (Show other bugs)
4.4
All Linux
urgent Severity high
: ---
: ---
Assigned To: Mikulas Patocka
: Reopened
Depends On:
Blocks: 461289
  Show dependency treegraph
 
Reported: 2006-10-17 19:31 EDT by Corey Marthaler
Modified: 2012-03-22 08:09 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-22 08:09:48 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2006-10-17 19:31:29 EDT
Description of problem:
I created an origin volume, put an ext filesystem on it, used 'check-it' to
write random files to that fs and then verify them, created a snapshot of that
volume, and then attempted to verify the snapshots data with 'check-it' as well.
That verify failed.

I'll try to reproduce this and gather more information.

SCENARIO - [fs_io_to_snaps]
Create snapshots of origin with fs data and verify that data on snapshot
Making origin volume
Placing an ext filesystem on origin volume
mke2fs 1.35 (28-Feb-2004)
Mounting origin volume

Writing files to /mnt/origin
checkit starting with:
CREATE
Num files:          500
Random Seed:        4321
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin

Checking files on /mnt/origin
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin
Making first snapshot of origin volume
Mounting 1st snap volume
Checking files on /mnt/fs_snap1
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/fs_snap1
Can not stat ycvbn: Input/output error
checkit verifiy failed

[root@link-08 ~]# ls -l /mnt/fs_snap1/
total 16
?---------  ? ?    ?        ?            ? asevvlghcnttmnrqrndk
?---------  ? ?    ?        ?            ? asqgwgpnbycmmbicqwotwfiyngsvbheb
?---------  ? ?    ?        ?            ? ax
?---------  ? ?    ?        ?            ?
bdojjdapcvsntyniqsenxhmwgnactfawkpgtujlxhhnbgbmyv
?---------  ? ?    ?        ?            ? bfislrhxncclnjodc
?---------  ? ?    ?        ?            ? bgjifdisttoxiefdbylqfsotmneyugjxnt
?---------  ? ?    ?        ?            ? bjcxstgmgmivtr
?---------  ? ?    ?        ?            ? bqlceahjrumfearrnppswtfmwlxpb
?---------  ? ?    ?        ?            ?
cpmigrpmdieafsjmrvbbjqovfinkdlghdvqjngxsrduwxfl
?---------  ? ?    ?        ?            ? cwileceuhniovnakcjyean
?---------  ? ?    ?        ?            ?
djmpvgirohiduwcitaoaukqindbiitblepdcvltmscpp
?---------  ? ?    ?        ?            ?
djpjqjisemqmdekowjmyiypimyvflbbqmscddkxhwovbsip
?---------  ? ?    ?        ?            ? dwrc
?---------  ? ?    ?        ?            ? egeummgwxnjsjcrjnykmitsjykcu
?---------  ? ?    ?        ?            ? emmkseyrpjieplqtrnqaqttwmxbcur
?---------  ? ?    ?        ?            ? eqbbcdmygnmiwoidgsrjocqpewnbrcqyvraxxoy
?---------  ? ?    ?        ?            ? eubrlxolbtrrfuobgwljeegmewrcn
?---------  ? ?    ?        ?            ?
ffvipfvbeeffhyvyguymmtkyyxpiywdeecpthluopdty
?---------  ? ?    ?        ?            ? fijbdkl
?---------  ? ?    ?        ?            ? fjepvnbdofbmkcxcmp
?---------  ? ?    ?        ?            ? fkcenwrwjfqvnbigbkcwfebxmoguaiffuh
?---------  ? ?    ?        ?            ? fpokbdgaptgtqbilgcsbnfihnomsjxgqnvbry
?---------  ? ?    ?        ?            ? fuakpcwnugkepqmtsegssniebjffjyqowqbo
?---------  ? ?    ?        ?            ? fysavfbglfsyjxllcnrtdrmekkqccrqhtkkprl
?---------  ? ?    ?        ?            ? gaojhsvbowrdfsa
?---------  ? ?    ?        ?            ? gecnoaqwuprarlwavketdghiled
?---------  ? ?    ?        ?            ? gndkjefqprkhpbgp
?---------  ? ?    ?        ?            ? gvcrsrdnrtikqmgmhqugpjkqdusakolqknk
?---------  ? ?    ?        ?            ? gwbki
?---------  ? ?    ?        ?            ?
gwlceuptmudnublqclwgxskhnwegcngkmtmtodpdassxugqys
?---------  ? ?    ?        ?            ? hi
?---------  ? ?    ?        ?            ? htkaxsrvdauvufomjrsqa
?---------  ? ?    ?        ?            ? htnfxfiqgdyokusjkjavhfmhufruhrlqnbvlggcnl
?---------  ? ?    ?        ?            ? hulfwxxckgiwdjipjssntwabahfl
?---------  ? ?    ?        ?            ? ikiyajvfgnkatbkonc
?---------  ? ?    ?        ?            ?
injvprblmnaircedmgwbpfeeaowdsmadcjytdbipqixime
?---------  ? ?    ?        ?            ? iorvyv
?---------  ? ?    ?        ?            ?
ivbslobgvspayjskteajandaetbtponbkovyfyfbrv
?---------  ? ?    ?        ?            ? j
?---------  ? ?    ?        ?            ?
jipqbhflkgnejvttnlxqqnsrnuwbsmtcvkuytbkgjakt
?---------  ? ?    ?        ?            ? jmrlduqinlfewvqaikvvgxkbsqxgypoj
?---------  ? ?    ?        ?            ? khkbcrbxvguwpctubfeukvxuoajsswcfe
?---------  ? ?    ?        ?            ? klyfeknnfnj
?---------  ? ?    ?        ?            ? kxglxjrfvedps
?---------  ? ?    ?        ?            ? lbffggaawkcvjbplgimjgno
?---------  ? ?    ?        ?            ? lefmugidccthwxgoamjyfegfh
drwx------  2 root root 16384 Oct 17 13:23 lost+found
?---------  ? ?    ?        ?            ? lupcrfuftujacskjjvrcfsfojysod
?---------  ? ?    ?        ?            ? lyhhglxgsctdjsylnmmbsababxpwmo
?---------  ? ?    ?        ?            ? majiugwhce
?---------  ? ?    ?        ?            ? mdfxmgjgiviurpwujnjuwdfrhs
?---------  ? ?    ?        ?            ? mlrhqsgvuqkcf
?---------  ? ?    ?        ?            ? mxehliiablptgviwp
?---------  ? ?    ?        ?            ? nabynccxfnktdumsjymlvhuomawdmaja
?---------  ? ?    ?        ?            ?
nlahgphipvqfxrfwnxbapddmqmlrcotpcvaimhsefkjfcqcrpe
?---------  ? ?    ?        ?            ? ntvhqaoxpgkupjowxoiegifitixqtrdjlyqe
?---------  ? ?    ?        ?            ? nxhkiwmrvxxbaealrbqpxntnpnafdrorqye
?---------  ? ?    ?        ?            ? ojjkn
?---------  ? ?    ?        ?            ? oqefbntqunnnxkfnhevitbkcjbhclgrcavhdmdth
?---------  ? ?    ?        ?            ?
oysfyhtnsojqkmlnbhphrhmhxdfaajxplrwkbrbuiknsyaiah
?---------  ? ?    ?        ?            ?
paidaouhvrxooobcmqidqfggwnvvasppsbuuppelkcdbs
?---------  ? ?    ?        ?            ? persaodgfivxgksywxfka
?---------  ? ?    ?        ?            ?
phfoastdvfbgjxijergwbnvposchqgdiqkwqdswbyajix
?---------  ? ?    ?        ?            ?
piysthagxsrdgqwrfqryvjmggfxofhmurlnnvqttjoyshxkmp
?---------  ? ?    ?        ?            ? puffhmubwrjgdawglpkwnfcu
?---------  ? ?    ?        ?            ? qmjeuxtmyweikxse
?---------  ? ?    ?        ?            ? qwnpaynvejrvgyyxsohkycuoulacwkh
?---------  ? ?    ?        ?            ? rgfvflmhskusduiouvlqgpecjsbfvmtqsymxmyh
?---------  ? ?    ?        ?            ?
rgkuxklvpsdgonftrkihmvmesxwhwcyqilmjwahosku
?---------  ? ?    ?        ?            ? rhqgfnuohumwxi
?---------  ? ?    ?        ?            ? rjjcfaqjgrupfjmgemkwqnwxollactqwedylfswl
?---------  ? ?    ?        ?            ? rjprabbogumxmowuaifpeohx
?---------  ? ?    ?        ?            ? rsrxtgabywhbtrlvxdjtunwlsqpwpk
?---------  ? ?    ?        ?            ? segfs
?---------  ? ?    ?        ?            ? sidnsyqhpkxutcdkk
?---------  ? ?    ?        ?            ? sifxflsahd
?---------  ? ?    ?        ?            ? swkowtcyudmnbufmymt
?---------  ? ?    ?        ?            ? tcj
?---------  ? ?    ?        ?            ? tcqmaatakfsqlhdnx
?---------  ? ?    ?        ?            ? tdocrdno
?---------  ? ?    ?        ?            ? tgjagyyujyxdhsbptmmoyfsltvecdwqwdawmcx
?---------  ? ?    ?        ?            ? tqrxxisssctejgjrvygsiwodcala
?---------  ? ?    ?        ?            ? ttqefpadgacsjplyavyvcvyqmbxjdeswald
?---------  ? ?    ?        ?            ?
txerofnemqugjmqekmiouqemayvvgrqarurgchlqaiwmxp
?---------  ? ?    ?        ?            ? ubdwjbtgqqkhibfqtpvcntbbcnrsouhlxmjjnf
?---------  ? ?    ?        ?            ? ugmjeioqgcwqitkqktbueyeqorywwr
?---------  ? ?    ?        ?            ? uikispa
?---------  ? ?    ?        ?            ?
uknmjbylujgkncuayvaexlaeodtvkuuhhktqltegdmqtpntq
?---------  ? ?    ?        ?            ? uooubjqgwdqelldaluotwnfjlfc
?---------  ? ?    ?        ?            ? utcyymeghhuxv
?---------  ? ?    ?        ?            ? vcdknbwocittfenjhnitnavxberpee
?---------  ? ?    ?        ?            ? vlqxuafkqdteihunkeuxvivyqfnvdmjyxaysa
?---------  ? ?    ?        ?            ?
vpjugthynbbsynxlflliibsjmbkltkgpcpkjktkavlsubs
?---------  ? ?    ?        ?            ? vupqbjqfbdwux
?---------  ? ?    ?        ?            ?
vuxbkpybrsyesdevbowsmleyioujkrhgmhkyxjapcctwfy
?---------  ? ?    ?        ?            ? wysttktjaibiuarshsmhehskwmprtpdqqvkn
?---------  ? ?    ?        ?            ? xhaxehwjye
?---------  ? ?    ?        ?            ?
xhidlpcnsucketafrmlmpfdsavuwawxyeiesaiitdmgkgjpav
?---------  ? ?    ?        ?            ? xkvecqkticwlcticqarmrnfmawwxeihevd
?---------  ? ?    ?        ?            ? xpbroguntoqttupkrbbemydikaymsit
?---------  ? ?    ?        ?            ? xtyhdlmkuqw
?---------  ? ?    ?        ?            ? xuktqllvkaqxtylojboab
?---------  ? ?    ?        ?            ? xxmslarihkntrjumangrvafaxqidtu
?---------  ? ?    ?        ?            ? xyrtfdchgewtmymn
?---------  ? ?    ?        ?            ? ycvbn
?---------  ? ?    ?        ?            ? yddoabxiwuq
?---------  ? ?    ?        ?            ? yfhbovvqqn
?---------  ? ?    ?        ?            ? yic
?---------  ? ?    ?        ?            ? yjcfqqaxjisksnrakdjy
?---------  ? ?    ?        ?            ? yjk
?---------  ? ?    ?        ?            ? yosecmxlaxq
?---------  ? ?    ?        ?            ? ypbeffunyhylkpghbwncfxjnpcdyjvhilk
?---------  ? ?    ?        ?            ? yrbtywmjweqgbuthcaxyajg
?---------  ? ?    ?        ?            ? yxnuyragykigkwyp


[root@link-08 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       72G  2.2G   66G   4% /
/dev/hda1              99M   45M   49M  48% /boot
none                  500M     0  500M   0% /dev/shm
/dev/mapper/snapper-origin
                      4.0G  156M  3.6G   5% /mnt/origin
/dev/mapper/snapper-fs_snap1
                      4.0G  8.0M  3.8G   1% /mnt/fs_snap1

[root@link-08 ~]# lvs -a -o+devices
  LV       VG      Attr   LSize Origin Snap%  Move Log Copy%  Devices
  fs_snap1 snapper swi-ao 2.50G origin   3.94                 /dev/sdc1(0)
  origin   snapper owi-ao 4.00G                               /dev/sdb1(0)

[root@link-08 ~]# dmsetup ls
snapper-origin-real     (253, 3)
VolGroup00-LogVol01     (253, 1)
snapper-fs_snap1-cow    (253, 4)
VolGroup00-LogVol00     (253, 0)
snapper-fs_snap1        (253, 5)
snapper-origin  (253, 2)


Version-Release number of selected component (if applicable):
[root@link-08 ~]# rpm -q lvm2
lvm2-2.02.06-6.0.RHEL4
[root@link-08 ~]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-7.0.RHEL4
Comment 1 Corey Marthaler 2006-10-18 12:01:43 EDT
Reproduced this many times, although not everytime. The corruption is seen as
soon as I attempt to stat the first file in the mounted snapshot.
Comment 5 Mikulas Patocka 2010-04-28 10:51:22 EDT
No more reports, bug 175830 is already fixed and closed, thus I'm closing this too.
Comment 6 Corey Marthaler 2011-01-12 12:12:16 EST
I just reproduced this while running the standard regression tests for the 4.9 errata.

Test Output:

SCENARIO - [fs_io_A]
Create snapshots of origin with fs data, and then verify that data on snapshots
Making origin volume
Placing an ext filesystem on origin volume
mke2fs 1.35 (28-Feb-2004)
Mounting origin volume

Writing files to /mnt/origin
checkit starting with:
CREATE
Num files:          500
Random Seed:        17762
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin

Checking files on /mnt/origin
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin
Making 1st snapshot of origin volume
Mounting 1st snap volume
Checking files on /mnt/fs_snap1
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/fs_snap1
Can not stat mlcecjdkldqxxocapfro: Stale NFS file handle
checkit verify failed

[root@grant-01 ~]# lvs -a -o +devices
  LV       VG         Attr   LSize  Origin Snap%  Devices
  fs_snap1 snapper    swi-ao  3.50G origin   0.00 /dev/sdc8(0)
  origin   snapper    owi-ao  4.00G               /dev/sdc7(0)

[root@grant-01 ~]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/snapper-origin    4.0G  160M  3.6G   5% /mnt/origin
/dev/mapper/snapper-fs_snap1  4.0G  160M  3.6G   5% /mnt/fs_snap1

[root@grant-01 ~]# ls -l /mnt/fs_snap1
total 16
?---------  ? ?    ?        ?            ? aafmp
?---------  ? ?    ?        ?            ? aakclf
?---------  ? ?    ?        ?            ? abirsgeqstl
?---------  ? ?    ?        ?            ? acfjiktdjar
?---------  ? ?    ?        ?            ? achhoynttfrclmgrhteoccnekdiumlco
?---------  ? ?    ?        ?            ? acymmtktjtmuimcnkrefoigyvxdplilnkmaxgmrqjhnrt
?---------  ? ?    ?        ?            ? affbbgahdpkpvyvcxwusppjorkjpqraryhs
[...]
Comment 7 Corey Marthaler 2011-01-12 14:46:22 EST
This is easy to hit now. Is this a possible regression or was this bug just not that important in prior rhel4.* releases?

SCENARIO - [fs_io_A]
Create snapshots of origin with fs data, and then verify that data on snapshots
Making origin volume
Placing an ext filesystem on origin volume
mke2fs 1.35 (28-Feb-2004)
Mounting origin volume

Writing files to /mnt/origin
grant-01: /usr/tests/sts-rhel4.9/bin/checkit -w /mnt/origin -f /tmp/checkit_origin_1 -n 500

Checking files on /mnt/origin
grant-01: /usr/tests/sts-rhel4.9/bin/checkit -w /mnt/origin -f /tmp/checkit_origin_1 -v

Making 1st snapshot of origin volume
grant-01: lvcreate -s /dev/snapper/origin -c 32 -n fs_snap1 -L 3.5G

[Corruption]


Version:
2.6.9-94.ELsmp

lvm2-2.02.42-9.el4    BUILT: Thu Oct 21 15:49:57 CDT 2010
lvm2-cluster-2.02.42-9.el4    BUILT: Thu Oct 21 15:46:55 CDT 2010
device-mapper-1.02.28-3.el4    BUILT: Thu Mar  4 14:48:16 CST 2010
cmirror-1.0.1-1    BUILT: Tue Jan 30 17:28:02 CST 2007
cmirror-kernel-2.6.9-43.14.el4    BUILT: Wed Dec 22 16:24:19 CST 2010
Comment 8 Corey Marthaler 2011-01-13 16:39:38 EST
If there is going to be a 4.9.z or a 4.10, I'll propose this bug for that release...
Comment 9 Mikulas Patocka 2011-01-14 13:10:21 EST
What is the physical device and driver you are using? Could you please recheck with different drivers to rule out driver bugs?

Is the data corruption permanent or temporary? If you unmount the snapshot and mount it again, is it still corrupted the same way? If you deactivate the volume group with vgchange -an and reactivate it with -ay, is it still corrupted in the same way?
Comment 10 Mikulas Patocka 2011-01-14 13:13:41 EST
Is it ext2 or ext3 filesystem?
Comment 11 Corey Marthaler 2011-01-14 15:29:26 EST
I don't think this is a device/driver issue, since this same test case works in rhel5 and rhel6 using the same device/driver.

This is an ext2 file system.
Comment 12 Corey Marthaler 2011-01-14 15:38:04 EST
This corruption appears permanent. Unmounting and remounting, as well as deactivating and reactivating, has no affect on the state of the snapshot.
Comment 13 Mikulas Patocka 2011-05-19 13:04:07 EDT
I tried this script, but couldn't reproduce the data corruption.

#!/bin/sh
umount /mnt/test
umount /mnt/test2
lvremove -f vg1/m1_snap1
lvremove -f vg1/m1
lvcreate -L 1G -n m1 vg1 || exit
mkfs.ext2 /dev/vg1/m1 || exit
mount -t ext2 /dev/vg1/m1 /mnt/test || exit
./checkit -w /mnt/test -f /tmp/checkit_origin_1 -n 500 || exit
./checkit -w /mnt/test -f /tmp/checkit_origin_1 -v || exit
lvcreate -s -L 500M -n m1_snap1 vg1/m1 || exit
mount -t ext2 /dev/vg1/m1_snap1 /mnt/test2 || exit
./checkit -w /mnt/test2 -f /tmp/checkit_origin_1 -v || exit

Please send an exact script that triggers it.
Comment 14 Milan Broz 2012-03-22 08:09:48 EDT
RHEL4 is no longer actively developed and also this bug is missing needinfo for months, closing.

Note You need to log in before you can comment on or make changes to this bug.