Bug 211235 - apparent file system corruption of snapshot fs
Summary: apparent file system corruption of snapshot fs
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: lvm2
Version: 4.4
Hardware: All
OS: Linux
urgent
high
Target Milestone: ---
: ---
Assignee: Mikuláš Patočka
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 461289
TreeView+ depends on / blocked
 
Reported: 2006-10-17 23:31 UTC by Corey Marthaler
Modified: 2012-03-22 12:09 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-03-22 12:09:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Corey Marthaler 2006-10-17 23:31:29 UTC
Description of problem:
I created an origin volume, put an ext filesystem on it, used 'check-it' to
write random files to that fs and then verify them, created a snapshot of that
volume, and then attempted to verify the snapshots data with 'check-it' as well.
That verify failed.

I'll try to reproduce this and gather more information.

SCENARIO - [fs_io_to_snaps]
Create snapshots of origin with fs data and verify that data on snapshot
Making origin volume
Placing an ext filesystem on origin volume
mke2fs 1.35 (28-Feb-2004)
Mounting origin volume

Writing files to /mnt/origin
checkit starting with:
CREATE
Num files:          500
Random Seed:        4321
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin

Checking files on /mnt/origin
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin
Making first snapshot of origin volume
Mounting 1st snap volume
Checking files on /mnt/fs_snap1
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/fs_snap1
Can not stat ycvbn: Input/output error
checkit verifiy failed

[root@link-08 ~]# ls -l /mnt/fs_snap1/
total 16
?---------  ? ?    ?        ?            ? asevvlghcnttmnrqrndk
?---------  ? ?    ?        ?            ? asqgwgpnbycmmbicqwotwfiyngsvbheb
?---------  ? ?    ?        ?            ? ax
?---------  ? ?    ?        ?            ?
bdojjdapcvsntyniqsenxhmwgnactfawkpgtujlxhhnbgbmyv
?---------  ? ?    ?        ?            ? bfislrhxncclnjodc
?---------  ? ?    ?        ?            ? bgjifdisttoxiefdbylqfsotmneyugjxnt
?---------  ? ?    ?        ?            ? bjcxstgmgmivtr
?---------  ? ?    ?        ?            ? bqlceahjrumfearrnppswtfmwlxpb
?---------  ? ?    ?        ?            ?
cpmigrpmdieafsjmrvbbjqovfinkdlghdvqjngxsrduwxfl
?---------  ? ?    ?        ?            ? cwileceuhniovnakcjyean
?---------  ? ?    ?        ?            ?
djmpvgirohiduwcitaoaukqindbiitblepdcvltmscpp
?---------  ? ?    ?        ?            ?
djpjqjisemqmdekowjmyiypimyvflbbqmscddkxhwovbsip
?---------  ? ?    ?        ?            ? dwrc
?---------  ? ?    ?        ?            ? egeummgwxnjsjcrjnykmitsjykcu
?---------  ? ?    ?        ?            ? emmkseyrpjieplqtrnqaqttwmxbcur
?---------  ? ?    ?        ?            ? eqbbcdmygnmiwoidgsrjocqpewnbrcqyvraxxoy
?---------  ? ?    ?        ?            ? eubrlxolbtrrfuobgwljeegmewrcn
?---------  ? ?    ?        ?            ?
ffvipfvbeeffhyvyguymmtkyyxpiywdeecpthluopdty
?---------  ? ?    ?        ?            ? fijbdkl
?---------  ? ?    ?        ?            ? fjepvnbdofbmkcxcmp
?---------  ? ?    ?        ?            ? fkcenwrwjfqvnbigbkcwfebxmoguaiffuh
?---------  ? ?    ?        ?            ? fpokbdgaptgtqbilgcsbnfihnomsjxgqnvbry
?---------  ? ?    ?        ?            ? fuakpcwnugkepqmtsegssniebjffjyqowqbo
?---------  ? ?    ?        ?            ? fysavfbglfsyjxllcnrtdrmekkqccrqhtkkprl
?---------  ? ?    ?        ?            ? gaojhsvbowrdfsa
?---------  ? ?    ?        ?            ? gecnoaqwuprarlwavketdghiled
?---------  ? ?    ?        ?            ? gndkjefqprkhpbgp
?---------  ? ?    ?        ?            ? gvcrsrdnrtikqmgmhqugpjkqdusakolqknk
?---------  ? ?    ?        ?            ? gwbki
?---------  ? ?    ?        ?            ?
gwlceuptmudnublqclwgxskhnwegcngkmtmtodpdassxugqys
?---------  ? ?    ?        ?            ? hi
?---------  ? ?    ?        ?            ? htkaxsrvdauvufomjrsqa
?---------  ? ?    ?        ?            ? htnfxfiqgdyokusjkjavhfmhufruhrlqnbvlggcnl
?---------  ? ?    ?        ?            ? hulfwxxckgiwdjipjssntwabahfl
?---------  ? ?    ?        ?            ? ikiyajvfgnkatbkonc
?---------  ? ?    ?        ?            ?
injvprblmnaircedmgwbpfeeaowdsmadcjytdbipqixime
?---------  ? ?    ?        ?            ? iorvyv
?---------  ? ?    ?        ?            ?
ivbslobgvspayjskteajandaetbtponbkovyfyfbrv
?---------  ? ?    ?        ?            ? j
?---------  ? ?    ?        ?            ?
jipqbhflkgnejvttnlxqqnsrnuwbsmtcvkuytbkgjakt
?---------  ? ?    ?        ?            ? jmrlduqinlfewvqaikvvgxkbsqxgypoj
?---------  ? ?    ?        ?            ? khkbcrbxvguwpctubfeukvxuoajsswcfe
?---------  ? ?    ?        ?            ? klyfeknnfnj
?---------  ? ?    ?        ?            ? kxglxjrfvedps
?---------  ? ?    ?        ?            ? lbffggaawkcvjbplgimjgno
?---------  ? ?    ?        ?            ? lefmugidccthwxgoamjyfegfh
drwx------  2 root root 16384 Oct 17 13:23 lost+found
?---------  ? ?    ?        ?            ? lupcrfuftujacskjjvrcfsfojysod
?---------  ? ?    ?        ?            ? lyhhglxgsctdjsylnmmbsababxpwmo
?---------  ? ?    ?        ?            ? majiugwhce
?---------  ? ?    ?        ?            ? mdfxmgjgiviurpwujnjuwdfrhs
?---------  ? ?    ?        ?            ? mlrhqsgvuqkcf
?---------  ? ?    ?        ?            ? mxehliiablptgviwp
?---------  ? ?    ?        ?            ? nabynccxfnktdumsjymlvhuomawdmaja
?---------  ? ?    ?        ?            ?
nlahgphipvqfxrfwnxbapddmqmlrcotpcvaimhsefkjfcqcrpe
?---------  ? ?    ?        ?            ? ntvhqaoxpgkupjowxoiegifitixqtrdjlyqe
?---------  ? ?    ?        ?            ? nxhkiwmrvxxbaealrbqpxntnpnafdrorqye
?---------  ? ?    ?        ?            ? ojjkn
?---------  ? ?    ?        ?            ? oqefbntqunnnxkfnhevitbkcjbhclgrcavhdmdth
?---------  ? ?    ?        ?            ?
oysfyhtnsojqkmlnbhphrhmhxdfaajxplrwkbrbuiknsyaiah
?---------  ? ?    ?        ?            ?
paidaouhvrxooobcmqidqfggwnvvasppsbuuppelkcdbs
?---------  ? ?    ?        ?            ? persaodgfivxgksywxfka
?---------  ? ?    ?        ?            ?
phfoastdvfbgjxijergwbnvposchqgdiqkwqdswbyajix
?---------  ? ?    ?        ?            ?
piysthagxsrdgqwrfqryvjmggfxofhmurlnnvqttjoyshxkmp
?---------  ? ?    ?        ?            ? puffhmubwrjgdawglpkwnfcu
?---------  ? ?    ?        ?            ? qmjeuxtmyweikxse
?---------  ? ?    ?        ?            ? qwnpaynvejrvgyyxsohkycuoulacwkh
?---------  ? ?    ?        ?            ? rgfvflmhskusduiouvlqgpecjsbfvmtqsymxmyh
?---------  ? ?    ?        ?            ?
rgkuxklvpsdgonftrkihmvmesxwhwcyqilmjwahosku
?---------  ? ?    ?        ?            ? rhqgfnuohumwxi
?---------  ? ?    ?        ?            ? rjjcfaqjgrupfjmgemkwqnwxollactqwedylfswl
?---------  ? ?    ?        ?            ? rjprabbogumxmowuaifpeohx
?---------  ? ?    ?        ?            ? rsrxtgabywhbtrlvxdjtunwlsqpwpk
?---------  ? ?    ?        ?            ? segfs
?---------  ? ?    ?        ?            ? sidnsyqhpkxutcdkk
?---------  ? ?    ?        ?            ? sifxflsahd
?---------  ? ?    ?        ?            ? swkowtcyudmnbufmymt
?---------  ? ?    ?        ?            ? tcj
?---------  ? ?    ?        ?            ? tcqmaatakfsqlhdnx
?---------  ? ?    ?        ?            ? tdocrdno
?---------  ? ?    ?        ?            ? tgjagyyujyxdhsbptmmoyfsltvecdwqwdawmcx
?---------  ? ?    ?        ?            ? tqrxxisssctejgjrvygsiwodcala
?---------  ? ?    ?        ?            ? ttqefpadgacsjplyavyvcvyqmbxjdeswald
?---------  ? ?    ?        ?            ?
txerofnemqugjmqekmiouqemayvvgrqarurgchlqaiwmxp
?---------  ? ?    ?        ?            ? ubdwjbtgqqkhibfqtpvcntbbcnrsouhlxmjjnf
?---------  ? ?    ?        ?            ? ugmjeioqgcwqitkqktbueyeqorywwr
?---------  ? ?    ?        ?            ? uikispa
?---------  ? ?    ?        ?            ?
uknmjbylujgkncuayvaexlaeodtvkuuhhktqltegdmqtpntq
?---------  ? ?    ?        ?            ? uooubjqgwdqelldaluotwnfjlfc
?---------  ? ?    ?        ?            ? utcyymeghhuxv
?---------  ? ?    ?        ?            ? vcdknbwocittfenjhnitnavxberpee
?---------  ? ?    ?        ?            ? vlqxuafkqdteihunkeuxvivyqfnvdmjyxaysa
?---------  ? ?    ?        ?            ?
vpjugthynbbsynxlflliibsjmbkltkgpcpkjktkavlsubs
?---------  ? ?    ?        ?            ? vupqbjqfbdwux
?---------  ? ?    ?        ?            ?
vuxbkpybrsyesdevbowsmleyioujkrhgmhkyxjapcctwfy
?---------  ? ?    ?        ?            ? wysttktjaibiuarshsmhehskwmprtpdqqvkn
?---------  ? ?    ?        ?            ? xhaxehwjye
?---------  ? ?    ?        ?            ?
xhidlpcnsucketafrmlmpfdsavuwawxyeiesaiitdmgkgjpav
?---------  ? ?    ?        ?            ? xkvecqkticwlcticqarmrnfmawwxeihevd
?---------  ? ?    ?        ?            ? xpbroguntoqttupkrbbemydikaymsit
?---------  ? ?    ?        ?            ? xtyhdlmkuqw
?---------  ? ?    ?        ?            ? xuktqllvkaqxtylojboab
?---------  ? ?    ?        ?            ? xxmslarihkntrjumangrvafaxqidtu
?---------  ? ?    ?        ?            ? xyrtfdchgewtmymn
?---------  ? ?    ?        ?            ? ycvbn
?---------  ? ?    ?        ?            ? yddoabxiwuq
?---------  ? ?    ?        ?            ? yfhbovvqqn
?---------  ? ?    ?        ?            ? yic
?---------  ? ?    ?        ?            ? yjcfqqaxjisksnrakdjy
?---------  ? ?    ?        ?            ? yjk
?---------  ? ?    ?        ?            ? yosecmxlaxq
?---------  ? ?    ?        ?            ? ypbeffunyhylkpghbwncfxjnpcdyjvhilk
?---------  ? ?    ?        ?            ? yrbtywmjweqgbuthcaxyajg
?---------  ? ?    ?        ?            ? yxnuyragykigkwyp


[root@link-08 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       72G  2.2G   66G   4% /
/dev/hda1              99M   45M   49M  48% /boot
none                  500M     0  500M   0% /dev/shm
/dev/mapper/snapper-origin
                      4.0G  156M  3.6G   5% /mnt/origin
/dev/mapper/snapper-fs_snap1
                      4.0G  8.0M  3.8G   1% /mnt/fs_snap1

[root@link-08 ~]# lvs -a -o+devices
  LV       VG      Attr   LSize Origin Snap%  Move Log Copy%  Devices
  fs_snap1 snapper swi-ao 2.50G origin   3.94                 /dev/sdc1(0)
  origin   snapper owi-ao 4.00G                               /dev/sdb1(0)

[root@link-08 ~]# dmsetup ls
snapper-origin-real     (253, 3)
VolGroup00-LogVol01     (253, 1)
snapper-fs_snap1-cow    (253, 4)
VolGroup00-LogVol00     (253, 0)
snapper-fs_snap1        (253, 5)
snapper-origin  (253, 2)


Version-Release number of selected component (if applicable):
[root@link-08 ~]# rpm -q lvm2
lvm2-2.02.06-6.0.RHEL4
[root@link-08 ~]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-7.0.RHEL4

Comment 1 Corey Marthaler 2006-10-18 16:01:43 UTC
Reproduced this many times, although not everytime. The corruption is seen as
soon as I attempt to stat the first file in the mounted snapshot.

Comment 5 Mikuláš Patočka 2010-04-28 14:51:22 UTC
No more reports, bug 175830 is already fixed and closed, thus I'm closing this too.

Comment 6 Corey Marthaler 2011-01-12 17:12:16 UTC
I just reproduced this while running the standard regression tests for the 4.9 errata.

Test Output:

SCENARIO - [fs_io_A]
Create snapshots of origin with fs data, and then verify that data on snapshots
Making origin volume
Placing an ext filesystem on origin volume
mke2fs 1.35 (28-Feb-2004)
Mounting origin volume

Writing files to /mnt/origin
checkit starting with:
CREATE
Num files:          500
Random Seed:        17762
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin

Checking files on /mnt/origin
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/origin
Making 1st snapshot of origin volume
Mounting 1st snap volume
Checking files on /mnt/fs_snap1
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_origin_1
Working dir:        /mnt/fs_snap1
Can not stat mlcecjdkldqxxocapfro: Stale NFS file handle
checkit verify failed

[root@grant-01 ~]# lvs -a -o +devices
  LV       VG         Attr   LSize  Origin Snap%  Devices
  fs_snap1 snapper    swi-ao  3.50G origin   0.00 /dev/sdc8(0)
  origin   snapper    owi-ao  4.00G               /dev/sdc7(0)

[root@grant-01 ~]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/snapper-origin    4.0G  160M  3.6G   5% /mnt/origin
/dev/mapper/snapper-fs_snap1  4.0G  160M  3.6G   5% /mnt/fs_snap1

[root@grant-01 ~]# ls -l /mnt/fs_snap1
total 16
?---------  ? ?    ?        ?            ? aafmp
?---------  ? ?    ?        ?            ? aakclf
?---------  ? ?    ?        ?            ? abirsgeqstl
?---------  ? ?    ?        ?            ? acfjiktdjar
?---------  ? ?    ?        ?            ? achhoynttfrclmgrhteoccnekdiumlco
?---------  ? ?    ?        ?            ? acymmtktjtmuimcnkrefoigyvxdplilnkmaxgmrqjhnrt
?---------  ? ?    ?        ?            ? affbbgahdpkpvyvcxwusppjorkjpqraryhs
[...]

Comment 7 Corey Marthaler 2011-01-12 19:46:22 UTC
This is easy to hit now. Is this a possible regression or was this bug just not that important in prior rhel4.* releases?

SCENARIO - [fs_io_A]
Create snapshots of origin with fs data, and then verify that data on snapshots
Making origin volume
Placing an ext filesystem on origin volume
mke2fs 1.35 (28-Feb-2004)
Mounting origin volume

Writing files to /mnt/origin
grant-01: /usr/tests/sts-rhel4.9/bin/checkit -w /mnt/origin -f /tmp/checkit_origin_1 -n 500

Checking files on /mnt/origin
grant-01: /usr/tests/sts-rhel4.9/bin/checkit -w /mnt/origin -f /tmp/checkit_origin_1 -v

Making 1st snapshot of origin volume
grant-01: lvcreate -s /dev/snapper/origin -c 32 -n fs_snap1 -L 3.5G

[Corruption]


Version:
2.6.9-94.ELsmp

lvm2-2.02.42-9.el4    BUILT: Thu Oct 21 15:49:57 CDT 2010
lvm2-cluster-2.02.42-9.el4    BUILT: Thu Oct 21 15:46:55 CDT 2010
device-mapper-1.02.28-3.el4    BUILT: Thu Mar  4 14:48:16 CST 2010
cmirror-1.0.1-1    BUILT: Tue Jan 30 17:28:02 CST 2007
cmirror-kernel-2.6.9-43.14.el4    BUILT: Wed Dec 22 16:24:19 CST 2010

Comment 8 Corey Marthaler 2011-01-13 21:39:38 UTC
If there is going to be a 4.9.z or a 4.10, I'll propose this bug for that release...

Comment 9 Mikuláš Patočka 2011-01-14 18:10:21 UTC
What is the physical device and driver you are using? Could you please recheck with different drivers to rule out driver bugs?

Is the data corruption permanent or temporary? If you unmount the snapshot and mount it again, is it still corrupted the same way? If you deactivate the volume group with vgchange -an and reactivate it with -ay, is it still corrupted in the same way?

Comment 10 Mikuláš Patočka 2011-01-14 18:13:41 UTC
Is it ext2 or ext3 filesystem?

Comment 11 Corey Marthaler 2011-01-14 20:29:26 UTC
I don't think this is a device/driver issue, since this same test case works in rhel5 and rhel6 using the same device/driver.

This is an ext2 file system.

Comment 12 Corey Marthaler 2011-01-14 20:38:04 UTC
This corruption appears permanent. Unmounting and remounting, as well as deactivating and reactivating, has no affect on the state of the snapshot.

Comment 13 Mikuláš Patočka 2011-05-19 17:04:07 UTC
I tried this script, but couldn't reproduce the data corruption.

#!/bin/sh
umount /mnt/test
umount /mnt/test2
lvremove -f vg1/m1_snap1
lvremove -f vg1/m1
lvcreate -L 1G -n m1 vg1 || exit
mkfs.ext2 /dev/vg1/m1 || exit
mount -t ext2 /dev/vg1/m1 /mnt/test || exit
./checkit -w /mnt/test -f /tmp/checkit_origin_1 -n 500 || exit
./checkit -w /mnt/test -f /tmp/checkit_origin_1 -v || exit
lvcreate -s -L 500M -n m1_snap1 vg1/m1 || exit
mount -t ext2 /dev/vg1/m1_snap1 /mnt/test2 || exit
./checkit -w /mnt/test2 -f /tmp/checkit_origin_1 -v || exit

Please send an exact script that triggers it.

Comment 14 Milan Broz 2012-03-22 12:09:48 UTC
RHEL4 is no longer actively developed and also this bug is missing needinfo for months, closing.


Note You need to log in before you can comment on or make changes to this bug.