Bug 852833 - Online ext4 resizing may cause fsck to report errors
Online ext4 resizing may cause fsck to report errors
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: e2fsprogs (Show other bugs)
16
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Carlos Maiolino
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-29 13:49 EDT by Jaroslav Kortus
Modified: 2013-01-24 17:38 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-24 17:38:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
meta dump before resize (320.15 KB, application/x-bzip2)
2012-08-29 14:37 EDT, Jaroslav Kortus
no flags Details

  None (edit)
Description Jaroslav Kortus 2012-08-29 13:49:15 EDT
Description of problem:
Online resize of ext4 file system causes mkfs.ext4 to report failures even if the data is ok.


Version-Release number of selected component (if applicable):
Linux fedora16 3.4.9-1.fc16.x86_64 #1 SMP Wed Aug 15 20:45:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
e2fsprogs-1.41.14-2.fc15.x86_64
e2fsprogs-libs-1.41.14-2.fc15.x86_64

It seems that only this version of e2fsprogs is affected.
Best chance to recreate was when -l+100%FREE extension was used.

It is probably fixed already in FC17, so I'm asking for update on FC16 while it's still maintained.

I've run md5sums on the data stored on resized FS and all matched. So no data seems to be destroyed, but the filesystem is reported as corrupted.

How reproducible:
100%

Steps to Reproduce:
1. see below
2.
3.
  
Actual results:
fsck reporting many errors after resized ext4 FS is unmounted

Expected results:
no reported errors

Additional info:
Script used to verify the bug:

#!/bin/bash 
dev=/dev/vdb1
pvcreate $dev
vgcreate test $dev
# inital size volume
lvcreate -L15g -n test test
lvs
vgs
vgchange -ay test
mkdir ext4
mkfs.ext4 /dev/test/test 
mount /dev/test/test ext4
# copy some data to the fs
cp -a /usr/lib ext4
umount ext4
# clean output should be here, just to make sure
fsck -fn /dev/test/test 
mount /dev/test/test ext4
# extend underlying device
lvextend -l+100%FREE /dev/test/test 
# resize ext4
resize2fs /dev/test/test
umount /dev/test/test 
# this fails the check with many errors
fsck.ext4 -fn /dev/test/test 
vgremove -ff test



Error output example snip:
Inode 1057307, i_size is 16324418925048666496, should be 0.  Fix? no

Inode 1057307, i_blocks is 44091962484585, should be 0.  Fix? no

Inode 1057308 is in use, but has dtime set.  Fix? no

Inode 1057308 has imagic flag set.  Clear? no

Inode 1057308 has a extra size (29824) which is invalid
Fix? no

Inode 1057308 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1057308 has an invalid root node.
Clear HTree index? no
Comment 1 Eric Sandeen 2012-08-29 14:21:07 EDT
Would it be possible to create an "e2image -r" of the populated & unmounted device prior to the resize, bzip2 it, and attach it?  And then after the resize, do dumpe2fs -h so I can see how big it was grown?  Then I can probably reproduce it w/o all the lvm gyrations.

Thanks,
-Eric
Comment 2 Jaroslav Kortus 2012-08-29 14:37:10 EDT
Created attachment 607978 [details]
meta dump before resize
Comment 3 Jaroslav Kortus 2012-08-29 14:37:59 EDT
dumpe2fs -h after resize:
Filesystem volume name:   <none>
Last mounted on:          /root/ext4
Filesystem UUID:          c5578d94-1c84-4e2a-811d-f6e68f00c5e9
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1966080
Block count:              7863296
Reserved block count:     354095
Free blocks:              7640447
Free inodes:              1961524
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      958
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Wed Aug 29 20:32:49 2012
Last mount time:          Wed Aug 29 20:32:55 2012
Last write time:          Wed Aug 29 20:33:10 2012
Mount count:              2
Maximum mount count:      20
Last checked:             Wed Aug 29 20:32:49 2012
Check interval:           15552000 (6 months)
Next check after:         Mon Feb 25 19:32:49 2013
Lifetime writes:          417 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      6baea7b9-6518-4123-9bbf-f3ed97605677
Journal backup:           inode blocks
Journal features:         (none)
Journal size:             128M
Journal length:           32768
Journal sequence:         0x0000007d
Journal start:            0
Comment 4 Eric Sandeen 2012-08-29 14:42:15 EDT
Great, thanks.  I'll try to find some time to look into whether it is truly fixed upstream and what fixed it.

-Eric
Comment 5 Eric Sandeen 2012-08-29 17:33:54 EDT
Hm, I can't recreate it with the image :(

[root@inode e2image-resize]# e2image -r image.orig ext4.before-resize.img 
e2image 1.41.14 (22-Dec-2010)

[root@inode e2image-resize]# truncate --size 32208060416 ext4.before-resize.img 
nt

[root@inode e2image-resize]# mount -o loop ext4.before-resize.img mnt/

[root@inode e2image-resize]# resize2fs /dev/loop0 
resize2fs 1.41.14 (22-Dec-2010)
Filesystem at /dev/loop0 is mounted on /mnt/test2/e2image-resize/mnt; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 2
Performing an on-line resize of /dev/loop0 to 7863296 (4k) blocks.
The filesystem on /dev/loop0 is now 7863296 blocks long.

[root@inode e2image-resize]# umount mnt
[root@inode e2image-resize]# e2fsck -fy ext4.before-resize.img 
e2fsck 1.41.14 (22-Dec-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
ext4.before-resize.img: 4556/1966080 files (0.1% non-contiguous), 222849/7863296 blocks

[root@inode e2image-resize]# rpm -q e2fsprogs
e2fsprogs-1.41.14-2.fc15.x86_64
Comment 6 Jaroslav Kortus 2012-08-30 09:05:49 EDT
it does not seem to be reproducible via loop device. I'm hitting only on top of LVM. 

Can you please try to reproduce it using LVM?
Comment 7 leezer3 2012-09-28 20:47:22 EDT
Somone else who has seen this checking in, although I'm using a Debian install with a custom rolled version of the latest mainline kernel, so I suspect this is a kernel bug (Or e2fsprogs, will get you the version I'm on when my fsck finishes) upto and including mainline, not a Red Hat specific.

Did basically the same as the OP- Expanded my LVM ( /dev/mapper/media-Media ) volume from 1.5tb to 2tb, and FSCK threw a fit on reboot.
Comment 8 Jaroslav Kortus 2012-11-27 09:26:49 EST
Eric, did you have a chance to reproduce it on top of LVM?
Comment 9 Eric Sandeen 2012-11-28 17:07:16 EST
Not yet, no.  Sorry.  I'll try to get to it...

If you want to test it w/ your e2image and a file-backed lvm volume, it'd be nice to know if it reproduces that way.
Comment 10 Eric Sandeen 2012-11-28 18:26:44 EST
Tried it w/ a rhel6 kernel and e2fsprogs-1.41.12-12.el6.x86_64
 and no problems; will retry w/ your versions.

What kernel did you test on?  online resize is a fair bit of kernel code too.
Comment 11 Eric Sandeen 2012-11-29 00:16:51 EST
Tried this in an F16 VM, both as originally installed by the livecd and after all updates.  Both with your original reproducer, and by using your pre-resize image.

I hit no corruption in any of the 4 cases I tried above . . .
Comment 12 Jaroslav Kortus 2012-12-01 18:11:01 EST
Hi Eric, thanks for trying.

I was trying to recreate the problem with clean disk and it is quite hard indeed. By some magic I managed to get it corrupted again. Unfortunately it was quite a try-and-see approach with different kernels and different operations, so I still do not have any exact reproducer.

What I have is the disk image on which the script above triggers the error. Maybe it's dependent on actual disk content. On this image the operations fail as described on fully updated FC16 installation in libvirt (kernel 3.6.7-4 x86_64).

I'm testing it on dual CPU virt with 2G RAM and the image as a second virtio disk. I've uploaded the image to http://mestapo.cz/virt-images/fc16-resizetest-resizedisk.img.tar.xz. This is ~80M compressed tar image of 30G sparse file I'm using as the disk image.

Can you please retry with this image and let me know if you were successful?

Thank you,
J.
Comment 13 Roberto Ragusa 2012-12-15 10:27:43 EST
Trivially reproducible here, after happening in production a few times on F16.

lvcfreate, mkfs, e2fsck (ok), e2fsck -f (ok), mount, touch x, lvextend, resize2fs, umount, e2fsck (ok!!!), e2fsck -f (TONS OF ERRORS STARTING IN THE EXTENDED ZONE)

(legend: "ggg" volume group, "lll" logical volume, "ppp" mount point)

I can upload dumpe2fs outputs and the final e2image if needed.

[root@linuxserver FS-CORRUPTION-TESTS]# uname -a;rpm -q e2fsprogs
Linux linuxserver.m.g 3.4.6-1.fc16.x86_64 #1 SMP Fri Jul 20 12:58:04 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
e2fsprogs-1.41.14-2.fc15.x86_64

--------------------------------------------------------------------

[root@linuxserver FS-CORRUPTION-TESTS]# lvcreate -L100G -n lll ggg
File descriptor 7 (pipe:[25711]) leaked on lvcreate invocation. Parent PID 2977: bash
  Logical volume "lll" created
[root@linuxserver FS-CORRUPTION-TESTS]# mkfs -t ext4 -i 69905 -G 256 -E resize=4290772992 /dev/mapper/ggg-lll 
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1 blocks, Stripe width=1 blocks
1548800 inodes, 26214400 blocks
1310720 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4290772992
800 block groups
32768 blocks per group, 32768 fragments per group
1936 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@linuxserver FS-CORRUPTION-TESTS]# e2fsck -n /dev/mapper/ggg-lll 
e2fsck 1.41.14 (22-Dec-2010)
/dev/mapper/ggg-lll: clean, 11/1548800 files, 146534/26214400 blocks
[root@linuxserver FS-CORRUPTION-TESTS]# e2fsck -n -f /dev/mapper/ggg-lll 
e2fsck 1.41.14 (22-Dec-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/ggg-lll: 11/1548800 files (0.0% non-contiguous), 146534/26214400 blocks
[root@linuxserver FS-CORRUPTION-TESTS]# dumpe2fs /dev/mapper/ggg-lll >100formatted
dumpe2fs 1.41.14 (22-Dec-2010)
[root@linuxserver FS-CORRUPTION-TESTS]# mkdir ppp
[root@linuxserver FS-CORRUPTION-TESTS]# mount /dev/mapper/ggg-lll ppp -o noatime
[root@linuxserver FS-CORRUPTION-TESTS]# date >ppp/timestamp.txt
[root@linuxserver FS-CORRUPTION-TESTS]# ls -l ppp
total 20
drwx------ 2 root root 16384 Dec 15 15:17 lost+found
-rw-r--r-- 1 root root    29 Dec 15 15:18 timestamp.txt
[root@linuxserver FS-CORRUPTION-TESTS]# dumpe2fs /dev/mapper/ggg-lll >100formatted_mounted
dumpe2fs 1.41.14 (22-Dec-2010)
[root@linuxserver FS-CORRUPTION-TESTS]# lvextend -L+20G ggg/lll
File descriptor 7 (pipe:[25711]) leaked on lvextend invocation. Parent PID 2977: bash
  Extending logical volume lll to 120.00 GiB
  Logical volume lll successfully resized
[root@linuxserver FS-CORRUPTION-TESTS]# resize2fs /dev/mapper/ggg-lll 
resize2fs 1.41.14 (22-Dec-2010)
Filesystem at /dev/mapper/ggg-lll is mounted on /V/service/FS-CORRUPTION-TESTS/ppp; on-line resizing required
old desc_blocks = 7, new_desc_blocks = 8
Performing an on-line resize of /dev/mapper/ggg-lll to 31457280 (4k) blocks.
The filesystem on /dev/mapper/ggg-lll is now 31457280 blocks long.

[root@linuxserver FS-CORRUPTION-TESTS]# dumpe2fs /dev/mapper/ggg-lll >100formatted_mounted_resized
dumpe2fs 1.41.14 (22-Dec-2010)
[root@linuxserver FS-CORRUPTION-TESTS]# umount /dev/mapper/ggg-lll
[root@linuxserver FS-CORRUPTION-TESTS]# e2fsck -n /dev/mapper/ggg-lll 
e2fsck 1.41.14 (22-Dec-2010)
/dev/mapper/ggg-lll: clean, 12/1858560 files, 166215/31457280 blocks
[root@linuxserver FS-CORRUPTION-TESTS]# e2fsck -n -f /dev/mapper/ggg-lll 
e2fsck 1.41.14 (22-Dec-2010)
Pass 1: Checking inodes, blocks, and sizes
Inode 1549025 is in use, but has dtime set.  Fix? no

Inode 1549025 has a extra size (17896) which is invalid
Fix? no

Inode 1549025 should not have EOFBLOCKS_FL set (size 18446744073709551615, lblk -1)
Clear? no

Inode 1549025, i_size is 18446744073709551615, should be 0.  Fix? no

Inode 1549025, i_blocks is 281474976710655, should be 0.  Fix? no

Inode 1549026 is in use, but has dtime set.  Fix? no

Inode 1549026 has a extra size (65535) which is invalid
Fix? no

Inode 1549026 has INDEX_FL flag set but is not a directory.
Clear HTree index? no

HTREE directory inode 1549026 has an invalid root node.
Clear HTree index? no

Inode 1549026 should not have EOFBLOCKS_FL set (size 18446744073709551615, lblk -1)
Clear? no

Inode 1549026, i_size is 18446744073709551615, should be 0.  Fix? no

Inode 1549026, i_blocks is 281474976710655, should be 0.  Fix? no

Inode 1549027 is in use, but has dtime set.  Fix? no

Inode 1549027 has a extra size (41900) which is invalid
Fix? no
[...]
Comment 14 Roberto Ragusa 2012-12-15 15:14:53 EST
Also reproduced on:

# uname -a;rpm -q e2fsprogs
Linux thinkpad 3.6.6-1.fc16.x86_64 #1 SMP Mon Nov 5 16:56:43 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
e2fsprogs-1.41.14-2.fc15.x86_64
Comment 15 Roberto Ragusa 2012-12-15 15:16:30 EST
Also reproduced when mkfs has no layout options (-i, -G, -E).
Comment 16 Roberto Ragusa 2012-12-16 04:37:25 EST
Confirmed as serious data-eating bug in resize2fs.

According to dump2efs the flex_bg setting of the fs is ignored while resizing, so we have this bizarre situation with the bitmaps: (799 is the last group before resizing)

Group 797: (Blocks 26116096-26148863) [INODE_UNINIT, BLOCK_UNINIT]
  Checksum 0x29fe, unused inodes 1936
  Block bitmap at 25165853 (bg #768 + 29), Inode bitmap at 25166109 (bg #768 + 285)
  Inode table at 25169845-25169965 (bg #768 + 4021)
  32768 free blocks, 1936 free inodes, 0 directories, 1936 unused inodes
  Free blocks: 26116096-26148863
  Free inodes: 1542993-1544928
Group 798: (Blocks 26148864-26181631) [INODE_UNINIT, BLOCK_UNINIT]
  Checksum 0x5678, unused inodes 1936
  Block bitmap at 25165854 (bg #768 + 30), Inode bitmap at 25166110 (bg #768 + 286)
  Inode table at 25169966-25170086 (bg #768 + 4142)
  32768 free blocks, 1936 free inodes, 0 directories, 1936 unused inodes
  Free blocks: 26148864-26181631
  Free inodes: 1544929-1546864
Group 799: (Blocks 26181632-26214399) [INODE_UNINIT]
  Checksum 0xd1a8, unused inodes 1936
  Block bitmap at 25165855 (bg #768 + 31), Inode bitmap at 25166111 (bg #768 + 287)
  Inode table at 25170087-25170207 (bg #768 + 4263)
  32768 free blocks, 1936 free inodes, 0 directories, 1936 unused inodes
  Free blocks: 26181632-26214399
  Free inodes: 1546865-1548800
Group 800: (Blocks 26214400-26247167)
  Checksum 0x012f, unused inodes 0
  Block bitmap at 26215321 (+921), Inode bitmap at 26215322 (+922)
  Inode table at 26214400-26214520
  32645 free blocks, 1936 free inodes, 0 directories
  Free blocks: 26214521-26215320, 26215323-26247167
  Free inodes: 1548801-1550736
Group 801: (Blocks 26247168-26279935)
  Checksum 0x21e5, unused inodes 0
  Block bitmap at 26248090 (+922), Inode bitmap at 26248091 (+923)
  Inode table at 26247168-26247288
  32645 free blocks, 1936 free inodes, 0 directories
  Free blocks: 26247289-26248089, 26248092-26279935
  Free inodes: 1550737-1552672
Comment 17 Roberto Ragusa 2012-12-16 04:43:50 EST
Solved when, all other conditions identical, resize2fs from F17 is run (downloaded the rpms, extracted the executable and a couple of libs, run with LD_LIBRARY_PATH tricks).
(e2fsprogs-1.42.3-3.fc17.x86_64.rpm)

This is the dumpe2fs result. bg stuff correctly used after group 800.

Group 797: (Blocks 26116096-26148863) [INODE_UNINIT, BLOCK_UNINIT]
  Checksum 0x31cc, unused inodes 1936
  Block bitmap at 25165853 (bg #768 + 29), Inode bitmap at 25166109 (bg #768 + 285)
  Inode table at 25169845-25169965 (bg #768 + 4021)
  32768 free blocks, 1936 free inodes, 0 directories, 1936 unused inodes
  Free blocks: 26116096-26148863
  Free inodes: 1542993-1544928
Group 798: (Blocks 26148864-26181631) [INODE_UNINIT, BLOCK_UNINIT]
  Checksum 0x4e4a, unused inodes 1936
  Block bitmap at 25165854 (bg #768 + 30), Inode bitmap at 25166110 (bg #768 + 286)
  Inode table at 25169966-25170086 (bg #768 + 4142)
  32768 free blocks, 1936 free inodes, 0 directories, 1936 unused inodes
  Free blocks: 26148864-26181631
  Free inodes: 1544929-1546864
Group 799: (Blocks 26181632-26214399) [INODE_UNINIT]
  Checksum 0xc99a, unused inodes 1936
  Block bitmap at 25165855 (bg #768 + 31), Inode bitmap at 25166111 (bg #768 + 287)
  Inode table at 25170087-25170207 (bg #768 + 4263)
  32768 free blocks, 1936 free inodes, 0 directories, 1936 unused inodes
  Free blocks: 26181632-26214399
  Free inodes: 1546865-1548800
Group 800: (Blocks 26214400-26247167) [INODE_UNINIT]
  Checksum 0x9eb8, unused inodes 0
  Block bitmap at 26214400 (+0), Inode bitmap at 26214560 (+160)
  Inode table at 26214720-26214840 (+320)
  13088 free blocks, 1936 free inodes, 0 directories
  Free blocks: 26234080-26247167
  Free inodes: 1548801-1550736
Group 801: (Blocks 26247168-26279935) [INODE_UNINIT, BLOCK_UNINIT]
  Checksum 0xe791, unused inodes 0
  Block bitmap at 26214401 (bg #800 + 1), Inode bitmap at 26214561 (bg #800 + 161)
  Inode table at 26214841-26214961 (bg #800 + 441)
  32768 free blocks, 1936 free inodes, 0 directories
  Free blocks: 26247168-26279935
  Free inodes: 1550737-1552672
Group 802: (Blocks 26279936-26312703) [INODE_UNINIT, BLOCK_UNINIT]
  Checksum 0xc8d8, unused inodes 0
  Block bitmap at 26214402 (bg #800 + 2), Inode bitmap at 26214562 (bg #800 + 162)
  Inode table at 26214962-26215082 (bg #800 + 562)
  32768 free blocks, 1936 free inodes, 0 directories
  Free blocks: 26279936-26312703
  Free inodes: 1552673-1554608
Comment 18 Eric Sandeen 2012-12-16 19:19:08 EST
So it looks fixed in F17's e2fsprogs?  Hrm, interesting.  I can just rebase, but it'd be good to know what, exactly, fixed it.

But it looks like it's doing online resize; by and large there is very little going on in userspace there; it simply invokes an ioctl.

I'll see if I can reproduce using your steps & your geometry.

Thanks,
-eric
Comment 19 Eric Sandeen 2012-12-16 22:53:44 EST
I tried this to reproduce:

#!/bin/bash

lvremove -f testvg/testlv
vgremove testvg
pvremove /dev/loop0
losetup -d /dev/loop0
rm -f loopfile
truncate --size=125g loopfile
losetup /dev/loop0 loopfile 
pvcreate testpv /dev/loop0
vgcreate testvg /dev/loop0
lvcreate -n testlv -L100G testvg
mkfs.ext4 -i 69905 -G 256 -E resize=4290772992 -b 4096 /dev/mapper/testvg-testlv
mkdir -p mnt
mount /dev/mapper/testvg-testlv mnt/
date > mnt/timestamp.txt
lvextend -L+20G testvg/testlv
resize2fs /dev/mapper/testvg-testlv 
umount /dev/mapper/testvg-testlv 
e2fsck -fn /dev/mapper/testvg-testlv 

and failed to reproduce any problem.  Does that look like the right sequence of steps?

Since it's on a sparse file, perhaps it has something to do with stale data on disk.  I'll try it against a real disk again, I guess...

This is starting to look like a problem with the inode tables being properly zeroed out.  Hm, although your dumpe2fs does show them as having the INODE_UNINIT flag set...

Your first trouble found was:

> Inode 1549025 is in use, but has dtime set.

and the new group #800 contains that inode, but also has INODE_UNINIT set.

> Group 800: (Blocks 26214400-26247167) [INODE_UNINIT]
...
>  Free inodes: 1548801-1550736

Hm.  So if e2fsprogs isn't properly recognizing that the inode table is uninitialized and should be ignored, but instead checking random garbage, that could sure do it.  But I don't see anything obvious upstream that fixed a bug like that.

Are you at all handy with git bisect?  Once you have a filesystem which shows corruption for "e2fsck -fn" you can just use that as a testcase to bisect e2fsprogs between v1.41.14 and the version that worked in F17.  Just "make; make -C e2fsck e2fsck.static" and run that static binary as a test.
Comment 20 Eric Sandeen 2012-12-16 22:58:36 EST
Even if I write junk into 120G->121G, I don't trip it.  so it doesn't seem to be a "data is already zeroed because the loopback file was sparse" sort of problem...
Comment 21 Eric Sandeen 2012-12-16 23:01:27 EST
Ahah, I just hit it with this; ok, I'll take it from here :)  (and the "bad inodes" all have bad data of the format "0xCDCDCDCD..." which is the pattern that xfs_io scribbled into the 119G->121G range)

#!/bin/bash

lvremove -f testvg/testlv
vgremove testvg
pvremove /dev/loop0
losetup -d /dev/loop0
rm -f loopfile
echo
echo "Starting over:"
echo
truncate --size=125g loopfile
xfs_io -f -F -c "pwrite 119g 2g" loopfile
losetup /dev/loop0 loopfile 
pvcreate testpv /dev/loop0
vgcreate testvg /dev/loop0
lvcreate -n testlv -L100G testvg
mkfs.ext4 -i 69905 -G 256 -E resize=4290772992 -b 4096 /dev/mapper/testvg-testlv
mkdir -p mnt
mount /dev/mapper/testvg-testlv mnt/
date > mnt/timestamp.txt
lvextend -L+20G testvg/testlv
resize2fs /dev/mapper/testvg-testlv 
umount /dev/mapper/testvg-testlv
Comment 22 Eric Sandeen 2012-12-16 23:41:00 EST
Testing kernel 3.6.7 and upstream e2fsprogs also shows the issue; I'm not sure it's fixed upstream, TBH.

(I rebuilt & tested F17's e2fsprogs and found the same problem remains)
Comment 23 Eric Sandeen 2012-12-16 23:45:39 EST
(ok at this point I am talking to myself) ;)

I take it back, f17's resize2fs does fix it as you said.  I was thinking e2fsck problem, but upstream resize2fs + e2fsck seems fine.

I'll shut up now until I have something definitive.
Comment 24 Eric Sandeen 2012-12-17 00:22:59 EST
Seems that this e2fsprogs commit fixed it:

commit 9f6ba888f027ba4daf57ac61a11a6dce98e42347
Author: Yongqiang Yang <xiaoqiangnk@gmail.com>
Date:   Wed Sep 14 11:55:59 2011 -0400

    resize2fs: add support for new in-kernel online resize ioctl
    
    This is needed to support online resizing for > 32-bit file systems
    
    Signed-off-by: Yongqiang Yang <xiaoqiangnk@gmail.com>
    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>

Unfortunately all that did was switch to a new interface; it didn't fix the old one.  Probably enough for now, but the old interface seems to still need some fixing.
Comment 25 Roberto Ragusa 2012-12-17 03:54:48 EST
Hi, I don't know exactly how the userspace and the kernel cooperate for resizing. What I can say is that the F16 resize2fs probably initializes more stuff (and takes a lot of time), while the F17 resize2fs returns almost immediately. There is also the evident difference that the F16 one does not respect the flex_bg layout (I don't know if this is wrong or just a mis-optimization), while the F17 one respects flex_bg.

(BTW, ...spent the weekend restoring a couple terabytes from backups...)

(As a tribute to justice in the world, I want to say, so that search engines index it, that this system has been in production since 2005 on reiserfs and abused and resized without any similar issues; it was rebuild recently on new hardware and switched to ext4 and now I really risked to lose the data, as the remote versioned backup is also (resized) ext4, and the remote-remote backup is also (resized) ext4. I'm not complaining, but reiserfs really has a much worse reputation than actually deserved).
Comment 26 Carlos Maiolino 2012-12-26 08:26:44 EST
newer resize2fs delays initialization of bitmaps and inode tables of added groups (if possible) and add multi groups each time.
Also, all work of group table allocations are done in kernel space, making resize interface able to support flex_bg.

This is why you see F17 respecting flex_bg and also being faster than F16

--Cheers,
Carlos
Comment 27 Carlos Maiolino 2013-01-08 09:57:44 EST
I've been working on it with Eric and looks like it's a kernel problem and that it has been introduced between kernel v3.2 and v3.3

I'm bisecting the kernel between these versions to understand what happened in the middle there.

hopefully I might have some news soon
Comment 28 Carlos Maiolino 2013-01-14 14:01:24 EST
Hi, the upstream commit: 93f90526 partially fixes this issue, it is not the best solution but it's enough for now.

The 'best' fix to this issue is pending upstream here: http://marc.info/?l=linux-ext4&m=135808457606725&w=2 

and if possible should be applied after 93f90526, when it comes upstream.
Comment 29 Fedora Update System 2013-01-16 18:11:13 EST
kernel-3.6.11-8.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/kernel-3.6.11-8.fc17
Comment 30 Fedora End Of Life 2013-01-16 19:55:45 EST
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 31 Fedora Update System 2013-01-18 14:01:48 EST
kernel-3.7.3-101.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/kernel-3.7.3-101.fc17
Comment 32 Fedora Update System 2013-01-19 22:01:20 EST
Package kernel-3.7.3-101.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.7.3-101.fc17'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-1025/kernel-3.7.3-101.fc17
then log in and leave karma (feedback).
Comment 33 Fedora Update System 2013-01-24 17:38:41 EST
kernel-3.7.3-101.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.