Bug 510379 - resize2fs corrupting ext4 filesystems on ENOSPC
resize2fs corrupting ext4 filesystems on ENOSPC
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: e2fsprogs (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: Eric Sandeen
Fedora Extras Quality Assurance
:
Depends On:
Blocks: 516996
  Show dependency treegraph
 
Reported: 2009-07-08 16:55 EDT by Eric Sandeen
Modified: 2009-10-06 10:29 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-10-06 10:29:59 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
problematic image, pre-resize (6.73 MB, application/octet-stream)
2009-07-09 17:57 EDT, Eric Sandeen
no flags Details

  None (edit)
Description Eric Sandeen 2009-07-08 16:55:11 EDT
Resizing the attached image to 589824 blocks will corrupt it.

Things to note:

* 589824 is actually well above the claimed minimum fs size
* Things go bad in ext2fs_extent_set_bmap()
 - "(re/un)mapping first block in extent"
   - modifies existing extent (removes the block that is moving)
 - goto again
 - "mapping unmapped logical block"
   - tries to insert node and fails with ENOSPC
 - extent tree is now corrupted, it lost a block
* It's handy to make an e2image -r of the attached image for testing.
* Even if ENOSPC doesn't corrupt extent tree, block bitmaps are still left bad
* We could try to do better front & back merging
* We could bail early on any modification requiring a split if we have few blocks left
* We could try to unwrap after the error
* We could try to insert the new node before we modify the old
* etc :)
Comment 1 Eric Sandeen 2009-07-08 17:17:43 EDT
Oh, you'll probably want http://marc.info/?l=linux-ext4&m=124699503718719&w=2 too or you may hit that bug first...
Comment 2 Valerie Aurora Henson 2009-07-09 16:47:26 EDT
Three solutions proposed on #ext4 today:

* Create a special purpose ad hoc undo manager

* Fix ext2fs_extent_set_bmap() to return in uncorrupted state

* Fix the size calculation (which may include other changes to moving blocks - e.g., extent merging - to make this easier to compute)

Questions:

* How hard is each individual solution to implement?

* What is the likelihood of future bugs appearing after each fix?

* What is the common case/use case/situation we have to fix the most?

I think this bug is triggered by the Fedora livecd creator, which used to iteratively resize the file system smaller and now attempts to do it all in one fell swoop.

Answers to above questions welcomed...
Comment 3 Valerie Aurora Henson 2009-07-09 17:26:41 EDT
Another proposed solution: Keep track of the free space as the resize progresses.  Once it hits a certain threshold (the worst case number of blocks needed for some granularity of operation?), quit at a consistent place.
Comment 4 Eric Sandeen 2009-07-09 17:57:15 EDT
Created attachment 351192 [details]
problematic image, pre-resize

resize this image to 589824 blocks & fsck to see the problem.
Comment 5 Valerie Aurora Henson 2009-07-09 18:57:56 EDT
From tytso on #ext4:

The ext4 kernel code assumes a depth of 5; we've *never* seen anything with more than a depth of 3.  We don't have anything that enforces that limit; maybe we should.

I guess what we can do is iterate over all of the extent nodes and count the number of index nodes that have n-1 entries filled.

we don't even need to check all of the index nodes, just the ones on the path from the root to the extent leaf that we need to modify.

I just looked at the code; and it's trivial to count the number of blocks that might need to be split.

might as well do it right the first time around.   I can whip up the patch in about 5-10 minutes.
Comment 6 Theodore Tso 2009-07-10 00:34:47 EDT
I have a proposed patch that fixes most of this problem here:

http://patchwork.ozlabs.org/patch/29666/

Note that finding this problem wasn't really all that difficult.  I just set a break point in resize2fs's get_new_block() where it returns 0 indicating an allocation failure, and at that point just walked up the stack trace finding the problem.

There is still may be a potential problem if we fail inside a recursive node split (i.e., where an inode has more than one level of the extent tree that needs splitting) but that wasn't what was hitting us here.  This was another case of ext2fs_extent_set_bmap() doing something destructive (specifically, shortening an existing extent from the front end) before checking to make sure the insert would be successful.   The fix for this isn't that hard.

As long as we don't lose any data, I think it's fair game to ask the user to run e2fsck after a resize2fs failure to fix up the block allocation bitmaps, which is what still remains to be done after this patch is applied.
Comment 7 Eric Sandeen 2009-10-06 10:29:59 EDT
This has been fixed upstream now by:

commit 0dc29161125fa4650e4b8832fc0f570435ef0fe3
Author: Theodore Ts'o <tytso@mit.edu>
Date:   Fri Jul 10 00:19:28 2009 -0400

    libext2fs: Make ext2fs_extent_set_bmap() more robust against ENOSPC

Note You need to log in before you can comment on or make changes to this bug.