Bug 1979179

Summary: Conversion of ext4 filesystem to btrfs fails with error.
Product: [Fedora] Fedora Reporter: piio <bugzilla>
Component: btrfs-progsAssignee: Josef Bacik <josef>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 35CC: adam900710, bugzilla, davide, esandeen, giorgos, josef, michel, ngompa13
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-12-13 15:26:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
coredump of btrfs-convert process
none
show_super_stats result none

Description piio 2021-07-05 07:11:01 UTC
Created attachment 1798025 [details]
coredump of btrfs-convert process

Description of problem:
Conversion of ext4 filesystem to btrfs fails with error

Version-Release number of selected component (if applicable):
btrfs-progs-5.12.1-2.fc35.x86_64

How reproducible:
In my filesystem always

Steps to Reproduce:
1. Run Fedora Rawhide livecd
2. fsck volume which i want to convert, no errors found

[root@localhost-live ~]# fsck.ext4 -fyv /dev/mapper/fedora--volgroup 
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      707293 inodes used (21.58%, out of 3276800)
        7406 non-contiguous files (1.0%)
        1851 non-contiguous directories (0.3%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 647435/2158
     7705257 blocks used (58.79%, out of 13107200)
           0 bad blocks
           1 large file

      567653 regular files
       63898 directories
          12 character device files
           0 block device files
           0 fifos
       48357 links
       75535 symbolic links (57494 fast symbolic links)
         186 sockets
------------
      755641 files


3. run conversion:
[root@localhost-live ~]# btrfs-convert -p /dev/mapper/fedora--volgroup 
create btrfs filesystem:
	blocksize: 4096
	nodesize:  16384
	features:  extref, skinny-metadata (default)
	checksum:  crc32c
free space report:
	total:     53687091200
	free:      578277376 (1.08%)
creating ext2 image file
creating btrfs metadata
btrfs unable to find ref byte nr 54126084096 parent 0 root 2  owner 0 offset 0
path->slots[0]: 0 path->nodes[0]:
Segmentation fault (core dumped)

Actual results:
btrfs-convert segfaults.
Coredump in attachment

Expected results:
Filesystem converted from ext4 to btrfs without any errors

Additional info:
I checked few times, with the same arguments it's always falling.
When I disabled datasum, ignored xattrs and ACL and disable inlining small files (btrfs-convert -d -n -i ), conversion succeded.

Comment 2 Chris Murphy 2021-07-05 22:06:40 UTC
piio would you attach the file from the following command for the ext4 file system you want to convert? substitute /dev/loop0 for your device

# debugfs -R show_super_stats /dev/loop0 > ext4-stats.txt

Comment 3 Qu Wenruo 2021-07-06 09:23:35 UTC
For the available space mismatch thing, it's more or less expected.

Btrfs needs large chunks of unallocated space, thus fragmented space is not going to be counted as available.

With around 500 MiB space left, I highly doubt if we have enough space for metadata (especially csum, which can be pretty big for this particular case)

Thus I thing the final error is related to the ENOSPC.


For better debugging and to enhance btrfs-convert for its error message, would you mind to dump the source ext4 fs by:

# e2image -Q <device>

And upload it?

Comment 4 piio 2021-07-06 09:57:52 UTC
Created attachment 1798520 [details]
show_super_stats result

debugfs show_super_stats result

Comment 5 piio 2021-07-06 10:33:16 UTC
gzipped e2image dump - https://we.tl/t-xNUVnY0ReA

Comment 6 Chris Murphy 2021-07-08 03:21:27 UTC
>Block count:              13107200
>Free blocks:              5395324

50G file system with 20G free? The super shows quite a lot of block groups unused, so it shouldn't be badly fragmented.

Comment 7 Qu Wenruo 2021-07-10 07:11:23 UTC
Can't reproduce the problem here.

The same btrfs-progs v5.12.1.

But convert seems to detect everything correctly:

free space report:
        total:     991297536000
        free:      152739930112 (15.41%)

The total size matches the raw image, and the free space also more or less matches the report from e2fsprogs.

The convert is taking a quite long time to run, but so far no crash.

I'm wondering why the initial report has a completely different (maybe something underflowed?) for the free space:

	total:     53687091200
	free:      578277376 (1.08%)

Or is the e2image dump a different filesystem?

Comment 8 Qu Wenruo 2021-07-10 07:18:38 UTC
My bad, confused with another image...

And yes, I can reproduce the reported free space problem now, but I can't reproduce the crash.

The error I got is a graceful error:

 creating btrfs metadata
 ERROR: failed to commit transaction: -28
 ERROR: error during copy_inodes -28
 WARNING: error during conversion, the original filesystem is not modified

Will look into the problem locally, and update when found something.

Comment 9 Qu Wenruo 2021-07-10 07:51:03 UTC
I'm afraid btrfs-convert is correct.

I dumped all used space reported from e2progs libs and calculated the free space.

It looks like, although the ext4 system has quite some free space left, the free space is very fragmented.
The average size of all free space is only around 350KiB.

Thus only a very small set of those free space can be used.

To be more accurate, there are 64848 free space fragments, but only 141 of them are larger than 16MiB, the minimal chunk size of btrfs.


Thus it's really super framgmented free space causing the problem.

Not sure about ext4, but maybe defrag could help?

Comment 10 Ben Cotton 2021-08-10 13:11:05 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 35 development cycle.
Changing version to 35.

Comment 11 piio 2021-08-22 14:58:39 UTC
Hi,

I'm sorry but till now I didn't have time to verify this issue.
I don't found better way to defragment free space of existing ext4, so I created gzipped archive with most of data from this partition (/usr and /opt where i got git repositories). After that I converted successfully ext4 to btrfs, after that I restored previously removed data.

Maybe is it possible to detect such highly-framented free space filesystem and suggest some actions to user? Or maybe there is any tool to automatically defragment ext4 free space?

Comment 12 Giorgos Skafidas 2022-09-10 15:12:16 UTC
(In reply to piio from comment #11)
> Hi,
> 
> I'm sorry but till now I didn't have time to verify this issue.
> I don't found better way to defragment free space of existing ext4, so I
> created gzipped archive with most of data from this partition (/usr and /opt
> where i got git repositories). After that I converted successfully ext4 to
> btrfs, after that I restored previously removed data.
> 
> Maybe is it possible to detect such highly-framented free space filesystem
> and suggest some actions to user? Or maybe there is any tool to
> automatically defragment ext4 free space?
I ran into this issue today as well. One way to defragment free space is to use resize2fs to shrink the file system by the amount of space one expects btrfs to require for conversion (a few GiB?), then expand it back to its original size. All data will have been moved towards the beginning of the file system and btrfs-convert will have contiguous space at the end of it to work with.

Comment 13 Ben Cotton 2022-11-29 16:59:29 UTC
This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 14 Ben Cotton 2022-12-13 15:26:02 UTC
Fedora Linux 35 entered end-of-life (EOL) status on 2022-12-13.

Fedora Linux 35 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.