Description of problem: Conversion of current ext4 to BTRFS not possible due to segmentation fault in the end of process. Version-Release number of selected component (if applicable): e2fsprogs-1.45.5-3.fc32 btrfs-progs-5.6.1-1.fc32 kernel-5.6.18-300.fc32 How reproducible: Try to convert ext4 partition from Fedora Workstation latest Live respin. Mine was dated 2020-06-15 (https://dl.fedoraproject.org/pub/alt/live-respins/F32-WORK-x86_64-LIVE-20200615.torrent). Steps to Reproduce: 1. fsck /dev/sdX 2. btrfs-convert /dev/sdX Actual results: See screenshot https://atim.fedorapeople.org/Screenshot%20from%202020-06-27%2011-06-58.png Expected results: ext4 partition converted to BTRFS. Additional info: Fedora 32 (x86_64) [updated from F29->F30->F31->F32] Partition scheme (non-LVM): /dev/sda2 46G 29G 15G 66% / /dev/sda3 112G 60G 46G 57% /home /dev/sda1 248M 46M 203M 19% /boot/efi
Could you try 'e2fsck -yfv /dev/sda2' and report back the results? And were there any kernel messages at the time of the segfault/failure?
@Crhis: sudo e2fsck -yfv /dev/sda2 e2fsck 1.45.5 (07-Jan-2020) Pass 1: Checking inodes, blocks, and sizes Inode 790986 extent tree (at level 2) could be narrower. Optimize? yes Inode 791012 extent tree (at level 2) could be narrower. Optimize? yes Pass 1E: Optimizing extent trees Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information fedora: ***** FILE SYSTEM WAS MODIFIED ***** 908080 inodes used (29.86%, out of 3041472) 2728 non-contiguous files (0.3%) 1265 non-contiguous directories (0.1%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 719705/631/2 7769578 blocks used (63.90%, out of 12158208) 0 bad blocks 2 large files 550998 regular files 83658 directories 0 character device files 0 block device files 0 fifos 398901 links 273352 symbolic links (187671 fast symbolic links) 63 sockets ------------ 1306972 files [liveuser@localhost-live ~]$
I tried but failed to reproduce the issue using https://dl.fedoraproject.org/pub/alt/live-respins/F32-WORK-x86_64-LIVE-20200701.iso I installed using non-lvm disk layout (partitions + ext4) and seems like I managed to convert both the boot and root partitions: [liveuser@localhost ~]$ sudo btrfs-convert /dev/sda3 create btrfs filesystem: blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c creating ext2 image file creating btrfs metadata copy inodes [o] [ 160948/ 159242] conversion complete [liveuser@localhost ~]$ sudo btrfs-convert /dev/sda1 create btrfs filesystem: blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c creating ext2 image file creating btrfs metadata copy inodes [o] [ 106/ 408] conversion complete [liveuser@localhost ~]$ lsblk -f NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT [...] sda ├─sda1 btrfs a5ecf524-d462-4771-a5ec-d538ff22dce8 ├─sda2 swap 1 782abc5e-f0c7-42c9-baaa-5cd333d5c8ab [SWAP] └─sda3 btrfs 0d1bbdc1-66a8-484e-9933-be70c5987362 sr0 iso9660 Joliet Extensio F32-WORK-x86_64 2020-07-01-16-43-26-00 0 100% /run/initramfs/liv [liveuser@localhost ~]$ uname -r 5.6.19-300.fc32.x86_64 [liveuser@localhost ~]$ rpm -q btrfs-progs btrfs-progs-5.6.1-1.fc32.x86_64 Did I miss anything? If you could send the virtual machine image somehow it could probably help speed up repro. Thanks, Daniel
> Did I miss anything? If you could send the virtual machine image somehow > it could probably help speed up repro. I waited for new respins with new kernel and new btrfs-progs. Also there is some interesting changes https://github.com/kdave/btrfs-progs/blob/master/CHANGES#L9 Also JFYI i am trying to convert on real machine, not VM. Or i just read it wrong? :) Anyway this is a good idea and maybe i can tar somehow my / and upload it somewhere... Some additional info: since this time i converted two other partitions on this machine. One successfully and and one (/home) not, but i have different bug with /home, not this one. Also i think i should freeing up more disk space next time when will try to convert again /. 66% (15G) could be not enough maybe?
There's definitely a bug here, it shouldn't crash. While it should be reversible, I highly recommend backups. Maybe one day down the road we can have a test day just for convert. A big challenge is getting enough aged (real world) file systems as examples. btrfs-progs 5.7 should be available by update now, it's in the stable repo for all Fedora versions. Can you try: $ sudo e2fsck -fyv And report those results. Also check for a coredump. $ coredumpctl If there's a coredump for btrfs listed, and the coredump file is 'present' then it's possible to process a backtrace. It's a bit tedious but I think it's better than uploading a large file system somewhere. Basically you'll do: $ coredumpctl gdb <pid> ## where <pid> is the number in the 3rd column for the line your coredump appears in. This will generate some text and end with a gdb prompt. Type exit to get out. But you and copy paste the recommended dnf debuginfo install command with the likely long list of debug info packages. In some cases, it'll take a few rounds of installing debug info packages, then back to 'coredumpctl gdb <pid>'. But eventually it won't ask for more debuginfos. And now at the gdb prompt type: thread apply all bt full that'll produce a page of backtrace, with an option at the bottom to press space or type c (enter) to continue with all the pages. I suggest taking all the output from this most recent gdb, stick it in a text file, and attach to the bug report.
> Also JFYI i am trying to convert on real machine, not VM. Ah got it, thought you were testing stuff on a VM. I guess having a core file would be pretty useful, like Chris said.
(In reply to Chris Murphy from comment #5) @Chris, some news here: now i tried to convert again this partition today. I used: - This respin https://dl.fedoraproject.org/pub/alt/live-respins/F32-WORK-x86_64-LIVE-20200715.torrent - Kernel: 5.7.8 - btrfs-progs-5.7.0-1.fc32 - i freed up more disk space, this time 59% used **No segfault this time but still failed:** [liveuser@localhost-live ~]$ uname -a Linux localhost-live 5.7.8-200.fc32.x86_64 #1 SMP Thu Jul 9 14:34:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [liveuser@localhost-live ~]$ sudo e2fsck -fyv /dev/sda2 e2fsck 1.45.5 (07-Jan-2020) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information 819723 inodes used (26.95%, out of 3041472) 3310 non-contiguous files (0.4%) 1203 non-contiguous directories (0.1%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 633459/567 6810573 blocks used (56.02%, out of 12158208) 0 bad blocks 2 large files 471075 regular files 78256 directories 0 character device files 0 block device files 0 fifos 320516 links 270320 symbolic links (185626 fast symbolic links) 63 sockets ------------ 1140230 files [liveuser@localhost-live ~]$ sudo btrfs-convert /dev/sda2 create btrfs filesystem: blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c creating ext2 image file ERROR: missing data block for bytenr 2555437056 ERROR: failed to create ext2_saved/image: -2 WARNING: an error occurred during conversion, filesystem is partially created but not finalized and not mountable
Following this failure, what do you get for the same e2fsck command? I think this bug may have been figured out, patch is pending. https://lore.kernel.org/linux-btrfs/20200720125109.93970-1-wqu@suse.com/
FEDORA-2020-b719b4ebe3 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-b719b4ebe3
FEDORA-2020-216f6116c0 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-216f6116c0
FEDORA-2020-216f6116c0 has been pushed to the Fedora 31 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2020-b719b4ebe3 has been pushed to the Fedora 32 stable repository. If problem still persists, please make note of it in this bug report.
Well, 'btrfs-progs-5.7-4.fc32' fixed previous issue, but not fixed this mine issue, reopening. This time i thought it will success finally and converting operation taked long time, but failed in the end: [liveuser@localhost-live ~]$ sudo btrfs-convert /dev/sda2 create btrfs filesystem: blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c creating ext2 image file creating btrfs metadata convert/source-ext2.c:845: ext2_copy_inodes: BUG_ON `ret` triggered, value -28 btrfs-convert(+0x13589)[0x558cfe4a6589] btrfs-convert(+0x14549)[0x558cfe4a7549] btrfs-convert(main+0xfc0)[0x558cfe4a22b0] /lib64/libc.so.6(__libc_start_main+0xf2)[0x7fd2c6666042] btrfs-convert(_start+0x2e)[0x558cfe4a35ce] Aborted [liveuser@localhost-live ~]$ sudo dnf info btrfs-progs Last metadata expiration check: 0:22:12 ago on Tue 28 Jul 2020 06:28:14 AM EDT. Installed Packages Name : btrfs-progs Version : 5.7 Release : 4.fc32 Architecture : x86_64 Size : 4.6 M Source : btrfs-progs-5.7-4.fc32.src.rpm --- Something make me believe that this happens because of lack free space (40%)...
Cool, thanks for retesting and reopening. Upstream is informed.
@artem, would you please upload the `e2image -Q` dump of /dev/sda2? I'm wondering if this is a real ENOSPC. Btrfs-convert is not directly using the free space, but a subset of the ext* free space, there are a lot of padding to avoid fragmented small data chunks. Thus that 40% free reported from ext* is just a up limit. If the used data space are too fragmented, then btrfs may have little space to utilize and cause ENOSPC. Anyway, I'll try to enhance the ENOSPC handling of btrfs-convert first.
Hi Artem, are you able to provide an image of the ext4 file system in question so we can progress this bug report? Thanks.
Hi, don't get me wrong, i am still interesting in converting my / into btrfs as well on this machine, all other partitions already converted/formatted. But this requires some time and everytime i am trying to do some backups and cleanups before this. IMO before image dump it's better to freeing up some disk space again and re-test conversion with latest respin. I'll try today and will provide a feedback. Also please can you give me some advises maybe how to exclude /etc/shadow and some other private data if there some stored on / partition? And maybe i can send in PM the link with image to you Chris, Neal and Qu Wenruo?
Oops, needinfo was automaticaly removed, reverting.
(In reply to Artem from comment #17) > But this requires some time and everytime i am trying to do some backups and > cleanups before this. IMO before image dump it's better to freeing up some > disk space again and re-test conversion with latest respin. Your effort to make btrfs better is appreciated. Note that the more you change the ext4 file system, the less likely we will be able to find and fix the bug. We really need the image of the ext4 file system that's failing to convert. > Also please can you give me some advises maybe how to exclude /etc/shadow > and some other private data if there some stored on / partition? e2image does not include any data. It only includes file system metadata, including file names. I do not see a way in e2image to mask file names.
> Note that the more you change the ext4 file system, the less likely we will be able to find and fix the bug. We really need the image of the ext4 file system that's failing to convert. Yep, that's why i still not formatted it into btrfs. :) > e2image does not include any data. It only includes file system metadata, including file names. I do not see a way in e2image to mask file names. Ah, right. I thought it will create full image with data. Done: https://atim.fedorapeople.org/for-rhbz/sda2-dump.qcow2.xz --- Before creating image i tried to convert again with latest respin (https://dl.fedoraproject.org/pub/alt/live-respins/F32-WORK-x86_64-LIVE-20200804.torrent) and got segfault again: sudo e2fsck -yfv /dev/sda2 e2fsck 1.45.5 (07-Jan-2020) Pass 1: Checking inodes, blocks, and sizes Inode 1966383 extent tree (at level 2) could be narrower. Optimize? yes Inode 1966384 extent tree (at level 2) could be narrower. Optimize? yes Pass 1E: Optimizing extent trees Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information fedora: ***** FILE SYSTEM WAS MODIFIED ***** 846035 inodes used (27.82%, out of 3041472) 3431 non-contiguous files (0.4%) 1248 non-contiguous directories (0.1%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 647508/558/2 7125607 blocks used (58.61%, out of 12158208) 0 bad blocks 2 large files 479724 regular files 82643 directories 0 character device files 0 block device files 0 fifos 380126 links 283595 symbolic links (197895 fast symbolic links) 64 sockets ------------ 1226152 files [liveuser@localhost-live ~]$ sudo btrfs-convert /dev/sda2 create btrfs filesystem: blocksize: 4096 nodesize: 16384 features: extref, skinny-metadata (default) checksum: crc32c creating ext2 image file creating btrfs metadata btrfs unable to find ref byte nr 50728550400 parent 0 root 2 owner 1 offset 0 path->slots[0]: 0 path->nodes[0]: Segmentation fault --- But! This time i noticed DE notification which said that / partition have not enough space, 51 MB. Root partition in this case is my RAM i guess since this Live Fedora image. So this could be not related to btrfs-convert bug at all, right?
I'm not sure why space is needed for the local sysroot to do this conversion. Does the 20200804 respin have btrfs-progs 5.7-4 or newer? It probably does but I wanna double check.
(In reply to Chris Murphy from comment #21) > Does the 20200804 respin have btrfs-progs 5.7-4 or newer? 20200804 respin contain btrfs-progs 5.7-4 version. > I'm not sure why space is needed for the local sysroot to do this conversion. Same here. I was AFK during conversion operation since it take some time and when came back i seen this notification. I'll try next time to monitor what's going on when btrfs-convert working... But this the fact and this weird anyway IMO. The previous partitions which i converted i did on the same machine as well and one of them was successful.
Update: yesterday i converted successfully '/' partition finally. - Now i was on f33 (upgraded from f32) - used Fedora-Workstation-Live-x86_64-33-20200917.n.0.iso as Live system - btrfs-progs-5.7-5.fc33 - ~65% disk space used After some GRUB, fstab tweaking and selinux relabeling all works fine. The old one metadata still available in comment #20 (https://atim.fedorapeople.org/for-rhbz/sda2-dump.qcow2.xz).
FEDORA-2020-c4bf3a681b has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-c4bf3a681b
FEDORA-2020-3596748fea has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-3596748fea
FEDORA-2020-c4bf3a681b has been pushed to the Fedora 32 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-c4bf3a681b` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-c4bf3a681b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-3596748fea has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-3596748fea` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3596748fea See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-3596748fea has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2020-c4bf3a681b has been pushed to the Fedora 32 stable repository. If problem still persists, please make note of it in this bug report.