Bug 889888

Created attachment 668141 [details]
first5MB.bin.bz2

First 5MB of /dev/sda.

So I've gone through a bunch of permutations that don't involved repartitioning, and can't reproduce this. I wonder if wipefs isn't wiping something that in the course of a subsequent repartition, then allows btrfs fi show to see remnants of a previous btrfs file system.

(In reply to comment #0)
...
> 1. Device sda single partition sda1 is one of six devices in a btrfs volume.
...

How did you create the initial disk state?

(In reply to comment #0)
...
> 3. wipefs -a /dev/sda, id's as MBR and wiped
...

Immediately after rebooting from the Live CD, I get an error trying to run "wipefs -a /dev/sda":

[root@localhost liveuser]# wipefs -n /dev/sda
offset               type
----------------------------------------------------------------
0x1fe                dos   [partition table]

[root@localhost liveuser]# wipefs -a /dev/sda
wipefs: error: /dev/sda: probing initialization failed: Device or resource busy
[root@localhost liveuser]# rpm -q util-linux
util-linux-2.22.1-2.1.fc18.x86_64

Tested by booting from the Live CD with:
$ qemu-kvm -m 2048 -hda f18-test-3.img -cdrom ~/xfr/fedora/F18/F18-Final/TC3/Fedora-18-TC3-x86_64-Live-Desktop.iso -usb -vga qxl -boot menu=on -usbdevice mouse

(In reply to comment #3)
> How did you create the initial disk state?

I'm not certain. The virtual disk had been used and reused. The most recent state was produced by specifying it and five other virtual disks, in Installation Destination, then in Manual Partitioning making a single Root mountpount, Btrfs.

Thanks. Here's a reproducer using the installer:

Create an empty 12 GiB disk image:
$ qemu-img create f18-test-3.img 12G

Start the Live CD and update the installer:
$ qemu-kvm -m 2048 -hda f18-test-3.img -cdrom ~/xfr/fedora/F18/F18-Final/TC3/Fedora-18-TC3-x86_64-Live-Desktop.iso -usb -vga qxl -boot menu=on -usbdevice mouse
anaconda-18.37.8-1.fc18

Start the installer:
Installation Options:
    Partition scheme configuration: BTRFS
    Check "Let me customize ..."
Manual Partitioning:
    Click "+" to add a new mount point.
    Mount Point: /
    Desired Capacity: [blank]
Complete installation.
Quit installer.

From a terminal:
# btrfs fi show
# lsblk -f /dev/sda

Start the installer:
Remove all mount points and partitions (There should be only one.).
Click to auto-configure storage.
Complete installation.
Quit installer.

From a terminal:
# btrfs fi show
# lsblk -f /dev/sda

(In reply to comment #6)
> Thanks. Here's a reproducer using the installer:
...

Here are the results after each install:

$ cat reproducer-results-1.txt 
[root@localhost liveuser]# btrfs fi show
Label: 'fedora'  uuid: ccca0d26-0dda-4806-bf6d-8c62ef875c25
	Total devices 1 FS bytes used 3.08GB
	devid    1 size 12.00GB used 6.04GB path /dev/sda1

Btrfs Btrfs v0.19
[root@localhost liveuser]# lsblk -f /dev/sda
NAME   FSTYPE LABEL  UUID                                 MOUNTPOINT
sda                                                       
└─sda1 btrfs  fedora ccca0d26-0dda-4806-bf6d-8c62ef875c25 /mnt/sysimage
[root@localhost liveuser]# # < reinstall here >
[root@localhost liveuser]# btrfs fi show
Label: 'fedora'  uuid: ccca0d26-0dda-4806-bf6d-8c62ef875c25
	Total devices 1 FS bytes used 1.04MB
	devid    1 size 12.00GB used 6.04GB path /dev/sda1

Btrfs Btrfs v0.19
[root@localhost liveuser]# lsblk -f /dev/sda
NAME                   FSTYPE      LABEL UUID                                   MOUNTPOINT
sda                                                                             
├─sda1                 ext4              5a12e6f2-2857-48e8-a29b-5437b81630e7   /mnt/sysimage/boot
└─sda2                 LVM2_member       AZlxMb-ZGn4-zO2v-ItmL-yaHL-n0TQ-cZnVG3 
  ├─fedora-swap (dm-2) swap              f1edee57-e53f-4632-bfe9-f152c3cdd346   [SWAP]
  └─fedora-root (dm-3) ext4              5a783d1f-8267-4e7a-abdb-a9dd5fcef3aa   /mnt/sysimage
[root@localhost liveuser]# rpm -q anaconda btrfs-progs util-linux
anaconda-18.37.8-1.fc18.x86_64
btrfs-progs-0.20.rc1.20121017git91d9eec-1.fc18.x86_64
util-linux-2.22.1-2.1.fc18.x86_64
[root@localhost liveuser]#

Created attachment 669257 [details]
anaconda.log

Created attachment 669258 [details]
program.log

Created attachment 669259 [details]
storage.log

(In reply to comment #4)
...
> Immediately after rebooting from the Live CD, I get an error trying to run
> "wipefs -a /dev/sda":
...
> [root@localhost liveuser]# wipefs -a /dev/sda
> wipefs: error: /dev/sda: probing initialization failed: Device or resource
> busy
...

The sequence needs to be:
# swapoff -a
# vgchange -a n fedora
# wipefs -a /dev/sda

The attached program.log has records for both installs:

[snippets from attached program.log]
...
10:13:39,902 INFO program: Running... swapoff -a
10:13:40,002 INFO program: Running... wipefs -a /dev/sda
...
10:13:40,675 INFO program: Running... wipefs -a /dev/sda1
...
10:13:41,016 INFO program: Running... mkfs.btrfs --data=single --label=fedora /dev/sda1
...

...
10:33:26,101 INFO program: Running... swapoff -a
...
15:33:26,479 INFO program: Running... /bin/mount -n -t btrfs -o defaults /dev/sda1 /tmp/btrfs-tmp.2ILsJ2v
10:33:26,567 INFO program: Running... btrfs subvol delete /tmp/btrfs-tmp.2ILsJ2v/root
10:33:26,593 INFO program: Delete subvolume '/tmp/btrfs-tmp.2ILsJ2v/root'
15:33:26,603 INFO program: Running... /bin/umount /tmp/btrfs-tmp.2ILsJ2v
...
10:33:35,316 INFO program: Running... wipefs -a /dev/sda1
10:33:35,381 INFO program: /dev/sda1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
...
10:33:36,259 INFO program: Running... wipefs -a /dev/sda1
...
10:33:36,451 INFO program: Running... mke2fs -t ext4 /dev/sda1
...

(In reply to comment #12)
...
> 10:13:41,016 INFO program: Running... mkfs.btrfs --data=single
> --label=fedora /dev/sda1
> ...

I omitted the records for subvolume creation:

...
15:13:41,939 INFO program: Running... /bin/mount -n -t btrfs -o defaults /dev/sda1 /tmp/btrfs-tmp.15Xk4f7R
10:13:42,004 INFO program: Running... btrfs subvol create /tmp/btrfs-tmp.15Xk4f7R/root
10:13:42,177 INFO program: Create subvolume '/tmp/btrfs-tmp.15Xk4f7R/root'
15:13:42,189 INFO program: Running... /bin/umount /tmp/btrfs-tmp.15Xk4f7R
...

I've got another case of this, and it's always multiple disk Btrfs volumes.

# btrfs fi show
Label: 'fedora_f18v'  uuid: c98ee018-6122-4541-aa71-80705b4ecab7
	Total devices 2 FS bytes used 3.03GB
	devid    2 size 80.00GB used 6.01GB path /dev/sdb1
	devid    1 size 75.57GB used 6.03GB path /dev/sda2

Label: 'fedora_f18v'  uuid: ddce17dd-037a-474e-81cf-3b4dc9197206
	Total devices 2 FS bytes used 3.08GB
	devid    1 size 80.00GB used 6.03GB path /dev/sda1
	*** Some devices missing

Btrfs Btrfs v0.19


# lsblk -f /dev/sda
NAME   FSTYPE LABEL       UUID                                 MOUNTPOINT
sda                                                            
├─sda1 ext4               6a2f4d98-7e57-4885-96ea-341dd6146fe9 /boot
├─sda2 btrfs  fedora_f18v c98ee018-6122-4541-aa71-80705b4ecab7 /home
└─sda3 swap               953ceb5c-5ebc-4658-abc7-98ab9a1066fb [SWAP]


The 2nd btrfs fi show result uuid: ddce... is bogus. That's the previous btrfs volume, which would have been a two disk, single partition only (on each disk), Btrfs volume, -d single -m raid1.

Unclear to me still if this is a wipefs bug, or a btrfs fi show bug. Or maybe both.

OK this is interesting. Brand new 80GB virtual disk, so it's completely zero'd.

1. fdisk to create one partition, all defaults.
2. Format that sda1 with mkfs.btrfs -L first.
3. btrfs fi show displays "first" labeled volume.
4. wipefs -a /dev/sda1 and it finds btrfs metadata, ostensibly removes it.
5. btrfs fi show does NOT show "first" anymore.
6. fdisk /dev/sda again, delete the one partition; create a new partition 500MB and a second one for remaining space.
7. btrfs fi show still does not show "first".
8. mkfs.ext4 /dev/sda1
9. btrfs fi show NOW SHOWS "first" on sda1!
10. Reboot
11. btrfs fi show still shows "first" on sda1, size 80GB.

Tentative conclusions:

A. wipefs isn't deleting all btrfs metadata it could. 

B. mkfs.ext4 is refreshing stale btrfs metadata in a way that causes btrfs fi show to see.

C. btrfs fi show seems to blatantly disregard the partition map, because it shows a volume the partition map cannot possibly validly support.

D. Nothing else sees this phantom btrfs volume. btrfs fi show seems confused.


btrfs-progs-0.20.rc1.20121017git92d9eec-1.fc18.x86_64
e2fs-progs-1.42.5-1.fc18.x86_64
kernel 3.7.1-2

Even easier to reproduce:

1. Partition of any size.
2. mkfs.btrfs /dev/sda1
3. wipefs -a /dev/sda1
4. btrfs fi show doesn't find it.
5. mkfs.ext4 /dev/sda1
6. btrfs fi show sees the stale volume.

Repartitioning has nothing to do with it. Happens with all sized partitions I've tested from 500MB to 2TB.

Nice work.

There appear to be redundant copies of the signature, "_BHRfS_M, in the partition. Wipefs zeroes the first one, but there are more further down in the partition. The label and the UUID are also replicated.

# mkfs.btrfs -L foo1 /dev/sda1
# dd if=/dev/sda | strings | cat -n
# wipefs -a /dev/sda1
# dd if=/dev/sda | strings | cat -n

Yeah even a dd if=/dev/zero for 50MB doesn't clear it. Maybe Btrfs has some magical properties after all, that allow it to tolerate some data recovery in the event it's been wipefs'd and formatted by accident.

The btrfs Wiki lists "fault tolerance, repair and easy administration" as objectives:
https://btrfs.wiki.kernel.org/index.php/Main_Page

The FAQ says to use wipefs "To actually remove the filesystem":

How to clean up old superblock ? 
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_to_clean_up_old_superblock_.3F

That is followed by this advice:

What if I don't have wipefs at hand?
# dd if=/dev/zero of=/dev/sda bs=1k count=4 seek=64

https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#What_if_I_don.27t_have_wipefs_at_hand.3F

The wiki is wrong. For one, if you try it, it doesn't work, btrfs fi show still sees the volume. Second, it's only 4096 bytes, and Btrfs has a 64KB pad before the beginning of the file system anyway. That pad, if it doesn't have grub in it, is all zeros anyway. So this dd command in the wiki does exactly nothing.

Anyway, it's increasingly clear to me it's a bug in btrfs-progs. *shrug*

The dd command does a seek:
# dd if=/dev/zero of=/dev/sda bs=1k count=4 seek=64
                                            ^^^^^^^

Anyway, I agree that this should go to btrfs-progs. wipefs is doing exactly what the wiki says to do.

OH right, well funny enough the dd command given, btrfs fi show still displays it, I don't even have to do resurrection with mkfs.ext4.

At least with wipefs, it does go away. So it seems btrfs fi show is just being extremely optimistic. Changing to btrfs-progs.

Btrfs stores several copies of the superblock (IIRC three). During the
scan it searches for a valid superblock. If the first is wrong, checks
the second and so on...
It stops the search when one of the following conditions is true:
1) the superblock "magic" field is zero (typically when a disk is removed)
2) the superblock is valid
3) there is no other superblock.

I think that wipefs zeros the "magic" field on the first superblock. So
the btrfs scan process ends at the first superblock.

When the disk is used again, it is possible that the first superblock is
overwritten and the "magic" field is not zero any more. So in case of
further scan by btrfs, the first superblock is ignored and the check is
performed to the second one and the third one, which could be valid, if
nobody zeroed its "magic" field.

(In reply to comment #19)
...
> What if I don't have wipefs at hand?
> # dd if=/dev/zero of=/dev/sda bs=1k count=4 seek=64
> 
> https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#What_if_I_don.
> 27t_have_wipefs_at_hand.3F
...

This seems incorrect. The offset should be relative to the beginning of the _partition_, not the disk:
# dd if=/dev/zero of=/dev/sda1 bs=1k count=4 seek=64
.............................^

NB: The wiki has been updated, but it still uses /dev/sda.
https://btrfs.wiki.kernel.org/index.php?title=Problem_FAQ&oldid=16201

Correcting myself. The file system can span the whole disk. The FAQ section on clearing the file system should note that the file system can be created on a disk or in a partition, so the device should be /dev/sda or /dev/sdaN, as appropriate.

Looks like there are three superblocks, maybe not including the mirrored copies (?). In any case, discussion on linux-btrfs list here:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg20947.html
patch for wipefs here:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg20959.html

Maybe the bug needs to be set back to util-linux.

Thanks, Chris and Goffredo.

Patching wipefs might fix the problem, but why isn't a missing superblock reported as a file system inconsistency? Couldn't data in a different file system mimic a btrfs superblock?

(In reply to comment #27)
> Patching wipefs might fix the problem,

I'm going to add support for the backup superblocks to libblkid.

> but why isn't a missing superblock
> reported as a file system inconsistency?

This is good question. I agree that the current btrfs behaviour is strange, the filesystem (utils) should be more paranoid...

util-linux-2.22.2-2.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/util-linux-2.22.2-2.fc18

While it barely makes sense to me, it seems that after updating util-linux, libblkid, libmount and libuuid from 2.22.1-2.4.fc18 to 2.22.2-2 my fedora is no longer able to behave as a NFS server. 

Mount attempts by clients (local, or various remote debian, ubuntu, fedora and freebsd clients) produce ESTALE return codes from mount(2) with NFS3, and "cannot allocate memory" with NFS4.

In the same upgrade session I also received and nfs-utils 1.2.6-14 to 1.2.7-3 update, but just downgrading that (yum downgrade) did not change things. Executing yum downgrade util-linux libuuid libblkid libmount and restarting restored the NFS server functionality.

util-linux-2.22.2-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/util-linux-2.22.2-3.fc18

Package util-linux-2.22.2-3.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing util-linux-2.22.2-3.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-1934/util-linux-2.22.2-3.fc18
then log in and leave karma (feedback).

util-linux-2.22.2-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.