Bug 889888 - wipefs does not completely wipe btrfs volume
wipefs does not completely wipe btrfs volume
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: btrfs-progs (Show other bugs)
18
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Josef Bacik
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-12-23 15:54 EST by Chris Murphy
Modified: 2014-08-13 21:59 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-07 21:21:25 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
first5MB.bin.bz2 (76.50 KB, application/bzip2)
2012-12-23 15:56 EST, Chris Murphy
no flags Details
anaconda.log (28.64 KB, text/plain)
2012-12-26 11:17 EST, Steve Tyler
no flags Details
program.log (105.09 KB, text/plain)
2012-12-26 11:18 EST, Steve Tyler
no flags Details
storage.log (287.67 KB, text/plain)
2012-12-26 11:19 EST, Steve Tyler
no flags Details

  None (edit)
Description Chris Murphy 2012-12-23 15:54:32 EST
Description of problem:
A partition formatted btrfs, is wiped using wipefs -a, but btrfs fi show still sees the partition as a member of a btrfs volume.

Version-Release number of selected component (if applicable):
util-linux-2.22.1-2.1.fc18.x86_64

How reproducible, and steps to reproduce:
100% with a particular device, but exact steps to reproduce needs regression.

1. Device sda single partition sda1 is one of six devices in a btrfs volume.
2. wipefs -a /dev/sda1, id'd as btrfs and wiped
3. wipefs -a /dev/sda, id's as MBR and wiped
4. Install F18 with anaconda-18.37.8-1 using "Guided" autopartition, scheme Btrfs.
This creates layout:
sda1 = ext4
sda2 = swap
sda3 = btrfs


Actual results:
parted -l sees sda1 as ext4
mount sees sda1 as ext4
yet btrfs fi show yields:

Label: 'fedora_f18v'  uuid: 7a8f1bd8-ba7c-4fd2-a0a7-87b43e6ca0a6
	Total devices 6 FS bytes used 3.08GB
	devid    1 size 7.97GB used 1.02GB path /dev/sda1
	*** Some devices missing


Expected results:
I don't expect btrfs to see sda1 as a btrfs device.


Additional info:
sda is the only physical device attached. The other member disks are not attached, so it's not like btrfs fi show is getting the uuid, and idea that sda1 is a member of a btrfs volume from some other volume. It must be getting this from sda.
Comment 1 Chris Murphy 2012-12-23 15:56:17 EST
Created attachment 668141 [details]
first5MB.bin.bz2

First 5MB of /dev/sda.
Comment 2 Chris Murphy 2012-12-23 17:45:20 EST
So I've gone through a bunch of permutations that don't involved repartitioning, and can't reproduce this. I wonder if wipefs isn't wiping something that in the course of a subsequent repartition, then allows btrfs fi show to see remnants of a previous btrfs file system.
Comment 3 Steve Tyler 2012-12-24 13:24:37 EST
(In reply to comment #0)
...
> 1. Device sda single partition sda1 is one of six devices in a btrfs volume.
...

How did you create the initial disk state?
Comment 4 Steve Tyler 2012-12-24 15:01:08 EST
(In reply to comment #0)
...
> 3. wipefs -a /dev/sda, id's as MBR and wiped
...

Immediately after rebooting from the Live CD, I get an error trying to run "wipefs -a /dev/sda":

[root@localhost liveuser]# wipefs -n /dev/sda
offset               type
----------------------------------------------------------------
0x1fe                dos   [partition table]

[root@localhost liveuser]# wipefs -a /dev/sda
wipefs: error: /dev/sda: probing initialization failed: Device or resource busy
[root@localhost liveuser]# rpm -q util-linux
util-linux-2.22.1-2.1.fc18.x86_64

Tested by booting from the Live CD with:
$ qemu-kvm -m 2048 -hda f18-test-3.img -cdrom ~/xfr/fedora/F18/F18-Final/TC3/Fedora-18-TC3-x86_64-Live-Desktop.iso -usb -vga qxl -boot menu=on -usbdevice mouse
Comment 5 Chris Murphy 2012-12-24 15:44:05 EST
(In reply to comment #3)
> How did you create the initial disk state?

I'm not certain. The virtual disk had been used and reused. The most recent state was produced by specifying it and five other virtual disks, in Installation Destination, then in Manual Partitioning making a single Root mountpount, Btrfs.
Comment 6 Steve Tyler 2012-12-26 10:53:45 EST
Thanks. Here's a reproducer using the installer:

Create an empty 12 GiB disk image:
$ qemu-img create f18-test-3.img 12G

Start the Live CD and update the installer:
$ qemu-kvm -m 2048 -hda f18-test-3.img -cdrom ~/xfr/fedora/F18/F18-Final/TC3/Fedora-18-TC3-x86_64-Live-Desktop.iso -usb -vga qxl -boot menu=on -usbdevice mouse
anaconda-18.37.8-1.fc18

Start the installer:
Installation Options:
    Partition scheme configuration: BTRFS
    Check "Let me customize ..."
Manual Partitioning:
    Click "+" to add a new mount point.
    Mount Point: /
    Desired Capacity: [blank]
Complete installation.
Quit installer.

From a terminal:
# btrfs fi show
# lsblk -f /dev/sda

Start the installer:
Remove all mount points and partitions (There should be only one.).
Click to auto-configure storage.
Complete installation.
Quit installer.

From a terminal:
# btrfs fi show
# lsblk -f /dev/sda
Comment 7 Steve Tyler 2012-12-26 11:07:03 EST
(In reply to comment #6)
> Thanks. Here's a reproducer using the installer:
...

Here are the results after each install:

$ cat reproducer-results-1.txt 
[root@localhost liveuser]# btrfs fi show
Label: 'fedora'  uuid: ccca0d26-0dda-4806-bf6d-8c62ef875c25
	Total devices 1 FS bytes used 3.08GB
	devid    1 size 12.00GB used 6.04GB path /dev/sda1

Btrfs Btrfs v0.19
[root@localhost liveuser]# lsblk -f /dev/sda
NAME   FSTYPE LABEL  UUID                                 MOUNTPOINT
sda                                                       
└─sda1 btrfs  fedora ccca0d26-0dda-4806-bf6d-8c62ef875c25 /mnt/sysimage
[root@localhost liveuser]# # < reinstall here >
[root@localhost liveuser]# btrfs fi show
Label: 'fedora'  uuid: ccca0d26-0dda-4806-bf6d-8c62ef875c25
	Total devices 1 FS bytes used 1.04MB
	devid    1 size 12.00GB used 6.04GB path /dev/sda1

Btrfs Btrfs v0.19
[root@localhost liveuser]# lsblk -f /dev/sda
NAME                   FSTYPE      LABEL UUID                                   MOUNTPOINT
sda                                                                             
├─sda1                 ext4              5a12e6f2-2857-48e8-a29b-5437b81630e7   /mnt/sysimage/boot
└─sda2                 LVM2_member       AZlxMb-ZGn4-zO2v-ItmL-yaHL-n0TQ-cZnVG3 
  ├─fedora-swap (dm-2) swap              f1edee57-e53f-4632-bfe9-f152c3cdd346   [SWAP]
  └─fedora-root (dm-3) ext4              5a783d1f-8267-4e7a-abdb-a9dd5fcef3aa   /mnt/sysimage
[root@localhost liveuser]# rpm -q anaconda btrfs-progs util-linux
anaconda-18.37.8-1.fc18.x86_64
btrfs-progs-0.20.rc1.20121017git91d9eec-1.fc18.x86_64
util-linux-2.22.1-2.1.fc18.x86_64
[root@localhost liveuser]#
Comment 8 Steve Tyler 2012-12-26 11:17:52 EST
Created attachment 669257 [details]
anaconda.log
Comment 9 Steve Tyler 2012-12-26 11:18:32 EST
Created attachment 669258 [details]
program.log
Comment 10 Steve Tyler 2012-12-26 11:19:10 EST
Created attachment 669259 [details]
storage.log
Comment 11 Steve Tyler 2012-12-26 11:35:30 EST
(In reply to comment #4)
...
> Immediately after rebooting from the Live CD, I get an error trying to run
> "wipefs -a /dev/sda":
...
> [root@localhost liveuser]# wipefs -a /dev/sda
> wipefs: error: /dev/sda: probing initialization failed: Device or resource
> busy
...

The sequence needs to be:
# swapoff -a
# vgchange -a n fedora
# wipefs -a /dev/sda
Comment 12 Steve Tyler 2012-12-26 13:41:00 EST
The attached program.log has records for both installs:

[snippets from attached program.log]
...
10:13:39,902 INFO program: Running... swapoff -a
10:13:40,002 INFO program: Running... wipefs -a /dev/sda
...
10:13:40,675 INFO program: Running... wipefs -a /dev/sda1
...
10:13:41,016 INFO program: Running... mkfs.btrfs --data=single --label=fedora /dev/sda1
...

...
10:33:26,101 INFO program: Running... swapoff -a
...
15:33:26,479 INFO program: Running... /bin/mount -n -t btrfs -o defaults /dev/sda1 /tmp/btrfs-tmp.2ILsJ2v
10:33:26,567 INFO program: Running... btrfs subvol delete /tmp/btrfs-tmp.2ILsJ2v/root
10:33:26,593 INFO program: Delete subvolume '/tmp/btrfs-tmp.2ILsJ2v/root'
15:33:26,603 INFO program: Running... /bin/umount /tmp/btrfs-tmp.2ILsJ2v
...
10:33:35,316 INFO program: Running... wipefs -a /dev/sda1
10:33:35,381 INFO program: /dev/sda1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
...
10:33:36,259 INFO program: Running... wipefs -a /dev/sda1
...
10:33:36,451 INFO program: Running... mke2fs -t ext4 /dev/sda1
...
Comment 13 Steve Tyler 2012-12-26 17:45:02 EST
(In reply to comment #12)
...
> 10:13:41,016 INFO program: Running... mkfs.btrfs --data=single
> --label=fedora /dev/sda1
> ...

I omitted the records for subvolume creation:

...
15:13:41,939 INFO program: Running... /bin/mount -n -t btrfs -o defaults /dev/sda1 /tmp/btrfs-tmp.15Xk4f7R
10:13:42,004 INFO program: Running... btrfs subvol create /tmp/btrfs-tmp.15Xk4f7R/root
10:13:42,177 INFO program: Create subvolume '/tmp/btrfs-tmp.15Xk4f7R/root'
15:13:42,189 INFO program: Running... /bin/umount /tmp/btrfs-tmp.15Xk4f7R
...
Comment 14 Chris Murphy 2013-01-05 17:33:16 EST
I've got another case of this, and it's always multiple disk Btrfs volumes.

# btrfs fi show
Label: 'fedora_f18v'  uuid: c98ee018-6122-4541-aa71-80705b4ecab7
	Total devices 2 FS bytes used 3.03GB
	devid    2 size 80.00GB used 6.01GB path /dev/sdb1
	devid    1 size 75.57GB used 6.03GB path /dev/sda2

Label: 'fedora_f18v'  uuid: ddce17dd-037a-474e-81cf-3b4dc9197206
	Total devices 2 FS bytes used 3.08GB
	devid    1 size 80.00GB used 6.03GB path /dev/sda1
	*** Some devices missing

Btrfs Btrfs v0.19


# lsblk -f /dev/sda
NAME   FSTYPE LABEL       UUID                                 MOUNTPOINT
sda                                                            
├─sda1 ext4               6a2f4d98-7e57-4885-96ea-341dd6146fe9 /boot
├─sda2 btrfs  fedora_f18v c98ee018-6122-4541-aa71-80705b4ecab7 /home
└─sda3 swap               953ceb5c-5ebc-4658-abc7-98ab9a1066fb [SWAP]


The 2nd btrfs fi show result uuid: ddce... is bogus. That's the previous btrfs volume, which would have been a two disk, single partition only (on each disk), Btrfs volume, -d single -m raid1.

Unclear to me still if this is a wipefs bug, or a btrfs fi show bug. Or maybe both.
Comment 15 Chris Murphy 2013-01-05 18:16:04 EST
OK this is interesting. Brand new 80GB virtual disk, so it's completely zero'd.

1. fdisk to create one partition, all defaults.
2. Format that sda1 with mkfs.btrfs -L first.
3. btrfs fi show displays "first" labeled volume.
4. wipefs -a /dev/sda1 and it finds btrfs metadata, ostensibly removes it.
5. btrfs fi show does NOT show "first" anymore.
6. fdisk /dev/sda again, delete the one partition; create a new partition 500MB and a second one for remaining space.
7. btrfs fi show still does not show "first".
8. mkfs.ext4 /dev/sda1
9. btrfs fi show NOW SHOWS "first" on sda1!
10. Reboot
11. btrfs fi show still shows "first" on sda1, size 80GB.

Tentative conclusions:

A. wipefs isn't deleting all btrfs metadata it could. 

B. mkfs.ext4 is refreshing stale btrfs metadata in a way that causes btrfs fi show to see.

C. btrfs fi show seems to blatantly disregard the partition map, because it shows a volume the partition map cannot possibly validly support.

D. Nothing else sees this phantom btrfs volume. btrfs fi show seems confused.


btrfs-progs-0.20.rc1.20121017git92d9eec-1.fc18.x86_64
e2fs-progs-1.42.5-1.fc18.x86_64
kernel 3.7.1-2
Comment 16 Chris Murphy 2013-01-05 19:12:27 EST
Even easier to reproduce:

1. Partition of any size.
2. mkfs.btrfs /dev/sda1
3. wipefs -a /dev/sda1
4. btrfs fi show doesn't find it.
5. mkfs.ext4 /dev/sda1
6. btrfs fi show sees the stale volume.

Repartitioning has nothing to do with it. Happens with all sized partitions I've tested from 500MB to 2TB.
Comment 17 Steve Tyler 2013-01-06 00:30:54 EST
Nice work.

There appear to be redundant copies of the signature, "_BHRfS_M, in the partition. Wipefs zeroes the first one, but there are more further down in the partition. The label and the UUID are also replicated.

# mkfs.btrfs -L foo1 /dev/sda1
# dd if=/dev/sda | strings | cat -n
# wipefs -a /dev/sda1
# dd if=/dev/sda | strings | cat -n
Comment 18 Chris Murphy 2013-01-06 00:41:22 EST
Yeah even a dd if=/dev/zero for 50MB doesn't clear it. Maybe Btrfs has some magical properties after all, that allow it to tolerate some data recovery in the event it's been wipefs'd and formatted by accident.
Comment 19 Steve Tyler 2013-01-06 01:01:08 EST
The btrfs Wiki lists "fault tolerance, repair and easy administration" as objectives:
https://btrfs.wiki.kernel.org/index.php/Main_Page

The FAQ says to use wipefs "To actually remove the filesystem":

How to clean up old superblock ? 
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_to_clean_up_old_superblock_.3F

That is followed by this advice:

What if I don't have wipefs at hand?
# dd if=/dev/zero of=/dev/sda bs=1k count=4 seek=64

https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#What_if_I_don.27t_have_wipefs_at_hand.3F
Comment 20 Chris Murphy 2013-01-06 01:12:48 EST
The wiki is wrong. For one, if you try it, it doesn't work, btrfs fi show still sees the volume. Second, it's only 4096 bytes, and Btrfs has a 64KB pad before the beginning of the file system anyway. That pad, if it doesn't have grub in it, is all zeros anyway. So this dd command in the wiki does exactly nothing.

Anyway, it's increasingly clear to me it's a bug in btrfs-progs. *shrug*
Comment 21 Steve Tyler 2013-01-06 01:22:38 EST
The dd command does a seek:
# dd if=/dev/zero of=/dev/sda bs=1k count=4 seek=64
                                            ^^^^^^^

Anyway, I agree that this should go to btrfs-progs. wipefs is doing exactly what the wiki says to do.
Comment 22 Chris Murphy 2013-01-06 01:45:36 EST
OH right, well funny enough the dd command given, btrfs fi show still displays it, I don't even have to do resurrection with mkfs.ext4.

At least with wipefs, it does go away. So it seems btrfs fi show is just being extremely optimistic. Changing to btrfs-progs.
Comment 23 Goffredo Baroncelli 2013-01-06 11:30:06 EST
Btrfs stores several copies of the superblock (IIRC three). During the
scan it searches for a valid superblock. If the first is wrong, checks
the second and so on...
It stops the search when one of the following conditions is true:
1) the superblock "magic" field is zero (typically when a disk is removed)
2) the superblock is valid
3) there is no other superblock.

I think that wipefs zeros the "magic" field on the first superblock. So
the btrfs scan process ends at the first superblock.

When the disk is used again, it is possible that the first superblock is
overwritten and the "magic" field is not zero any more. So in case of
further scan by btrfs, the first superblock is ignored and the check is
performed to the second one and the third one, which could be valid, if
nobody zeroed its "magic" field.
Comment 24 Steve Tyler 2013-01-06 16:53:23 EST
(In reply to comment #19)
...
> What if I don't have wipefs at hand?
> # dd if=/dev/zero of=/dev/sda bs=1k count=4 seek=64
> 
> https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#What_if_I_don.
> 27t_have_wipefs_at_hand.3F
...

This seems incorrect. The offset should be relative to the beginning of the _partition_, not the disk:
# dd if=/dev/zero of=/dev/sda1 bs=1k count=4 seek=64
.............................^

NB: The wiki has been updated, but it still uses /dev/sda.
https://btrfs.wiki.kernel.org/index.php?title=Problem_FAQ&oldid=16201
Comment 25 Steve Tyler 2013-01-06 17:24:34 EST
Correcting myself. The file system can span the whole disk. The FAQ section on clearing the file system should note that the file system can be created on a disk or in a partition, so the device should be /dev/sda or /dev/sdaN, as appropriate.
Comment 26 Chris Murphy 2013-01-06 18:02:53 EST
Looks like there are three superblocks, maybe not including the mirrored copies (?). In any case, discussion on linux-btrfs list here:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg20947.html
patch for wipefs here:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg20959.html

Maybe the bug needs to be set back to util-linux.
Comment 27 Steve Tyler 2013-01-06 22:00:26 EST
Thanks, Chris and Goffredo.

Patching wipefs might fix the problem, but why isn't a missing superblock reported as a file system inconsistency? Couldn't data in a different file system mimic a btrfs superblock?
Comment 28 Karel Zak 2013-01-08 13:05:34 EST
(In reply to comment #27)
> Patching wipefs might fix the problem,

I'm going to add support for the backup superblocks to libblkid.

> but why isn't a missing superblock
> reported as a file system inconsistency?

This is good question. I agree that the current btrfs behaviour is strange, the filesystem (utils) should be more paranoid...
Comment 29 Fedora Update System 2013-01-11 04:54:58 EST
util-linux-2.22.2-2.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/util-linux-2.22.2-2.fc18
Comment 30 Andreas Kohn 2013-01-24 13:16:45 EST
While it barely makes sense to me, it seems that after updating util-linux, libblkid, libmount and libuuid from 2.22.1-2.4.fc18 to 2.22.2-2 my fedora is no longer able to behave as a NFS server. 

Mount attempts by clients (local, or various remote debian, ubuntu, fedora and freebsd clients) produce ESTALE return codes from mount(2) with NFS3, and "cannot allocate memory" with NFS4.

In the same upgrade session I also received and nfs-utils 1.2.6-14 to 1.2.7-3 update, but just downgrading that (yum downgrade) did not change things. Executing yum downgrade util-linux libuuid libblkid libmount and restarting restored the NFS server functionality.
Comment 31 Fedora Update System 2013-02-03 11:16:23 EST
util-linux-2.22.2-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/util-linux-2.22.2-3.fc18
Comment 32 Fedora Update System 2013-02-04 22:09:50 EST
Package util-linux-2.22.2-3.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing util-linux-2.22.2-3.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-1934/util-linux-2.22.2-3.fc18
then log in and leave karma (feedback).
Comment 33 Fedora Update System 2013-02-07 21:21:28 EST
util-linux-2.22.2-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.