Bug 1306808 - blivet.errors.FSError: mount failed: 32
blivet.errors.FSError: mount failed: 32
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: python-blivet (Show other bugs)
24
x86_64 Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Vojtech Trefny
Fedora Extras Quality Assurance
AcceptedBlocker abrt_hash:0081a77191b...
:
: 1253481 (view as bug list)
Depends On:
Blocks: F24BetaBlocker
  Show dependency treegraph
 
Reported: 2016-02-11 13:31 EST by Chris Murphy
Modified: 2016-05-17 22:45 EDT (History)
18 users (show)

See Also:
Fixed In Version: python-blivet-1.20.0-1 python-blivet-1.20.0-1.fc24
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-23 19:46:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: anaconda-tb (810.69 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: anaconda.log (109.89 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: environ (494 bytes, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: journalctl (328.23 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: lsblk_output (2.49 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: nmcli_dev_list (2.22 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: os_info (528 bytes, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: program.log (71.81 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: storage.log (192.03 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
File: ifcfg.log (4.23 KB, text/plain)
2016-02-11 13:31 EST, Chris Murphy
no flags Details
patch to filter monut options (from Ivan Sizov) (660 bytes, patch)
2016-04-08 14:35 EDT, Adam Williamson
no flags Details | Diff

  None (edit)
Description Chris Murphy 2016-02-11 13:31:13 EST
Description of problem:
Crashes after clicking begin installation.

Version-Release number of selected component:
anaconda-core-24.11-1.fc24.x86_64

The following was filed automatically by anaconda:
anaconda 24.11-1 exception report
Traceback (most recent call first):
  File "/usr/lib/python3.5/site-packages/blivet/tasks/fsmount.py", line 123, in doTask
    raise FSError("mount failed: %s" % rc)
  File "/usr/lib/python3.5/site-packages/blivet/formats/fs.py", line 645, in _setup
    self._mount.doTask(chrootedMountpoint, options=options)
  File "/usr/lib/python3.5/site-packages/blivet/formats/__init__.py", line 483, in setup
    self._setup(**kwargs)
  File "/usr/lib/python3.5/site-packages/blivet/osinstall.py", line 632, in mountFilesystems
    chroot=rootPath)
  File "/usr/lib/python3.5/site-packages/blivet/blivet.py", line 1407, in mountFilesystems
    readOnly=readOnly, skipRoot=skipRoot)
  File "/usr/lib/python3.5/site-packages/blivet/osinstall.py", line 1105, in turnOnFilesystems
    storage.mountFilesystems()
  File "/usr/lib64/python3.5/site-packages/pyanaconda/install.py", line 195, in doInstall
    turnOnFilesystems(storage, mountOnly=flags.flags.dirInstall, callbacks=callbacks_reg)
  File "/usr/lib64/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.5/site-packages/pyanaconda/threads.py", line 253, in run
    threading.Thread.run(self, *args, **kwargs)
blivet.errors.FSError: mount failed: 32

Additional info:
cmdline:        /usr/bin/python3  /sbin/anaconda --liveinst --method=livecd:///dev/mapper/live-base
cmdline_file:   BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Fedora-Live-WS-x86_64-rawhide-20 rootfstype=auto ro rd.live.image rd.luks=0 rd.md=0 rd.dm=0 elevator=noop
executable:     /sbin/anaconda
hashmarkername: anaconda
kernel:         4.5.0-0.rc2.git3.1.fc24.x86_64
other involved packages: python3-libs-3.5.1-3.fc24.x86_64, python3-blivet-1.18-1.fc24.noarch
product:        Fedora
release:        Fedora release 24 (Rawhide)
type:           anaconda
version:        rawhide

Potential duplicate: bug 1253481
Comment 1 Chris Murphy 2016-02-11 13:31:19 EST
Created attachment 1123268 [details]
File: anaconda-tb
Comment 2 Chris Murphy 2016-02-11 13:31:20 EST
Created attachment 1123269 [details]
File: anaconda.log
Comment 3 Chris Murphy 2016-02-11 13:31:21 EST
Created attachment 1123270 [details]
File: environ
Comment 4 Chris Murphy 2016-02-11 13:31:23 EST
Created attachment 1123271 [details]
File: journalctl
Comment 5 Chris Murphy 2016-02-11 13:31:24 EST
Created attachment 1123272 [details]
File: lsblk_output
Comment 6 Chris Murphy 2016-02-11 13:31:25 EST
Created attachment 1123273 [details]
File: nmcli_dev_list
Comment 7 Chris Murphy 2016-02-11 13:31:26 EST
Created attachment 1123274 [details]
File: os_info
Comment 8 Chris Murphy 2016-02-11 13:31:27 EST
Created attachment 1123275 [details]
File: program.log
Comment 9 Chris Murphy 2016-02-11 13:31:28 EST
Created attachment 1123276 [details]
File: storage.log
Comment 10 Chris Murphy 2016-02-11 13:31:29 EST
Created attachment 1123277 [details]
File: ifcfg.log
Comment 11 Chris Murphy 2016-02-11 13:38:49 EST
11:29:08,880 INFO program: Running... mount -t btrfs -o subvolid=5,subvol=root /dev/sda3 /mnt/sysimage

It shouldn't use both subvolid and subvol mount options at the same time.

Feb 11 13:29:08 localhost kernel: BTRFS: subvol 'root' does not match subvolid 5

And also the logic is wrong because it's mounting two different subvolumes.
Comment 12 Chris Murphy 2016-02-13 16:28:45 EST
Probable dup, bug 1253481.
Comment 13 Fedora Blocker Bugs Application 2016-02-15 02:37:00 EST
Proposed as a Blocker for 24-beta by Fedora user chrismurphy using the blocker tracking app because:

 "When using the custom partitioning flow, the installer must be able to: Create mount points backed... btrfs volumes. Correctly interpret, and modify .... btrfs volumes."

The direct problem is that anaconda is using an invalid mount command, which fails and then the installer crashes. However it doesn't always use the wrong mount command, it seems to only happen with a totally empty Btrfs volume, e.g. one that's been formatted outside of the installer (even when using mkfs defaults, same as the installer). If it contains subvolumes, such as from previously installed systems, the problem doesn't happen.

Looks like it snuck through testing in Fedora 23, and users are hitting it.
Comment 14 Petr Schindler 2016-02-15 13:04:00 EST
Discussed at 2016-02-15 blocker review meeting: [1]. 

We decided to delay the decision: this seems a strong candidate as there are many people hitting the 'potential dupe' bug, but reporter did not explain precisely what he actually did, we need that clarified to make a decision

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2016-02-15/f24-blocker-review.2016-02-15-17.00.html
Comment 15 Chris Murphy 2016-02-15 14:11:17 EST
There's more than one way to hit this but a simple version is:
1. New qcow2
2. fdisk to create a 500M and "remainder" partitions; 2 partitions.
3. mkfs.btrfs /dev/vda2  ## does not matter what the options are, including defaults
4. Custom partitioning
5. Change partition scheme popup to btrfs
6. Click under Unknown, click sda1, check reformat, select ext4, mount point set to /boot
7. + to add mount point for / and optionally anything else.
Comment 16 Chris Murphy 2016-02-15 20:49:37 EST
Oops.
8. Click Done, Done (due to no swap in the example), Begin Installation.
Crash.

Two clues for what's going on:
1. anaconda-blivet have used 'subvolid=5,subvol=root' mount option when starting with empty btrfs filesystems the installer did not create, for some time. It goes back to at least Fedora 21. 
2. What's changed is since kernel 4.2, conflicts in those options aren't permitted anymore. [1] The mount failure aggravates the installer into crashing.

But I'm uncertain why installer created fs causes subvolid=5 to be supressed. Maybe the reuse empty fs is sufficiently rare that it's just a stale code path.

This is loosely similar to bug 1198437 where fstab btrfs entries always had both mount options, which were also in conflict in the same way as in this bug. Fortunately that bug was fixed, or users upgrading kernels would have had boot failures.

[1]
https://lkml.org/lkml/2015/4/9/908

This includes the kernel patch and discussion on the change. Gist is subvolid and subvol together are OK if they point to the same subvolume. But if they conflict, better to fail to mount because what the user expects is ambiguous. Using subvolid=5, in particular, always fails with subvol=<name> because id=5 points to the unnamed default subvolume created at mkfs time that also can't be deleted.
Comment 17 Michael DePaulo 2016-02-21 09:52:24 EST
Similar problem has been detected:

I tried to install rawhide (F24 2016-02-20) to subvolumes under an existing F23 Btrfs partition.

For example, F23's opt subvolume is "opt", while rawhide's opt subvolume was supposed to be "opt00". The "00" suffix was added automatically by Anaconda.

Other details:
UEFI mode
Reusing the F23 ext4 /home LV.
Shrunk the Windows NTFS partition in order to create a new /boot partition.

addons:         com_redhat_kdump
cmdline:        /usr/bin/python3  /sbin/anaconda
cmdline_file:   BOOT_IMAGE=/images/pxeboot/vmlinuz inst.stage2=hd:LABEL=Fedora-rawhide-x86_64 quiet
dnf.rpm.log:    Feb 21 09:11:13 INFO --- logging initialized ---
hashmarkername: anaconda
kernel:         4.5.0-0.rc4.git3.1.fc24.x86_64
package:        anaconda-24.13-1
product:        Fedora
reason:         blivet.errors.FSError: mount failed: 32
release:        Cannot get release name.
version:        rawhide
Comment 18 Chris Murphy 2016-02-21 16:05:13 EST
OK anytime I reuse an existing Btrfs volume with either F23 or F24 installers, I hit this. Only if the installer creates the fs does it NOT try to mount with subvolid=5, and thus the problem doesn't happen.
Comment 19 Petr Schindler 2016-02-22 14:23:45 EST
Discussed at 2016-02-22 blocker review meeting: [1]. 

This bug was accepted as Beta blocker: this seems to be a clear violation of "When using the custom partitioning flow, the installer must be able to: ... Assign mount points to existing storage volumes" , where an earlier section of the criterion makes clear that btrfs "storage volumes" are in the set expected to be supported

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2016-02-22/f24-blocker-review.2016-02-22-17.00.html
Comment 20 Jan Kurik 2016-02-24 09:30:25 EST
This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase
Comment 21 Reuben W. Perelman 2016-04-01 15:34:59 EDT
Similar problem has been detected:

Clicked "begin installation", and the error message came up soon after starting the installation.

cmdline:        /usr/bin/python3  /sbin/anaconda --liveinst --method=livecd:///dev/mapper/live-base
cmdline_file:   BOOT_IMAGE=/images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-24_A-7 rd.live.image rd.live.check quiet
hashmarkername: anaconda
kernel:         4.5.0-0.rc7.git0.2.fc24.x86_64
other involved packages: python3-blivet-1.19-1.fc24.noarch, system-python-libs-3.5.1-7.fc24.x86_64
package:        anaconda-core-24.13-1.fc24.x86_64
packaging.log:  
product:        Fedora
reason:         blivet.errors.FSError: mount failed: 32
release:        Fedora release 24 (Twenty Four)
version:        24
Comment 22 Reuben W. Perelman 2016-04-01 15:41:05 EDT
Was using custom partitioning. Created subvolumes for / and /home on existing btrfs partition /dev/sda2, /boot on new partition /dev/sda4, and /boot/efi on existing partition /dev/sda1.
Comment 23 Adam Williamson 2016-04-08 14:33:01 EDT
*** Bug 1253481 has been marked as a duplicate of this bug. ***
Comment 24 Adam Williamson 2016-04-08 14:35 EDT
Created attachment 1145233 [details]
patch to filter monut options (from Ivan Sizov)

Ivan Sizov posted this patch in the dupe, I'm re-attaching it here for convenience. All credit to him.
Comment 25 Adam Williamson 2016-04-08 14:39:09 EDT
Mmmmmm, monuts - the famous cross between the madeleine and the donut...
Comment 26 Vojtech Trefny 2016-04-15 05:38:48 EDT
Updates.img for Fedora 24: https://vtrefny.fedorapeople.org/img/subvol.img
Comment 27 MarkRuff 2016-04-17 00:21:50 EDT
Another user experienced a similar problem:

Trying to install Fedora... crash after selecting "Begin Installation"
(same happened when trying to install Fedora 24 alpha, so I'm back to Fedora 23... same issue now?!)

cmdline:        /usr/bin/python3  /sbin/anaconda --liveinst --method=livecd:///dev/mapper/live-base
cmdline_file:   BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Fedora-Live-WS-x86_64-23-10 rootfstype=auto ro rd.live.image quiet  rhgb rd.luks=0 rd.md=0 rd.dm=0 
hashmarkername: anaconda
kernel:         4.2.3-300.fc23.x86_64
other involved packages: python3-blivet-1.12.8-1.fc23.noarch, python3-libs-3.4.3-5.fc23.x86_64
package:        anaconda-core-23.19.10-1.fc23.x86_64
packaging.log:  
product:        Fedora
reason:         blivet.errors.FSError: mount failed: 32
release:        Fedora release 23 (Twenty Three)
version:        23
Comment 28 Chris Murphy 2016-04-17 18:30:10 EDT
Problem doesn't happen with Fedora-Workstation-Live-x86_64-24-20160411.n.1.iso + https://vtrefny.fedorapeople.org/img/subvol.img applied.
Comment 29 Fedora Update System 2016-04-18 16:57:11 EDT
python-blivet-1.20.0-1.fc24 anaconda-24.13.4-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-45ca29d07c
Comment 30 Fedora Update System 2016-04-19 16:25:44 EDT
anaconda-24.13.4-1.fc24, python-blivet-1.20.0-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-45ca29d07c
Comment 31 Fedora Update System 2016-04-23 19:45:44 EDT
anaconda-24.13.4-1.fc24, python-blivet-1.20.0-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Comment 32 S P Arif Sahari Wibowo 2016-05-16 09:26:19 EDT
Another user experienced a similar problem:

Installing in existing LVM-partitioned disk on some btrfs volume

cmdline:        /usr/bin/python3  /usr/sbin/anaconda --kickstart /root/2016e1,ks.cfg
cmdline_file:   BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Fedora-Live-Xfce-x86_64-23-10 rootfstype=auto ro rd.live.image quiet  rhgb rd.luks=0 rd.md=0 rd.dm=0 
dnf.rpm.log:    May 16 08:22:52 INFO --- logging initialized ---
hashmarkername: anaconda
kernel:         4.2.3-300.fc23.x86_64
other involved packages: python3-libs-3.4.3-5.fc23.x86_64, python3-blivet-1.12.8-1.fc23.noarch
package:        anaconda-core-23.19.10-1.fc23.x86_64
product:        Fedora
reason:         blivet.errors.FSError: mount failed: 32
release:        Fedora release 23 (Twenty Three)
version:        23
Comment 33 S P Arif Sahari Wibowo 2016-05-17 22:45:03 EDT
Another user experienced a similar problem:

Installing Fedora 23 on multiple btrfs-formatted logical volumes.

cmdline:        /usr/bin/python3  /usr/sbin/anaconda --kickstart /root/2016e1,ks.cfg
cmdline_file:   BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Fedora-Live-Xfce-x86_64-23-10 rootfstype=auto ro rd.live.image quiet  rhgb rd.luks=0 rd.md=0 rd.dm=0 
hashmarkername: anaconda
kernel:         4.2.3-300.fc23.x86_64
other involved packages: python3-libs-3.4.3-5.fc23.x86_64, python3-blivet-1.12.8-1.fc23.noarch
package:        anaconda-core-23.19.10-1.fc23.x86_64
product:        Fedora
reason:         blivet.errors.FSError: mount failed: 32
release:        Fedora release 23 (Twenty Three)
version:        23

Note You need to log in before you can comment on or make changes to this bug.