Bug 812826 - RFE: fstab: allow x-systemd-after= mount option to allow explicit ordering of fstab mounts
RFE: fstab: allow x-systemd-after= mount option to allow explicit ordering of...
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
:
: 881481 (view as bug list)
Depends On:
Blocks: systemd-RFE
  Show dependency treegraph
 
Reported: 2012-04-16 07:10 EDT by dagurasu
Modified: 2016-01-22 05:21 EST (History)
15 users (show)

See Also:
Fixed In Version: systemd-220-1.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1300997 (view as bug list)
Environment:
Last Closed: 2015-06-11 09:56:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description dagurasu 2012-04-16 07:10:14 EDT
Description of problem:

In fstab have a root filesystem, anda separate data disk/filesystem, bind mounts, and bindfs mounts listed in that order, with all the bind mounts and bindfs mounts relocating directories from the data disk.  This fstab worked fine in fedora 14. Now the bindfs mounts break startup. There are no systemd options and all fstab entries can be mounted with mount -a (works).  No custom systemd .mount files were added.  In fact it's a fresh upgrade from fedora 14 so nothing in systemd has been altered. 

When bootup fails, in the emergency terminal I see that the data disk is not mounted yet. Systemd documentation claims that mount fstab dependencies are calculated automatically.  The bind mounts (as opposed to bindfs) work fine.  I suspect that the generally different "device" field syntax of bindfs mounts, and especially fuse in general, breaks the dependency analysis.  If I comment out these mounts the system starts fine and I can un-comment and mount them after booting.  I'm sure I can engineer some dependency hack or .mount files to fix this for me.

Version-Release number of selected component (if applicable):
Fedora 16 fully updated about 2 days ago.

How reproducible:
always

Steps to Reproduce:
1. Use fstab described above which worked in Fedora 14
2. Boot up.
3. 
  
Actual results:
Boot fails when the bind-fs fails because the underlying file system is not mounted yet.

Expected results:
Boot succeeds with bindfs mounts delayed until after  the pre-requisite storage mount.

If systemd cannot determine the dependencies of an fstab entry it should fall back to the fstab standard and assume that anything listed above it should be mounted first.

Additional info:
Comment 1 Michal Schmidt 2012-04-16 07:26:28 EDT
Would you attach said fstab?
Comment 2 dagurasu 2012-04-17 01:32:32 EDT
It's a simple enough point but sure I'll make a simple test case for you.. here it is:

/dev/dm-1  /             ext4    defaults        0 1
/dev/sda1  /boot         ext4    defaults        0 2
/dev/sdb1  /data   ext3    defaults        0 2

/data               /test0    ext3  bind  0 0
/data/test1         /test1    ext3  bind  0 0
bindfs#/data/test2  /test2    fuse  rw    0 0


/dev/mapper/VolGroup-lv_swap swap      swap    defaults        0 0
tmpfs                      /dev/shm    tmpfs   defaults        0 0
devpts                     /dev/pts    devpts  gid=5,mode=620  0 0
sysfs                      /sys        sysfs   defaults        0 0
proc                       /proc       proc    defaults        0 0


I just booted this just now..  It boots fine with ONLY the bindfs line commented and it fails on /test with it un-commented.  Either way.. after it's un-commented (manually or at at the emergency prompt), mount -a works just fine (which I believe is what init.d/localmount used).

This test case is running in a toy vm so I can play with it easily.

Really this is 2 bugs IMO: 
1) mount order is wrong
2) failure behavior is wrong IMO, but I guess I don't remember how that worked before, because bug 1 didn't exist before but I thought mount failures didn't break bootup.  You remove a non-root esata drive and boot fails?  Sure there are about 4 or 5 ways around that probably.. but still, is that expected behavior?
Comment 3 dagurasu 2012-04-17 02:04:34 EDT
Simple workaround: add nofail to the bindfs mount.  It's hard to consider that hack a bug fix though, but for me now it's certainly a fine solution.  

It turns out that /test2 does actually get mounted on boot in that case, so it's a better fix than it might appear.  It must be that systemd makes multiple passes at mounting things.  Adding comment=system.d to the /data drive did not help.  

If only every bug could be avoided by typing "nofail".  :)
Comment 4 dagurasu 2012-04-17 02:08:02 EDT
quoting myself 2 posts up: "It boots fine with ONLY the bindfs line
commented and it fails on /test with it un-commented.  "  
      
... that should of course say  "... fails on /test2 ..."
Comment 5 Michal Schmidt 2012-04-23 12:06:53 EDT
(In reply to comment #2)
> bindfs#/data/test2  /test2    fuse  rw    0 0

It is true that systemd does not understand the bindfs syntax and won't infer the ordering dependency.

Parsing the bindfs syntax would not be general enough.
The only practicable fix that I can see is to add After dependencies to fuse mounts on all mounts that are listed above them in fstab.
Comment 6 dagurasu 2012-05-03 04:12:52 EDT
That is exactly the response I expected.

Of course.. if systemd can't understand an fstab entry it should honor the traditional meaning of fstab which is exactly the fix described.  You can't try to outsmart the user when you don't understand the syntax and it's not very nice breaking what used to work just because you wish there was a better way.

I get not wanting to attempt to support every major fusefs syntax too.

Can we call it confirmed then?
Comment 7 Lennart Poettering 2013-01-14 16:07:33 EST
We can't really automatically detect dependencies of these kinds, and simply adding deps between all mounts simply by the order they are listed in would effectively serialize the boot, which is certainly not desirable.

I am thinking of simply adding x-systemd-after=/foobar/waldo as a new mount option people can use to explciitly tell systemd about dependencies. That would work pretty nicely I guess.
Comment 8 Lennart Poettering 2013-01-14 16:12:09 EST
*** Bug 881481 has been marked as a duplicate of this bug. ***
Comment 9 Matthew Woehlke 2013-01-14 16:45:16 EST
Note: the suggestion that ended up happening in Bug 881481 was to have a general systemd dependency syntax, e.g. '_dep=foobar.mount'; IOW, taking systemd unit names rather than mount-points, which would allow the same system to handle this, supercede _netdev, and probably would be able to cover cases we haven't even thought of yet. (This plus _netdev would in theory suffice for Bug 881481, but I'd be in favor of the single general system covering both.)
Comment 10 Lennart Poettering 2013-01-15 18:27:08 EST
Well, the "_" namespace is mount's own. Systemd should extend the fstab syntax only in its own "x-systemd-" namespace. 

But yeah, there are three options:

x-systemd-after=<PATH>  -- the explicit path to wait for

x-systemd-after=<UNIT>  -- the systemd unit name to wait for

x-systemd-requires-mounts-for=<PATH>  -- referring to all mount unit that are necessary for this file to be around, possibly multiple. THis is a superset of the first option

The latter would work more or less like RequiresMountsFor= in unit files.

I think the third syntax is probably the most useful one here, and should solve the usecases at hand. Adding the second syntax later on, should the need arise should be OK, too.
Comment 11 Fedora End Of Life 2013-01-16 12:43:26 EST
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 12 Cesar Eduardo Barros 2013-01-20 09:47:58 EST
Does it need a x- prefix? Take a look at https://tools.ietf.org/html/rfc6648 (Deprecating the "X-" Prefix and Similar Constructs in Application Protocols).
Comment 13 dagurasu 2013-03-21 12:02:25 EDT
Lennart Poettering"We can't really automatically detect dependencies of these kinds, and simply adding deps between all mounts simply by the order they are listed in would effectively serialize the boot, which is certainly not desirable."

With respect, this isn't true.  Well the first part is:  You cannot automatically detect dependencies of fuse mounts (or potentially something else like them).  The second part though is not true.  You do not need to serialize the whole boot.

You do not need to make every mount depend on the mount before it (unless every mount is a fuse mount).  You only need to make every fuse mount depend on all the mounts before it (in no particular order).  That is VERY different from serializing the whole mount process.

If you have mounts 1 through 5 that are normal and 6 is a fuse mount and 7 through 20 are normal.  Then 1 through five can be parallelized as much as normal dependency rules allow, but they must all finish before 6 is performed.  Then 7 through 20 carry on.  That might not be ideal always but it's far from fully serial and it's the right thing to do if you don't know.

If there are no special options specified, this is exactly what should happen because that is the standard defined by fstab and will cause the least breakage as more people(distributions) adopt systemd. Special options to allow the user to provide hints to improve on that situation from there are great, but should not be required to prevent breakage.
Comment 14 dagurasu 2013-03-21 12:25:33 EDT
To keep this in perspective, most people probably aren't using bindfs mounts in a customized fstab. The small amount of serialization constraint I describe is a loss of a few seconds at most for a small but not tiny group of people.  By implementing some ability to add hints, it's a fixable loss of a few seconds for a small group of people.  

However, requiring the hints to make it work is instead going to result in boot failure for probably a high fraction of that small but not tiny group.  Some fraction of those won't ever figure out why before moving on.  Some fraction just won't have time to figure out why this morning and will break their toe cursing about it and then limp through the day without their computer.
Comment 15 Fedora End Of Life 2013-12-21 03:35:05 EST
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 16 James DeFelice 2014-08-02 18:41:46 EDT
I'm trying to use a fuse mount (funionfs) that "stacks" several NFS-mounted directories. The fuse mounts are broken upon a fresh boot since systemd mounts them first, and then mounts the NFS ones. 

I've added noauto and comment=systemd.automount to both the NFS and fuse fstab entries, attempting to delay the fuse mounts until after the NFS mounts are already up. It's no good -- the fuse mounts always come up before the NFS mounts.

I'm running Fedora 20. Is there any hope for getting this fixed in that distro, or do I have to wait (for F21, perhaps)??
Comment 17 Michal Schmidt 2014-08-11 05:23:48 EDT
Until a fstab option is implemented I can suggest a workaround - add the missing ordering dependencies using drop-in unit files.

For example, if "/test" needs to be mounted after "/data", you can express that using a two-line file like this:

# cat /etc/systemd/system/test.mount.d/ordering.conf
[Unit]
After=data.mount
Comment 18 Karel Zak 2015-05-12 08:15:38 EDT
I have sent a patch with this to systemd-devel. We will see...

 [PATCH] fstab-generator: add x-systemd.{after,requires-mounts-for}=
Comment 19 Lennart Poettering 2015-05-18 10:07:44 EDT
This is implemented now upstream. (though under the name x-systemd.requires=).

Note You need to log in before you can comment on or make changes to this bug.