Description of problem:
If you in your fstab have a bind mount that is supposed to mount on top of an
GFS2 file system, you're in for a surprise. What happens is that the bind-mount
is carried out from rc.sysinit and _then_ the GFS2 filesystem is mounted, which
probably is not what was intended. I suggest that /etc/rc.d/init.d/gfs2 remounts
all bind mounts that originate on GFS2 file systems
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. put the following lines into /etc/fstab:
/dev/VolGroup00/LogVol01 /tmp/gfs2 gfs2 defaults 0 0
/tmp/gfs2 /tmp/foo none defaults,bind 0 0
2. restart your machine
3. inspect your mounts
/tmp/foo does not reflect the content of /tmp/gfs2
/tmp/foo should be mouned on top of /tmp/gfs2
I don't see why gfs2 needs to do anything special here.... from the fstab man page:
The order of records in fstab is important
because fsck(8), mount(8), and umount(8) sequentially iterate through
fstab doing their thing.
so if this is not happening then its not really a gfs2 problem, its an init
scripts problem I think. Also there is no way to tell from the path whether a
particular bind mount relates to some particular gfs2 mount or not without
specifying mount order somewhere since other fs might be mounted on mount points
on gfs2 filesystems or other bind mounts might result in a different mount tree.
There might also be places where we want to mount bind mounts before gfs2
mounts, so I don't think specifying an ordering by fs type is a good plan either.
gfs2 is special since all other mounts are done by /etc/rc.sysinit while gfs2
mounts are done later on by /etc/init.d/gfs2. So even if a bind mount is
specified after a gfs2 mount in fstab it will be mounted before the gfs2 file
system, breaking the semantics you quoted. But yes, I see it's not entirely
trivial to determine which bind mounts should be remounted after the gfs2 mounts...
Yes, it looks like the bug is /etc/init.d/gfs2. I need to look into why we are
doing this and not just mounting the fs like any other. I can't think of an
obvious reason at the moment.
The reason is that unless the file system happen to be non-clustered
(lockproto=lock_nolock), you need cman and probably clvmd up and running before
you can mount the GFS2 file system.
True, but thats done in /etc/init.d/cman. I've just had a quick look through the
code and so far as I can see NFS (and other network fs) also suffer from the
So ideally we'd not need any GFS2 scripts beyond the cman one, and we'd be able
to use the standard init scripts. Again though I don't see that this is really a
GFS2 problem since several other fs suffer from the same thing. Its really down
to ordering of operations in the boot scripts. If the mount -a was done in a
/etc/init.d/ script, then we could make the cman script run first. That would
solve the problem for non-root GFS2 at least.
While we are talking about GFS2 and initscripts, its also worth pointing out a
similar bug: #207697 where things are not happening in the right order. This
time its a shutdown issue.
Changed component to initscripts to see if we can perhaps get a discussion going
as to the best way to fix this particular problem, preferable for all affected
*** Bug 235204 has been marked as a duplicate of this bug. ***
If you have a bind mount that depends on something not mounted normally, you
should probably either noauto it, or mark it with _netdev (or something similar)
that will cause rc.sysinit to ignore it (that's a hack.)
It seems to be mostly a 'don't do that' scenario, though.
This isn't relating to things not normally mounted, this is normal system
startup, and we have in fact several issues which all need to be solved
eventually, so what we'd like to get fixed up is:
1. Mount ordering. Currently GFS2 mounts are done later than other fs mounts due
to the need to start up the cluster first (via /etc/init.d/cman). This is a
problem for two reasons: firstly as mentioned above, it means that bind mounts
will not happen in the right order wrt GFS2 mounts, and secondly there is the
GFS2 as root problem. In the case of the boot ordering, we might potentially use
_netdev, but that still requires changes since /etc/init.d/netfs doesn't mount
gfs2 mounts by default.
Just like NFS we do depend upon the network being up in order to get quorum
etc., so I guess we might be able to use _netdev provided we can be sure that
ordering (i.e. in fstab order) would be respected within all entries marked _netdev.
2. Unmounting (at shutdown, manually umounting has no problems). Currently there
is a problem in that the first thing that happens at shutdown is that userland
processes get killed off and then the filesystems are unmounted. Normally thats
what we want, but for GFS2 that kills off our cluster daemons and then umount
hangs. So we need a way to ensure that our daemons survive the initial killing
spree so that umount can proceed normally.
(In reply to comment #9)
> 1. Mount ordering. Currently GFS2 mounts are done later than other fs mounts due
> to the need to start up the cluster first (via /etc/init.d/cman). This is a
> problem for two reasons: firstly as mentioned above, it means that bind mounts
> will not happen in the right order wrt GFS2 mounts, and secondly there is the
> GFS2 as root problem. In the case of the boot ordering, we might potentially use
> _netdev, but that still requires changes since /etc/init.d/netfs doesn't mount
> gfs2 mounts by default.
> Just like NFS we do depend upon the network being up in order to get quorum
> etc., so I guess we might be able to use _netdev provided we can be sure that
> ordering (i.e. in fstab order) would be respected within all entries marked
Ordering of devices with _netdev should work as ordering of non-netdev devices
does under 'normal' fstab use.
David, does _netdev do all you need? It seems a reasonable solution to me.
Bill, it looks like we ought to be using /etc/init.d/netfs and _netdev to mount
GFS2 filesystems rather than /etc/init.d/gfs2, do you agree? Also there is still
the umount at shutdown issue to solve.
Yes, _netdev does the trick (-:
If cman starts first when needed, we could mount gfs2 there. Is there anything
else special that needs done?
Dave Teigland thinks so, due to quorum requirements:
<dct> swhiteho, _netdev and netfs aren't sufficient for gfs -- we've been over
this a couple times before :-)
So perhaps he can elaborate on this and see if we can make some progress on this
issue. This is an issue that has cropped up before a few times and it would be
nice to be able to come to a conclusion.
Also, wrt comment #13, yes, we probably could mount gfs2 there, but then how do
we specify the ordering wrt bind mounts (either order needs to be supported)
and/or mounts of other types too?
I guess the only "special" thing which would need to be done is to ensure that
any network connected block devices (iSCSI is quite common) are scanned by
device mapper before we try to mount gfs2 volumes.
Now it seems even with /etc/init.d/netfs that the man page's ordering is
violated since the various types of netfs are mounted according to type as
opposed to strict fstab ordering withing the _netdev tagged entries.
Even so, we do need to tackle the "gfs2 as root" problem too at some stage,
which will likely need a different solution so far as I can see at the moment.
Updating the bug to reflect what we are asking for more closely. Ideally we want to be able to set up a general dependency system for mounting filesystems. Something similar to _netdev but generalised to other services (e.g. cluster filesystems) would be ideal.
We need to be able to make things like bind mounts depend upon mounts of other filesystems during system startup, and some of the filesystem mounts need to depend upon such things as availability of the network (e.g. NFS) or cluster membership (e.g. GFS2).
We also have a shutdown problem which we'd like to solve.... when a node is shutdown, all the processes are killed before the system tries to umount the filesystems. When it comes to umount GFS2 (then unless the GFS2 file system was mounted by the initscripts, in which case the GFS2 script can umount earlier in the shutdown sequence) the umount hangs, due to the GFS2 controld being killed along with the other userland processes. So ideally we'd like to see the dependencies being used (plus maybe fuser? or similar) to umount the filesystems in an order which avoids these problems.
So I know that this isn't a trivial request, but it would be very useful, so I wonder how we might be able to work together towards this end?
Probably something involving a newer version of upstart than what we have, along with potentially a storage daemon of some sort. It's not anything we can do really soon.
This bug has been triaged
So, now that the gfs2-utils package has a systemd initscript in rawhide, I think we can say that this bug is done. Obviously, please reopen this if there are any objections, but I think we can close this now as the issue is (at long last) resolved.