Description of problem: If you in your fstab have a bind mount that is supposed to mount on top of an GFS2 file system, you're in for a surprise. What happens is that the bind-mount is carried out from rc.sysinit and _then_ the GFS2 filesystem is mounted, which probably is not what was intended. I suggest that /etc/rc.d/init.d/gfs2 remounts all bind mounts that originate on GFS2 file systems Version-Release number of selected component (if applicable): initscripts-8.60-1 gfs2-utils-0.1.25-1.fc7 How reproducible: every time Steps to Reproduce: 1. put the following lines into /etc/fstab: /dev/VolGroup00/LogVol01 /tmp/gfs2 gfs2 defaults 0 0 /tmp/gfs2 /tmp/foo none defaults,bind 0 0 2. restart your machine 3. inspect your mounts Actual results: /tmp/foo does not reflect the content of /tmp/gfs2 Expected results: /tmp/foo should be mouned on top of /tmp/gfs2
I don't see why gfs2 needs to do anything special here.... from the fstab man page: The order of records in fstab is important because fsck(8), mount(8), and umount(8) sequentially iterate through fstab doing their thing. so if this is not happening then its not really a gfs2 problem, its an init scripts problem I think. Also there is no way to tell from the path whether a particular bind mount relates to some particular gfs2 mount or not without specifying mount order somewhere since other fs might be mounted on mount points on gfs2 filesystems or other bind mounts might result in a different mount tree. There might also be places where we want to mount bind mounts before gfs2 mounts, so I don't think specifying an ordering by fs type is a good plan either.
gfs2 is special since all other mounts are done by /etc/rc.sysinit while gfs2 mounts are done later on by /etc/init.d/gfs2. So even if a bind mount is specified after a gfs2 mount in fstab it will be mounted before the gfs2 file system, breaking the semantics you quoted. But yes, I see it's not entirely trivial to determine which bind mounts should be remounted after the gfs2 mounts...
Yes, it looks like the bug is /etc/init.d/gfs2. I need to look into why we are doing this and not just mounting the fs like any other. I can't think of an obvious reason at the moment.
The reason is that unless the file system happen to be non-clustered (lockproto=lock_nolock), you need cman and probably clvmd up and running before you can mount the GFS2 file system.
True, but thats done in /etc/init.d/cman. I've just had a quick look through the code and so far as I can see NFS (and other network fs) also suffer from the same problem. So ideally we'd not need any GFS2 scripts beyond the cman one, and we'd be able to use the standard init scripts. Again though I don't see that this is really a GFS2 problem since several other fs suffer from the same thing. Its really down to ordering of operations in the boot scripts. If the mount -a was done in a /etc/init.d/ script, then we could make the cman script run first. That would solve the problem for non-root GFS2 at least. While we are talking about GFS2 and initscripts, its also worth pointing out a similar bug: #207697 where things are not happening in the right order. This time its a shutdown issue.
Changed component to initscripts to see if we can perhaps get a discussion going as to the best way to fix this particular problem, preferable for all affected filesystems.
*** Bug 235204 has been marked as a duplicate of this bug. ***
If you have a bind mount that depends on something not mounted normally, you should probably either noauto it, or mark it with _netdev (or something similar) that will cause rc.sysinit to ignore it (that's a hack.) It seems to be mostly a 'don't do that' scenario, though.
This isn't relating to things not normally mounted, this is normal system startup, and we have in fact several issues which all need to be solved eventually, so what we'd like to get fixed up is: 1. Mount ordering. Currently GFS2 mounts are done later than other fs mounts due to the need to start up the cluster first (via /etc/init.d/cman). This is a problem for two reasons: firstly as mentioned above, it means that bind mounts will not happen in the right order wrt GFS2 mounts, and secondly there is the GFS2 as root problem. In the case of the boot ordering, we might potentially use _netdev, but that still requires changes since /etc/init.d/netfs doesn't mount gfs2 mounts by default. Just like NFS we do depend upon the network being up in order to get quorum etc., so I guess we might be able to use _netdev provided we can be sure that ordering (i.e. in fstab order) would be respected within all entries marked _netdev. 2. Unmounting (at shutdown, manually umounting has no problems). Currently there is a problem in that the first thing that happens at shutdown is that userland processes get killed off and then the filesystems are unmounted. Normally thats what we want, but for GFS2 that kills off our cluster daemons and then umount hangs. So we need a way to ensure that our daemons survive the initial killing spree so that umount can proceed normally.
(In reply to comment #9) > 1. Mount ordering. Currently GFS2 mounts are done later than other fs mounts due > to the need to start up the cluster first (via /etc/init.d/cman). This is a > problem for two reasons: firstly as mentioned above, it means that bind mounts > will not happen in the right order wrt GFS2 mounts, and secondly there is the > GFS2 as root problem. In the case of the boot ordering, we might potentially use > _netdev, but that still requires changes since /etc/init.d/netfs doesn't mount > gfs2 mounts by default. > > Just like NFS we do depend upon the network being up in order to get quorum > etc., so I guess we might be able to use _netdev provided we can be sure that > ordering (i.e. in fstab order) would be respected within all entries marked _netdev. Ordering of devices with _netdev should work as ordering of non-netdev devices does under 'normal' fstab use.
David, does _netdev do all you need? It seems a reasonable solution to me. Bill, it looks like we ought to be using /etc/init.d/netfs and _netdev to mount GFS2 filesystems rather than /etc/init.d/gfs2, do you agree? Also there is still the umount at shutdown issue to solve.
Yes, _netdev does the trick (-:
If cman starts first when needed, we could mount gfs2 there. Is there anything else special that needs done?
Dave Teigland thinks so, due to quorum requirements: <dct> swhiteho, _netdev and netfs aren't sufficient for gfs -- we've been over this a couple times before :-) So perhaps he can elaborate on this and see if we can make some progress on this issue. This is an issue that has cropped up before a few times and it would be nice to be able to come to a conclusion. Also, wrt comment #13, yes, we probably could mount gfs2 there, but then how do we specify the ordering wrt bind mounts (either order needs to be supported) and/or mounts of other types too? I guess the only "special" thing which would need to be done is to ensure that any network connected block devices (iSCSI is quite common) are scanned by device mapper before we try to mount gfs2 volumes. Now it seems even with /etc/init.d/netfs that the man page's ordering is violated since the various types of netfs are mounted according to type as opposed to strict fstab ordering withing the _netdev tagged entries. Even so, we do need to tackle the "gfs2 as root" problem too at some stage, which will likely need a different solution so far as I can see at the moment.
Updating the bug to reflect what we are asking for more closely. Ideally we want to be able to set up a general dependency system for mounting filesystems. Something similar to _netdev but generalised to other services (e.g. cluster filesystems) would be ideal. We need to be able to make things like bind mounts depend upon mounts of other filesystems during system startup, and some of the filesystem mounts need to depend upon such things as availability of the network (e.g. NFS) or cluster membership (e.g. GFS2). We also have a shutdown problem which we'd like to solve.... when a node is shutdown, all the processes are killed before the system tries to umount the filesystems. When it comes to umount GFS2 (then unless the GFS2 file system was mounted by the initscripts, in which case the GFS2 script can umount earlier in the shutdown sequence) the umount hangs, due to the GFS2 controld being killed along with the other userland processes. So ideally we'd like to see the dependencies being used (plus maybe fuser? or similar) to umount the filesystems in an order which avoids these problems. So I know that this isn't a trivial request, but it would be very useful, so I wonder how we might be able to work together towards this end?
Probably something involving a newer version of upstart than what we have, along with potentially a storage daemon of some sort. It's not anything we can do really soon.
This bug has been triaged
So, now that the gfs2-utils package has a systemd initscript in rawhide, I think we can say that this bug is done. Obviously, please reopen this if there are any objections, but I think we can close this now as the issue is (at long last) resolved.