Bug 214877

Summary: mount's "can't create lock file" message sometimes means failure, sometimes not
Product: [Fedora] Fedora Reporter: Mark McLoughlin <markmc>
Component: util-linux-ngAssignee: Karel Zak <kzak>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: law
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-02-18 09:19:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
util-linux-mtab-create.patch
none
util-linux-ng-2.13.1-rc1-consistent-mtab-update-failure.patch none

Description Mark McLoughlin 2006-11-09 20:35:52 UTC
The context of this is that if you set $READONLY to "yes" in
/etc/sysconfig/readonly-root, rc.sysinit will bind mount the files listed in
/etc/rwtab to a tmpfs mounted at /var/lib/stateless/writable. Also, if you have
a disk labelled "stateless-state", it will mount that disk at
/var/lib/stateless/stateand bind-mount the files listed in /etc/statetab to there.

So, lot's of mounting going on. But that's not really important.

The problem is that because /etc/ isn't writable, you get error messages along
the lines of "can't create lock file /etc/mtab~1504" whenever you mount without -n

However, depending on whether /etc/mtab exists that error is fatal (i.e. the
mount failed) or it just means that /etc/mtab isn't mounted. It would seem to be
sensible to either consistently fail (or not) to mount if we can't update /etc/mtab.

Attached patch defers the attempt to create /etc/mtab until after the mount has
completed.

Comment 1 Mark McLoughlin 2006-11-09 20:35:52 UTC
Created attachment 140824 [details]
util-linux-mtab-create.patch

Comment 2 Karel Zak 2006-11-09 23:26:01 UTC
Your request make sense, but if create_mtab() fails:
  * old version: the system is still in consistent state (= no mount, no mtab)
  * with your patch: the system is inconsistent (= device succesfully mounted, 
    but no mtab, the mount returns error). 

I think this is the reason why the mount util checks all things befor real mount(2).

Also, the mount uses the lock file for mtab updates too. The solution should be
more generic (= useful for create_mtab() and for update_mtab())

I think the best way will be found a way how detect real place (FS) where is
writeable /etc/mtab and create the lock file on the same FS. I think implement a
heuristic that from /proc/mounts detects the bind mount for /etc/mtab won't be
so difficult. It means the lock should be /var/lib/stateless/writable/mtab~<pid>
if there is a bind for mtab to /var/lib/stateless/writable/mtab.

You have to update all programs which touch the mtab and use same lock method of
course.

I hope I didn't overlook something :-)

Comment 3 Mark McLoughlin 2006-11-10 15:51:59 UTC
(In reply to comment #2)
> Your request make sense, but if create_mtab() fails:
>   * old version: the system is still in consistent state (= no mount, no mtab)
>   * with your patch: the system is inconsistent (= device succesfully mounted, 
>     but no mtab, the mount returns error). 

Note, update_mtab() can fail in exactly this way right now

My proposal is that such a failure is fine, but create_mtab() should fail in the
same way

We have two sane choices, I think:

  1) If updating mtab fails, then the mount fails
  2) Whether updating mtab succeeds has no impact on the mount

But it's not sane for it to sometimes be (1) and sometimes be (2)

Maybe we just need to fix it to be (1) ... but we'd also need to figure out some
way to allow the updating to succeed on readonly-root (i.e. fix the locking)

> I think this is the reason why the mount util checks all things befor real
mount(2).

Except it doesn't (and can't?) check for everything - i.e. whether the locking
will succeed.

> Also, the mount uses the lock file for mtab updates too. The solution should be
> more generic (= useful for create_mtab() and for update_mtab())
> 
> I think the best way will be found a way how detect real place (FS) where is
> writeable /etc/mtab and create the lock file on the same FS. I think implement a
> heuristic that from /proc/mounts detects the bind mount for /etc/mtab won't be
> so difficult. It means the lock should be /var/lib/stateless/writable/mtab~<pid>
> if there is a bind for mtab to /var/lib/stateless/writable/mtab.

Something like that would be feasible ... but it's a little hairy. Hmm.

> You have to update all programs which touch the mtab and use same lock method of
> course.

Suck. :-)

Comment 4 Jeff Law 2006-11-29 21:38:33 UTC
This has been idle for a couple weeks, it's time to figure out how we're going
to deal with this problem.  In fact, it's become more important as we need it
addressed so that we can use anaconda to instantiate a stateless image given
system profile metadata (1).

I think we've agreed that we need to reach some level of consistency in
creation/updating of mtab.  The only question is how to best do that.

Since /etc/mtab~ and its relatives are lock files, I'd really prefer to see them
in /var/lock; the main objection to that was what to do when /var is a separate
filesystem and isn't currently mounted (as might be the case early in the boot
process).  Right?

ISTM that problem can be cleanly addressed by issuing:

mount -n /var
mount -n /var/lock

Immediately after we've performed filesystem checks (IIRC, this is what Solaris
used to do eons ago).

For a normal system both operations would be NOPs. 

For a system with /var on its own FS, the first command would mount it and we
can continue on our merry way.  The second command would be a NOP.

For a readonly-root system (without a separate /var) we'd have an entry in
/etc/fstab for /var/lock using tmpfs.

None of this seems particularly difficult to implement.  Then it's a "simple"
matter of updating all the appropriate utilities.  Right?

Jeff  

Comment 5 Karel Zak 2006-11-29 23:29:17 UTC
OK. I think we can move it to /var/lock/mtab (depends on initscript changes of
course), but there should be at least symlink from /etc/mtab to this new
location (for backward compatibility).

By the way, the best way is discuss changes like this at fedora-devel list.

Comment 6 Karel Zak 2006-11-30 10:27:01 UTC
Hmm.. now I think about rescue mode. The basic idea is keep the root partition
usable and consistent in itself. It means without any other partition :-(

Maybe the mount (and the others utils) can stat(/var/lock) and only if this
operation failed use old good /etc/mtab. Comments?

Comment 7 Jeff Law 2006-11-30 17:01:56 UTC
Just to be 100% clear, we're talking about moving mtab~ and mtabXXXX~, the lock
files, not /etc/mtab.

With that in mind, I don't think we should bother with a symlink to the lock
file; it doesn't make a lot of sense to me.  Every application which mucks with
the lock file should be updated rather than relying on a symlink.

To ensure rescue mode works all I think we need to do is create /var/lock in the
root filesystem.  That is a NOP for normal system.  For systems with a separate
/var it provides a suitable lock directory when running in rescue mode (which is
probably a good idea anyway).

And yes, this should ultimately move to fedora-devel; I'll take care of that
once we've got a solution we're pretty sure will work cleanly.

Jeff

Comment 8 Jeff Law 2006-11-30 19:04:41 UTC
One more note.  Mark's patch only deals with getting some consistency in regards
to error messages and the like.  It does not address the need to move the lock
file out of /etc.  They are separate, but related issues.

Jeff

Comment 9 Mark McLoughlin 2006-11-30 19:07:51 UTC
Yeah ... see bug #214891

Comment 10 Karel Zak 2006-11-30 22:21:05 UTC
Ah.. yes mtab~, sorry for misunderstanding.



Comment 11 Mark McLoughlin 2007-12-18 09:54:38 UTC
Created attachment 289873 [details]
util-linux-ng-2.13.1-rc1-consistent-mtab-update-failure.patch

Comment 12 Mark McLoughlin 2007-12-18 09:59:34 UTC
kzak: the patch and the subject of this bug got lost in the discussion around
moving the mtab locks to /var/lock (bug #214891)

What the patch does is goes from the situation where:

  1) If /etc/mtab doesn't exist and /etc is read-only, you get the "can't create
     lock file" message and the mount fails
  2) If /etc/mtab does exist and /etc is read-only,you get the same message but
     the mount succeeds

Clearly, the failure to update /etc/mtab should either cause the mount to fail
or not ... sometimes causing it to fail, and sometimes not (each with the same
message) is not useful.

The attached patch takes the approach of making it so failure to update/create
/etc/mtab does not cause the mount to fail

Comment 13 Karel Zak 2008-01-23 13:35:17 UTC
I'll apply the patch to the upstream devel tree and rawhide. Thanks and sorry
for delay.