Bug 209964 - NFS Client R/O in anaconda preinstall environment
NFS Client R/O in anaconda preinstall environment
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Dickson
Brian Brock
:
: 211204 217793 240698 241372 242533 248599 311221 (view as bug list)
Depends On:
Blocks: nosharecache
  Show dependency treegraph
 
Reported: 2006-10-08 22:46 EDT by Colin Coe
Modified: 2010-10-22 02:20 EDT (History)
21 users (show)

See Also:
Fixed In Version: RHBA-2007-0959
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-11-07 14:13:28 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Screen shot NFS client not being r/w (28.42 KB, image/png)
2006-10-08 22:46 EDT, Colin Coe
no flags Details
Upstream patches that add the "nosharecache" option (9.23 KB, patch)
2007-05-31 08:23 EDT, Steve Dickson
no flags Details | Diff
Purposed Upstream patch for mount.nfs (5.95 KB, patch)
2007-05-31 08:27 EDT, Steve Dickson
no flags Details | Diff
Initial kernel patch to add nosharecache option (broken, but a start maybe) (6.62 KB, patch)
2007-05-31 08:50 EDT, Ian Kent
no flags Details | Diff
Updated upstream patch 2, much the same as the upstream version (1.82 KB, patch)
2007-05-31 08:54 EDT, Ian Kent
no flags Details | Diff

  None (edit)
Description Colin Coe 2006-10-08 22:46:06 EDT
Description of problem:
NFS Client in RHEL5 Client BETA1 doesn't support r/w access to my NFS server.

Version-Release number of selected component (if applicable):
RHEL5 Clinet BETA1

How reproducible:


Steps to Reproduce:
1. Begin installing RHEL5 BETA1
2. When console available on ttys1:
3. mkdir /mnt/tmp
4. mount -t nfs -o rw server:/path/to/dir
5. mount

  
Actual results:
Note that 'ro' is listed in the options and you cannot write to the NFS server.

Expected results:
'rw' should have been listed in the options list.

Additional info:
See attached .png file
Comment 1 Colin Coe 2006-10-08 22:46:08 EDT
Created attachment 138025 [details]
Screen shot NFS client not being r/w
Comment 2 Jani Myyry 2006-11-20 13:03:24 EST
We have encountered a similar problem in RHEL5 beta 2. The operating system does
not respect ro/rw options after the first mount but uses the option given during
the first mount in the subsequent mounts. For example:

# grep nfs /etc/fstab
nfsserver:/vol/bin/gen  /m/fs/gen  nfs ro,vers=3,tcp,hard,intr,bg 0 0
nfsserver:/vol/bin/rw   /m/fs/rw   nfs rw,vers=3,tcp,hard,intr,bg 0 0

# mount|grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsserver:/vol/bin/gen on /m/fs/gen type nfs (ro,...)
nfsserver:/vol/bin/rw on /m/fs/rw type nfs (rw,...)

# grep nfs /proc/mounts
nfsserver:/vol/bin/gen /m/fs/gen nfs ro,... 0 0
nfsserver:/vol/bin/rw /m/fs/rw nfs ro,... 0 0

# touch /m/fs/rw/test
touch: cannot touch `/m/fs/rw/test': Read-only file system

This behaviour occurs per file system type (nfsv3, nfsv4) ie. all nfsv3 mounts
get the ro/rw option of the first nfsv3 mount and all nfsv4 mounts get the ro/rw
option of the first nfsv4 mount. 
Comment 3 Jani Myyry 2006-11-21 05:59:21 EST
More info: the bug seems to be related with FC6 bug 207670. There are also some
differences between RHEL4.4 and RHEL5b2 with identical (two) nfs mounts:

RHEL4.4:
# ls -al /var/lib/nfs/rpc_pipefs/nfs/*/info
-r--------  1 root root 0 Aug 17 15:39 /var/lib/nfs/rpc_pipefs/nfs/clnt0/info
-r--------  1 root root 0 Aug 17 15:39 /var/lib/nfs/rpc_pipefs/nfs/clnt1/info

RHEL5b2:
# ls -al /var/lib/nfs/rpc_pipefs/nfs/*/info
-r--------  1 root root 0 Nov 21 12:35 /var/lib/nfs/rpc_pipefs/nfs/clnt0/info
Comment 4 Steve Dickson 2006-12-21 10:11:15 EST
*** Bug 217793 has been marked as a duplicate of this bug. ***
Comment 5 Steve Dickson 2006-12-21 10:18:12 EST
*** Bug 211204 has been marked as a duplicate of this bug. ***
Comment 6 Steve Dickson 2006-12-21 10:19:37 EST
Here is the upstream discussion on this subject:
([RFC] [PATCH] Per-mountpoint read-only and noatime revisite)
http://linux.derkeiler.com/Mailing-Lists/Kernel/2005-01/9038.html

It seem VFS support is needed to fix this problem...
Comment 7 RHEL Product and Program Management 2006-12-21 10:28:41 EST
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 8 Chris Johns 2007-01-23 12:09:31 EST
I'd like to upgrade this to high severity, since it's really affecting our
ability to run production systems. We depend on being able to mount a piece of a
filesystem tree read-only, shared between many systems, and overlay other
pieces, from the same NFS server filesystem, read-write on each client. Having
the client-specific pieces also mounted read-only isn't going to work at all.

Michael Walker from Cassatt has had an email exchange with Trond Myklebust on
the SourceForge NFS email list, and apparently there are patches floating around
to fix this supposed feature (actually, bug), but those patches have not been
integrated into the mainline kernel tree yet.

The link is: http://sourceforge.net/mailarchive/message.php?msg_id=37315025

My request is that, unless and until this behavior is fixed, would Red Hat be
willing to disable this code and revert to the previous behavior, such as exists
in Red Hat ELAS4?
Comment 9 Steve Dickson 2007-01-24 14:15:14 EST
Subject: Re: [NFS] NFS mounts being mounted read-only depending upon	mount	order???
Date: Mon, 20 Nov 2006 16:43:47 -0500
From: Trond Myklebust <trond.myklebust@fys.uio.no>

There _are_ workarounds available. You can use the 'fsid' export option
to tell knfsd to export the directories as if they were different
filesystems.

For instance if I have

        /export/home	 141.211.133.0/24(rw,sync,no_subtree_check,fsid=0)
        /export/home/trondmy 141.211.133.0/24(rw,sync,no_subtree_check,fsid=3)

in /etc/exports on the server, then I can do

mount -t nfs -oro server:/export/home /mnt/foo
mount -t nfs -orw server:/export/home/trondmy /mnt/bar

with no trouble. From 'cat /proc/mounts'

server:/export/home /mnt/foo nfs
rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server
0 0
server:/export/home/trondmy /mnt/bar nfs
ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server
0 0

as expected.

> > Can you point me to where the required 'mount --bind' fixes are
> > being discussed?  I haven't been able to find the
> > thread in the 'lkml' (not that it isn't there somewhere).
> > 
> > Without the matching 'mount --bind' fixes this is a very serious
> > regression for us?

I believe the most recent patchsets can be found from

http://marc.theaimsgroup.com/?l=linux-kernel&m=115447687730218&w=2

Cheers,
  Trond


Comment 10 Chris Johns 2007-01-24 14:34:03 EST
Yes, we know about the 'fsid' option on the exports line, and we could modify
our code to generate unique fsid values for /etc/exports.

However, it won't work for us in the case of a customer using an external NFS
filer, such as a NetApp, to export the top-level directory to many different
servers, each of which is mounting individual sub-directories from the
filesystem with different permissions. That's actually the nominal case in a
production environment.

In that case, the customer would be required to add hundreds of entries to their
filer, coming up with new unique IDs for each directory, and modify this every
time a new server is added. That's not a useful situation in a large data center
using Red Hat EL5 clients.

In fact, as I mentioned before, either forcing the clients to learn about the
server's filesystem layout to get the mount behavior they expect, or
deliberately fooling the clients, by using the fsid option, is contrary to the
spirit of NFS, where the client only needs to know the path to a directory,
without knowing in which filesystem it exists on the server.
Comment 14 Brad Hinson 2007-05-25 11:15:50 EDT
*** Bug 241372 has been marked as a duplicate of this bug. ***
Comment 16 Ian Kent 2007-05-30 23:02:36 EDT
*** Bug 240698 has been marked as a duplicate of this bug. ***
Comment 17 Ian Kent 2007-05-30 23:24:39 EDT
fyi to all, current status,

The source of this problem is the sharing of the super block for
mounts for a server export with the same fsid which is a change
introduced upstream.

A couple of fairly simple patches were posted to the NFS list
that should fix this and initial (very basic) testing on a
patched Fedora kernel indicates that it works. In order to
prevent the sharing a mount option is needed so mount.nfs(8)
and mount.nfs4(8) need changes to provide for the option.
Since control over the mount options may not be available
in some cases I'm not sure what the final resolution of
this will involve. Right now I'm just working on the mount
issue itself.

However, RHEL 5 has the fscache system included in the kernel
which is not compatible with preventing the super block sharing
and I'm trying to add checks for incompatible use of the options.
When testing on the latest RHEL 5 kernel I get a hard system hang.
No doubt there's something wrong in my changes.

So that's it so far.

Ian
Comment 18 Steve Dickson 2007-05-31 08:23:49 EDT
Created attachment 155801 [details]
Upstream patches that add the "nosharecache" option

Note: these commit blobs are relative to Trond's git tree.
Comment 19 Steve Dickson 2007-05-31 08:27:56 EDT
Created attachment 155802 [details]
Purposed Upstream patch for mount.nfs
Comment 22 Ian Kent 2007-05-31 08:50:30 EDT
Created attachment 155803 [details]
Initial kernel patch to add nosharecache option (broken, but a start maybe)

This is the first of the two upstream patches to provide
an option to not use the shared super block functionality.

I'm fairly sure the added check for incompatible mount
options is not right but I don't understand why I get a
hard hang for any mount using "nocache". 

Ian
Comment 23 Ian Kent 2007-05-31 08:54:11 EDT
Created attachment 155804 [details]
Updated upstream patch 2, much the same as the upstream version

Another thing comes to mind.
I've probably chosen a badly to eliminate the clash between
the define for fscache and nosharecache options.
Over to you for this Steve.
Comment 24 Don Zickus 2007-06-12 14:52:00 EDT
in 2.6.18-24.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5
Comment 25 Steve Dickson 2007-06-21 13:06:26 EDT
*** Bug 242533 has been marked as a duplicate of this bug. ***
Comment 27 Ian Kent 2007-07-06 23:40:14 EDT
(In reply to comment #26)
> Hi,
> 
> Some reports from the customer trying the test kernel:

I thought the "nosharecache" mount option needed to be
given to enable this and of course a mount that knows
about this option is needed.

> 
> ----------
> I was able to get the test system to boot up using the kernel options
> "nosmp noapic nolapic acpi=0".  But it still spits out "soft lockup"
> BUGs every once in a while.  At least I was able to test out the NFS
> bits:
> 
> [root@adcgar05 ~]# cat /proc/mounts | grep eng
> [root@adcgar05 ~]# cd /mnt
> [root@adcgar05 mnt]# mkdir mount1 mount2
> [root@adcgar05 mnt]# mount -o ro,rsize=8192,wsize=8192
> eng:/vol/vol18/sysadmin_tmp mount1
> [root@adcgar05 mnt]# cat /proc/mounts | grep eng
> eng:/vol/vol18/sysadmin_tmp /mnt/mount1 nfs
>
ro,vers=3,rsize=8192,wsize=8192,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
> 0 0
> [root@adcgar05 mnt]# mount -o
> rw,rsize=16384,wsize=16384,acregmin=1,acdirmin=1
> eng:/vol/vol18/site-config mount2
> mount.nfs: /mnt/mount2 is already mounted or busy
> 
> This is not expected behavior.  I expect the second mount attempt to
> succeed.  Notice that I'm mounting two completely different filesystems
> (despite the fact that they're both on the same volume on the filer)
> 
> This "fix" would actually make matters worse in our environment, because
> we would no longer be "allowed" to mount multiple things from the same
> volume from a filer, rendering NFS effectively useless to us.
> 
> * * * * * * * * * * 
> Weird...somehow autofs seems to at least be able to do multiple mounts...
> 
> [root@adcgar05 /]# cat /proc/mounts | grep vol18
> 
> ** Nothing mounted on eng:/vol/vol18...
> 
> [root@adcgar05 /]# cd /tool/site-config
> [root@adcgar05 site-config]# cat /proc/mounts | grep site-config
> eng:/vol/vol18/site-config /tool/site-config nfs
>
rw,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
> 0 0
> 
> ** OK now eng:/vol/vol18/site-config is mounted with shown options
> 
> [root@adcgar05 site-config]# cd /mnt
> [root@adcgar05 mnt]# mount -t nfs -o ro eng:/vol/vol18/sysadmin_tmp
> mount1
> mount.nfs: /mnt/mount1 is already mounted or busy
> 
> ** Doh! Can't mount anything else off vol18
> 
> [root@adcgar05 mnt]# mount -t nfs -o rw eng:/vol/vol18/sysadmin_tmp
> mount1
> mount.nfs: /mnt/mount1 is already mounted or busy
> 
> ** Can't even mount anything else if given the same flags as previous
> mount
> 
> [root@adcgar05 mnt]# mount eng:/vol/vol18/site-config mount1
> mount.nfs: /mnt/mount1 is already mounted or busy
> 
> ** Can't even mount without options!
> 
> [root@adcgar05 mnt]# cd /tool/sysadmin_tmp
> [root@adcgar05 sysadmin_tmp]# cat /proc/mounts | grep eng.*vol18
> eng:/vol/vol18/site-config /tool/site-config nfs
>
rw,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
> 0 0
> eng:/vol/vol18/sysadmin_tmp /tool/sysadmin_tmp nfs
>
rw,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
> 0 0
> 
> ** And yet, autofs seems to have had no trouble with it...
> 
> [root@adcgar05 mnt]# ypmatch site-config auto.tool
> -intr   eng:/vol/vol18/&
> [root@adcgar05 mnt]# ypmatch sysadmin_tmp auto.tool
> -intr        eng:/vol/vol18/&
> 
> 
> This event sent from IssueTracker by pvn 
>  issue 115656
>               

Comment 30 Jeff Layton 2007-07-18 13:16:35 EDT
*** Bug 248599 has been marked as a duplicate of this bug. ***
Comment 33 Mike Gahagan 2007-08-23 14:55:38 EDT
Verified by booting the installer from the 0822.1 tree and sucessfully mounted a
share read-write. 
Comment 34 Steve Dickson 2007-08-30 15:43:04 EDT
*** Bug 243913 has been marked as a duplicate of this bug. ***
Comment 35 Don Domingo 2007-08-30 20:24:43 EDT
clearing requires_release_note flag, as duplicate bug is already being used to
track doc changes for this.
Comment 37 Jeff Layton 2007-09-28 13:46:21 EDT
*** Bug 311221 has been marked as a duplicate of this bug. ***
Comment 40 errata-xmlrpc 2007-11-07 14:13:28 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0959.html

Note You need to log in before you can comment on or make changes to this bug.