Description of problem: NFS Client in RHEL5 Client BETA1 doesn't support r/w access to my NFS server. Version-Release number of selected component (if applicable): RHEL5 Clinet BETA1 How reproducible: Steps to Reproduce: 1. Begin installing RHEL5 BETA1 2. When console available on ttys1: 3. mkdir /mnt/tmp 4. mount -t nfs -o rw server:/path/to/dir 5. mount Actual results: Note that 'ro' is listed in the options and you cannot write to the NFS server. Expected results: 'rw' should have been listed in the options list. Additional info: See attached .png file
Created attachment 138025 [details] Screen shot NFS client not being r/w
We have encountered a similar problem in RHEL5 beta 2. The operating system does not respect ro/rw options after the first mount but uses the option given during the first mount in the subsequent mounts. For example: # grep nfs /etc/fstab nfsserver:/vol/bin/gen /m/fs/gen nfs ro,vers=3,tcp,hard,intr,bg 0 0 nfsserver:/vol/bin/rw /m/fs/rw nfs rw,vers=3,tcp,hard,intr,bg 0 0 # mount|grep nfs sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsserver:/vol/bin/gen on /m/fs/gen type nfs (ro,...) nfsserver:/vol/bin/rw on /m/fs/rw type nfs (rw,...) # grep nfs /proc/mounts nfsserver:/vol/bin/gen /m/fs/gen nfs ro,... 0 0 nfsserver:/vol/bin/rw /m/fs/rw nfs ro,... 0 0 # touch /m/fs/rw/test touch: cannot touch `/m/fs/rw/test': Read-only file system This behaviour occurs per file system type (nfsv3, nfsv4) ie. all nfsv3 mounts get the ro/rw option of the first nfsv3 mount and all nfsv4 mounts get the ro/rw option of the first nfsv4 mount.
More info: the bug seems to be related with FC6 bug 207670. There are also some differences between RHEL4.4 and RHEL5b2 with identical (two) nfs mounts: RHEL4.4: # ls -al /var/lib/nfs/rpc_pipefs/nfs/*/info -r-------- 1 root root 0 Aug 17 15:39 /var/lib/nfs/rpc_pipefs/nfs/clnt0/info -r-------- 1 root root 0 Aug 17 15:39 /var/lib/nfs/rpc_pipefs/nfs/clnt1/info RHEL5b2: # ls -al /var/lib/nfs/rpc_pipefs/nfs/*/info -r-------- 1 root root 0 Nov 21 12:35 /var/lib/nfs/rpc_pipefs/nfs/clnt0/info
*** Bug 217793 has been marked as a duplicate of this bug. ***
*** Bug 211204 has been marked as a duplicate of this bug. ***
Here is the upstream discussion on this subject: ([RFC] [PATCH] Per-mountpoint read-only and noatime revisite) http://linux.derkeiler.com/Mailing-Lists/Kernel/2005-01/9038.html It seem VFS support is needed to fix this problem...
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
I'd like to upgrade this to high severity, since it's really affecting our ability to run production systems. We depend on being able to mount a piece of a filesystem tree read-only, shared between many systems, and overlay other pieces, from the same NFS server filesystem, read-write on each client. Having the client-specific pieces also mounted read-only isn't going to work at all. Michael Walker from Cassatt has had an email exchange with Trond Myklebust on the SourceForge NFS email list, and apparently there are patches floating around to fix this supposed feature (actually, bug), but those patches have not been integrated into the mainline kernel tree yet. The link is: http://sourceforge.net/mailarchive/message.php?msg_id=37315025 My request is that, unless and until this behavior is fixed, would Red Hat be willing to disable this code and revert to the previous behavior, such as exists in Red Hat ELAS4?
Subject: Re: [NFS] NFS mounts being mounted read-only depending upon mount order??? Date: Mon, 20 Nov 2006 16:43:47 -0500 From: Trond Myklebust <trond.myklebust.no> There _are_ workarounds available. You can use the 'fsid' export option to tell knfsd to export the directories as if they were different filesystems. For instance if I have /export/home 141.211.133.0/24(rw,sync,no_subtree_check,fsid=0) /export/home/trondmy 141.211.133.0/24(rw,sync,no_subtree_check,fsid=3) in /etc/exports on the server, then I can do mount -t nfs -oro server:/export/home /mnt/foo mount -t nfs -orw server:/export/home/trondmy /mnt/bar with no trouble. From 'cat /proc/mounts' server:/export/home /mnt/foo nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server 0 0 server:/export/home/trondmy /mnt/bar nfs ro,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=server 0 0 as expected. > > Can you point me to where the required 'mount --bind' fixes are > > being discussed? I haven't been able to find the > > thread in the 'lkml' (not that it isn't there somewhere). > > > > Without the matching 'mount --bind' fixes this is a very serious > > regression for us? I believe the most recent patchsets can be found from http://marc.theaimsgroup.com/?l=linux-kernel&m=115447687730218&w=2 Cheers, Trond
Yes, we know about the 'fsid' option on the exports line, and we could modify our code to generate unique fsid values for /etc/exports. However, it won't work for us in the case of a customer using an external NFS filer, such as a NetApp, to export the top-level directory to many different servers, each of which is mounting individual sub-directories from the filesystem with different permissions. That's actually the nominal case in a production environment. In that case, the customer would be required to add hundreds of entries to their filer, coming up with new unique IDs for each directory, and modify this every time a new server is added. That's not a useful situation in a large data center using Red Hat EL5 clients. In fact, as I mentioned before, either forcing the clients to learn about the server's filesystem layout to get the mount behavior they expect, or deliberately fooling the clients, by using the fsid option, is contrary to the spirit of NFS, where the client only needs to know the path to a directory, without knowing in which filesystem it exists on the server.
*** Bug 241372 has been marked as a duplicate of this bug. ***
*** Bug 240698 has been marked as a duplicate of this bug. ***
fyi to all, current status, The source of this problem is the sharing of the super block for mounts for a server export with the same fsid which is a change introduced upstream. A couple of fairly simple patches were posted to the NFS list that should fix this and initial (very basic) testing on a patched Fedora kernel indicates that it works. In order to prevent the sharing a mount option is needed so mount.nfs(8) and mount.nfs4(8) need changes to provide for the option. Since control over the mount options may not be available in some cases I'm not sure what the final resolution of this will involve. Right now I'm just working on the mount issue itself. However, RHEL 5 has the fscache system included in the kernel which is not compatible with preventing the super block sharing and I'm trying to add checks for incompatible use of the options. When testing on the latest RHEL 5 kernel I get a hard system hang. No doubt there's something wrong in my changes. So that's it so far. Ian
Created attachment 155801 [details] Upstream patches that add the "nosharecache" option Note: these commit blobs are relative to Trond's git tree.
Created attachment 155802 [details] Purposed Upstream patch for mount.nfs
Created attachment 155803 [details] Initial kernel patch to add nosharecache option (broken, but a start maybe) This is the first of the two upstream patches to provide an option to not use the shared super block functionality. I'm fairly sure the added check for incompatible mount options is not right but I don't understand why I get a hard hang for any mount using "nocache". Ian
Created attachment 155804 [details] Updated upstream patch 2, much the same as the upstream version Another thing comes to mind. I've probably chosen a badly to eliminate the clash between the define for fscache and nosharecache options. Over to you for this Steve.
in 2.6.18-24.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5
*** Bug 242533 has been marked as a duplicate of this bug. ***
(In reply to comment #26) > Hi, > > Some reports from the customer trying the test kernel: I thought the "nosharecache" mount option needed to be given to enable this and of course a mount that knows about this option is needed. > > ---------- > I was able to get the test system to boot up using the kernel options > "nosmp noapic nolapic acpi=0". But it still spits out "soft lockup" > BUGs every once in a while. At least I was able to test out the NFS > bits: > > [root@adcgar05 ~]# cat /proc/mounts | grep eng > [root@adcgar05 ~]# cd /mnt > [root@adcgar05 mnt]# mkdir mount1 mount2 > [root@adcgar05 mnt]# mount -o ro,rsize=8192,wsize=8192 > eng:/vol/vol18/sysadmin_tmp mount1 > [root@adcgar05 mnt]# cat /proc/mounts | grep eng > eng:/vol/vol18/sysadmin_tmp /mnt/mount1 nfs > ro,vers=3,rsize=8192,wsize=8192,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng > 0 0 > [root@adcgar05 mnt]# mount -o > rw,rsize=16384,wsize=16384,acregmin=1,acdirmin=1 > eng:/vol/vol18/site-config mount2 > mount.nfs: /mnt/mount2 is already mounted or busy > > This is not expected behavior. I expect the second mount attempt to > succeed. Notice that I'm mounting two completely different filesystems > (despite the fact that they're both on the same volume on the filer) > > This "fix" would actually make matters worse in our environment, because > we would no longer be "allowed" to mount multiple things from the same > volume from a filer, rendering NFS effectively useless to us. > > * * * * * * * * * * > Weird...somehow autofs seems to at least be able to do multiple mounts... > > [root@adcgar05 /]# cat /proc/mounts | grep vol18 > > ** Nothing mounted on eng:/vol/vol18... > > [root@adcgar05 /]# cd /tool/site-config > [root@adcgar05 site-config]# cat /proc/mounts | grep site-config > eng:/vol/vol18/site-config /tool/site-config nfs > rw,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng > 0 0 > > ** OK now eng:/vol/vol18/site-config is mounted with shown options > > [root@adcgar05 site-config]# cd /mnt > [root@adcgar05 mnt]# mount -t nfs -o ro eng:/vol/vol18/sysadmin_tmp > mount1 > mount.nfs: /mnt/mount1 is already mounted or busy > > ** Doh! Can't mount anything else off vol18 > > [root@adcgar05 mnt]# mount -t nfs -o rw eng:/vol/vol18/sysadmin_tmp > mount1 > mount.nfs: /mnt/mount1 is already mounted or busy > > ** Can't even mount anything else if given the same flags as previous > mount > > [root@adcgar05 mnt]# mount eng:/vol/vol18/site-config mount1 > mount.nfs: /mnt/mount1 is already mounted or busy > > ** Can't even mount without options! > > [root@adcgar05 mnt]# cd /tool/sysadmin_tmp > [root@adcgar05 sysadmin_tmp]# cat /proc/mounts | grep eng.*vol18 > eng:/vol/vol18/site-config /tool/site-config nfs > rw,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng > 0 0 > eng:/vol/vol18/sysadmin_tmp /tool/sysadmin_tmp nfs > rw,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng > 0 0 > > ** And yet, autofs seems to have had no trouble with it... > > [root@adcgar05 mnt]# ypmatch site-config auto.tool > -intr eng:/vol/vol18/& > [root@adcgar05 mnt]# ypmatch sysadmin_tmp auto.tool > -intr eng:/vol/vol18/& > > > This event sent from IssueTracker by pvn > issue 115656 >
*** Bug 248599 has been marked as a duplicate of this bug. ***
Verified by booting the installer from the 0822.1 tree and sucessfully mounted a share read-write.
*** Bug 243913 has been marked as a duplicate of this bug. ***
clearing requires_release_note flag, as duplicate bug is already being used to track doc changes for this.
*** Bug 311221 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2007-0959.html