From Bugzilla Helper: User-Agent: Mozilla/4.77 [en] (X11; U; Linux 2.2.19-7.0.8 i686) Description of problem: Since upgrading RH 6.2 NFS server to kernel 2.2.19-6.2.7 and nfs-utils-0.3.1-0.6.x.1, RH 7 clients with kernel 2.2.19-7.0.1 and nfs-utils-0.3.1-7 have been getting spurious NFS failures when attempting valid mount requests. Some mounts will work, others will fail, when one fails it will then repeatedly fail until an "exportfs -r" is executed on teh NFS server. How reproducible: Sometimes Steps to Reproduce: 1. Export filesystems on RH 6.2 server with 2.2.19-6.2.7 (nfs-utils-0.3.1-0.6.x.1) 2. Attempt mounts from RH 7 clients with 2.2.19-7.0.1 (nfs-utils-0.3.1-7 3. Mounts will sometimes fail. If it fails once, will continue to fail. Actual Results: Fails and the following is logged to messages on server: Jul 26 14:21:40 alice rpc.mountd: authenticated mount request from 10.21.0.75:903 for /home (/home) Jul 26 14:21:40 alice rpc.mountd: getfh failed: Operation not permitted Expected Results: Mount should work. Additional info: Hopefully this is just some sort of stupidity on my part, but I don't see anything wrong with either the server or client setup and this problem just started happening after upgrade to 6.2 NFS server. If this is a bug, maybe a problem with the RH7 clients? (Does not seem to be affecting ay other clients.) Here is my /etc/exports: /home *.ma.virata.com(rw) /home 10.21.0.*(rw) /home/emweb *.cam.virata.com(ro) /home/emweb *.ral.virata.com(ro) /home/emweb *.sb.virata.com(ro) /home/emweb *.sc.virata.com(ro) /home/emweb *.ta.virata.com(ro) /usr/local *.ma.virata.com(rw) /usr/lib *.ma.virata.com(ro) Here is /var/lib/nfs/etab: /home/emweb *.ta.virata.com(ro,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /home/emweb *.sc.virata.com(ro,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /home/emweb *.sb.virata.com(ro,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /home/emweb *.ral.virata.com(ro,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /home/emweb *.cam.virata.com(ro,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /usr/local *.ma.virata.com(rw,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /usr/lib *.ma.virata.com(ro,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /home 10.21.0.*(rw,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) /home *.ma.virata.com(rw,async,wdelay,hide,secure,root_squash,no_all_squash,subtree_check,secure_locks,mapping=identity,anonuid=-2,anongid=-2) Here it /var/lib/nfs/rmtab: 10.21.0.140:/home:0x00000001 tweedledum.ma.virata.com:/usr/local/flexlm:0x00000001 tweedledum.ma.virata.com:/home:0x00000001 tweedledum.ma.virata.com:/usr/local/src:0x00000001 tweedledum.ma.virata.com:/usr/local/share:0x00000001 fish.ma.virata.com:/home:0x00000001 caterpillar.ma.virata.com:/home:0x00000001 caterpillar.ma.virata.com:/usr/local/src:0x00000001 caterpillar.ma.virata.com:/usr/local/share:0x00000001 caterpillar.ma.virata.com:/usr/local/man:0x00000001 caterpillar.ma.virata.com:/usr/local/info:0x00000001 caterpillar.ma.virata.com:/usr/local/i686-glibc:0x00000001 monitor.ma.virata.com:/home/emweb:0x00000001 queenold.ma.virata.com:/usr/local/flexlm:0x00000001 queenold.ma.virata.com:/home:0x00000001 tweedledee.ma.virata.com:/home:0x00000001 tweedledee.ma.virata.com:/usr/local/flexlm:0x00000001 tweedledee.ma.virata.com:/usr/local/src:0x00000001 tweedledee.ma.virata.com:/usr/local/share:0x00000001 dhcp38.ma.virata.com:/home:0x00000001 dhcp38.ma.virata.com:/usr/local/src:0x00000002 dhcp38.ma.virata.com:/usr/local/share:0x00000002 dhcp38.ma.virata.com:/usr/local/man:0x00000002 dhcp38.ma.virata.com:/usr/local/info:0x00000002 dhcp38.ma.virata.com:/usr/local/i686-glibc:0x00000002 dhcp23.ma.virata.com:/home:0x00000001 dhcp23.ma.virata.com:/usr/local/src:0x00000001 dhcp23.ma.virata.com:/usr/local/share:0x00000001 dhcp23.ma.virata.com:/usr/local/man:0x00000001 dhcp23.ma.virata.com:/usr/local/info:0x00000001 dhcp23.ma.virata.com:/usr/local/i686-glibc:0x00000001 dhcp23.ma.virata.com:/usr/local/flexlm:0x00000001 queen-of-hearts.ma.virata.com:/home:0x00000001 queen-of-hearts.ma.virata.com:/usr/local/flexlm:0x00000001 dhcp93.ma.virata.com:/home:0x00000001 dhcp93.ma.virata.com:/usr/local/bin:0x00000001 dhcp84.ma.virata.com:/home:0x00000001 dhcp84.ma.virata.com:/usr/local/bin:0x00000006 virata.com:/home/pradcliff:0x00000001 10.21.0.137:/home:0x00000001 10.21.0.81:/home:0x00000001 dhcp81.ma.virata.com:/usr/local/bin:0x00000001 10.21.0.84:/home:0x00000005 dhcp104.ma.virata.com:/usr/local/bin:0x00000003 dhcp104.ma.virata.com:/home/pradcliff:0x00000002 dhcp19.ma.virata.com:/home:0x00000001 dhcp75.ma.virata.com:/usr/local:0x00000003 10.21.0.38:/home:0x00000001 10.21.0.20:/home:0x00000007 dhcp75.ma.virata.com:/usr/lib:0x00000001 10.21.0.250:/home:0x00000003 10.21.0.104:/home/pradcliff:0x00000001 10.21.0.74:/home:0x00000001 Here is an strace of rpc.mountd of a failing request: select(1024, [3 4], NULL, NULL, NULL) = 1 (in [3]) recvfrom(3, "U;&5\0\0\0\0\0\0\0\2\0\1\206\245\0\0\0\3\0\0\0\1\0\0\0\1\0\0\0D;`[m\0\0\0\24dhcp75.ma.virata.com\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\0\0\0\0\1\0\0\0\2\0\0\0\3\0\0\0\4\0\0\0\6\0\0\0\n\0\0\0\0\0\0\0\0\0\0\0\5/home\0\0\0", 8800, 0, {sin_family=AF_INET, sin_port=htons(864), sin_addr=inet_addr("10.21.0.75")}}, [16]) = 120 stat("/var/lib/nfs/etab", {st_mode=S_IFREG|0644, st_size=1321, ...}) = 0 lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 time([996170605]) = 996170605 rt_sigaction(SIGPIPE, {0x400de978, [], 0x4000000}, {SIG_IGN}, 8) = 0 send(6, "<13>Jul 26 14:03:25 rpc.mountd: authenticated mount request from 10.21.0.75:864 for /home (/home)\n", 98, 0) = 98 rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0 stat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 nfsservctl(0x8, 0xbfffdb9c, 0x8054ba0) = -1 EINVAL (Invalid argument) nfsservctl(0x7, 0xbfffdba0, 0x8054ae0) = -1 EPERM (Operation not permitted) time([996170605]) = 996170605 rt_sigaction(SIGPIPE, {0x400de978, [], 0x4000000}, {SIG_IGN}, 8) = 0 send(6, "<12>Jul 26 14:03:25 rpc.mountd: getfh failed: Operation not permitted\n", 70, 0) = 70 rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0 sendto(3, "U;&5\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\r", 28, 0, {sin_family=AF_INET, sin_port=htons(864), sin_addr=inet_addr("10.21.0.75")}}, 16) = 28 select(1024, [3 4], NULL, NULL, NULL <unfinished ...> Doing an "exportfs -r" on the NFS server throws this error: [root@alice /root]# exportfs -r dhcp75.ma.virata.com:/home: Invalid argument This is the client which is having problems. "exportfs -r" seems to "fix" the problem, at least temporarily. I have strace output of the above command if you want to see it. Could not make much of it myself. The mix of IP address and hostnames in the rmtab seems strange to me. All of the hosts listed as IP addresses have valid forward and rev DNS entries, would expect everyone to show up with their *.ma.virata.com name in that file.
I also
I also have this issue. Again, when I upgraded from 6.2 to 7.1, I can not mount any NFS shares from the new 7.1 server. Interestingly, the / partition (which was formatted during install is effected, as well as two legacy partitions, which were not formatted. I havce 6.2 and 7.0 clients that will not pount the NFS shares from the 7.1 server. The clients report a 'Permission denied" error. I am suspecting a kernel bug, any ideas ?
Well I fixed it. (for now...) stopped NFS deleted /var/lib/nfs/etab, rmtab, mtab started nfs all worked. I ??SUSPECT?? thatit is something to do with the status information from the old clients, rebuilding this data onto the new server. I am no nfs expert, but you get the drift.
Likewise to richard.au, problem went away after: clients unmount stop NFS on server delete /var/lib/nfs/etab,rmtab,mtab start NFS on server clients mount I've concluded: Only clients doing NFS v3 were having this problem. No NFS v2 clients saw any problems. Further, problem clients had been doing NFS v2 prior to the server upgrade since previous kernel did not implement NFS v3. I suspect that use of the following syntax may be involved with the problem: /home 10.21.0.*(rw) My reading of man page for "exports", above should be legal, but perhaps I'm wrong and you must use the net/mask syntax like the following? /home 10.21.0.0/24(rw) If legal, I suspect the 2.2.19-6.2.7 kernel is doing something wrong here, not matching wildcards properly when '10.21.0.*' systax is used, resulting in "permission denied" we see from nfsservctl. Will test this theory more when I get some time. Worth noting that man page for "nfsservctl" needs updating, should include 7 and 8 as valid args of 'cmd'.
Sounds like a bug, but it was sooo long ago... Bob liked to collect bugs, I see.