Description of problem: Latest nfs-utils on my x86-64 server breaks ltsp on my i386 clients. Version-Release number of selected component (if applicable): nfs-utils-1.1.2-9 How reproducible: Always. Steps to Reproduce: 1. Have an uptodate x86-64 server 2. Setup ltsp with i386 clients 3. Boot a client Actual results: The clients can fetch vmlinuz and initrd off the server but bootup fails because the client fails to mount its nfs root. The text copied manually from the client screen: ... Red Hat nash version 6.0.52 starting mount: RPC: Authentication error: why = Failed (unspecified error) nfsmount: error mounting 172.31.100.254:/opt/ltsp/i386 on /sysroot as nfs: Bad file descriptor setuproot: moving /dev failed: No such file or directory setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory cp: error opening /sysroot/dev/.dhclient-eth0.leases: No such file or directory Mount failed for selinuxfs on /selinux: No such file or directory switchroot: mount failed: No such file or directory Expected results: Boot ltsp clients just fine. Additional info: Downgrading to nfs-utils-1.1.2-7.x86_64 on the server fixes it. Upgrading to nfs-utils-1.1.2-9 breaks clients again. I restarted nfs and nfslock after changing nfs-utils version and rebooted a client.
I get precisely the same messages, again on an x84_64 server running FC9, with the thin client running i386. It may be worth showing the output of exportfs -v here: /opt/ltsp <world>(ro,async,wdelay,no_root_squash,no_subtree_check) I've been unable to test downgrading to nfs-utils-1.1.2-7.x86_64 because it doesn't seem to be available anywhere I can find.
I have the same nfs entry: [root@db00 ~]# exportfs -v /opt/ltsp <world>(ro,async,wdelay,no_root_squash,no_subtree_check) I guess you also created it via ltsp-build-client. I found one non-up-to-date Fedora mirror that still carries the previous version as of January 17, 2009: http://fedora.patan.com.ar/linux/updates/testing/9/x86_64.newkey/ I saved both the x86_64 and src RPMs, I can attach them to this bug if needed.
http://kojipkgs.fedoraproject.org/packages/nfs-utils/ You can find all previous versions of packages here.
nfs-utils-1.1.4-6.fc10.x86_64 upgrade broke i386 client mounts as well.
nfs-utils-1.1.4-6.fc10.i386 upgraded on the server broke i386 client mounts. This update could have broken mounts from nash mount used by mkinitrd for NFS root boot.
Created attachment 329297 [details] wireshark packet capture wireshark packet capture from initrd DHCP to nash NFS mount request.
https://admin.fedoraproject.org/updates/F10/FEDORA-2009-0266 Axel Thimm pointed out that it seems to be denying mounts when the client IP address lacks a reverse DNS lookup. This was not a requirement in the past. This breaks working NFS client/server deployments.
Maybe this is happening due to * Sat Dec 20 2008 Steve Dickson <steved> 1.1.2-8 - Re-enabled and fixed/enhanced tcp wrappers. The clients' ips are not resolving, check the server's logs for lines like mountd[6866]: Warning: Client IP address '192.168.100.10' not found in host lookup mountd[6895]: connect from 192.168.100.10 to proc (0) in mountd: request from unauthorized host This affects F9 and F10 servers the same.
Yes, I have those messages when using 1.1.2-9. Fortunately, I already run named. Interestingly, after adding both my thin clients to DNS, only one of them started working. The other started working only after I added this line to /etc/hosts.allow: mountd: 172.31.100.
Yes... if mountd (or statd) cannot resolve an IP address (similar to sshd and other system daemons work) mounts (and locks) will be denied. This code (the tcp wrapper code) has always been enabled, it just never worked correctly. From Comment #9: > Interestingly, after adding > both my thin clients to DNS, only one of them started working So the one client that didn't start working failed with the same warning about the IP address not being found or was there some other problem?? So is this truly a Regression? Do we really want mountd/statd accepting connection from IP addresses it knows nothing about??
(In reply to comment #10) > So is this truly a Regression? Do we really want mountd/statd accepting > connection from IP addresses it knows nothing about?? Absolutely. If exported without any restrictions, then DNS is absolutely irrelevant. Same if IP adresses are used in /etc/exports. If have seen some pages saying that working DNS is required for NFS, but I cannot find such a MUST in the RFCs (1094, 1813).
(In reply to comment #10) > From Comment #9: > > Interestingly, after adding > > both my thin clients to DNS, only one of them started working > > So the one client that didn't start working failed with the same > warning about the IP address not being found or was there some other > problem?? Yes: ... Jan 18 11:15:00 db00 dhcpd: DHCPACK on 172.31.100.100 to 00:02:a5:59:88:24 via ltspbr0 Jan 18 11:15:02 db00 mountd[14303]: Warning: Client IP address '172.31.100.100' not found in host lookup Jan 18 11:15:02 db00 mountd[14465]: connect from 172.31.100.100 to proc (1) in mountd: request from unauthorized host The client printed the same errors as I wrote in the original error report.
(In reply to comment #4) > nfs-utils-1.1.4-6.fc10.x86_64 upgrade broke i386 client mounts as well. Works fine here. All ltsp terminals boot normally. We do have full reverse DNS for all terminals though.
nfs-utils-1.1.2-10.fc9 has been submitted as an update for Fedora 9. http://admin.fedoraproject.org/updates/nfs-utils-1.1.2-10.fc9
nfs-utils-1.1.4-7.fc10 has been submitted as an update for Fedora 10. http://admin.fedoraproject.org/updates/nfs-utils-1.1.4-7.fc10
(In reply to comment #14) > nfs-utils-1.1.2-10.fc9 has been submitted as an update for Fedora 9. > http://admin.fedoraproject.org/updates/nfs-utils-1.1.2-10.fc9 I tried this update but it doesn't work for me without using a reverse DNS. Same error occurs on the client and I got this message again: Jan 20 17:50:47 db00 mountd[28284]: authenticated mount request from 172.31.100.100:601 for /opt/ltsp/i386 (/opt/ltsp) It doesn't matter whether "mountd: 172.31.100." is in /etc/hosts.allow or not.
Here is some background information of how this happened. A long standing bug in NFS since August 2000 is that it did not support tcp wrappers /etc/hosts.deny and /etc/hosts.allow. This was fixed upstream, but then they discovered an issue. /etc/hosts.deny: mountd: badhost.example.com If DNS is down or the source IP address lacks a reverse lookup, then it has no way of knowing that it is meant to be denied. It allows the connection. This was considered to a be an unacceptable bug. If someone specifies a hostname, from a security perspective they expect the failure case to err to the side of caution and deny by default if cannot affirm that the incoming client is not a listed denied hostname. Thus "deny upon reverse DNS failure" became the default behavior in nfs-utils and this ticket. I disagreed strongly. I believe that that should be NOTABUG for several reasons: * BAD POLICY and MISCONFIGURATION. TCP wrappers is behaving exactly how it is defined in policy. Hostname in hosts.deny (itself always a bad idea) is dependent on the DNS server to be properly configured and operating. Failure due to hostnames in /etc/hosts.deny is MISCONFIGURATION. If they are really concerned about unknown clients connecting to that service, then they should use a wildcard like "mountd: ALL" and allow specific hosts or IP ranges in /etc/hosts.allow. * Host names in iptables or tcp wrappers rules are ALWAYS a bad idea because you are relying upon an unreliable and insecure external data source that is DNS. DNS can be too easily spoofed, hijacked, denial of service attacked, and it lacks any semblance of authentication or encryption. * It is wrong for a service wrapped by tcp wrappers to over-engineer itself to workaround misconfiguration on the part of the sysadmin. * It is inconsistent with the behavior of any other service wrapped by tcp wrappers. "sshd: badhost.example.com" allows incoming connections if reverse DNS lookup fails. Does this mean sshd and every other service wrapped by tcp wrappers should also be modified? * This is inconsistent with iptables. "iptables -A INPUT -p tcp --dport 22 -s badhost.example.com -j REJECT" might also fail to reject an incoming connection under similar DNS-related conditions. It would be clearly wrong for sshd to second-guess and parse iptables rules, and make its own decision based its own reverse DNS query matching hostnames found in those iptables rules. Why is it OK to second guess tcp wrappers but not iptables? * Even if we agree that this is a bug that should be fixed and not simply sysadmin misconfiguration, the correct layer to fix it is in tcp wrappers itself, not the individual wrapped services. We had a lengthy discussion and have come to reluctant agreement on how to proceed from here. It was decided by the filesystem people that even if it is only the result of misconfiguration, nfs-utils should protect and deny by default. Steve is working on the next build of nfs-utils with slightly different logic. It will behave exactly as it did before this change (no tcp wrappers or deny on reverse DNS lookup failure). It will only enable tcp wrapper and the new denial behavior if any mountd, statd or wildcard lines appear in /etc/hosts.deny or /etc/hosts.allow. We will hopefully have a new build tomorrow. Workarounds until this is fixed... ================================== 1) Remove tcpwrappers (if you don't need quotas this might be OK) 2) Downgrade nfs-utils to these versions: http://kojipkgs.fedoraproject.org/packages/nfs-utils/1.1.2/7.fc9/ http://kojipkgs.fedoraproject.org/packages/nfs-utils/1.1.4/4.fc10/ 3) Stop using NFS entirely. LTSP users can switch to NBD instead, which is a little more work, but clients boot a little faster. https://fedorahosted.org/k12linux/NBDRootConfiguration
nfs-utils-1.1.4-7.fc10 has been pushed to the Fedora 10 stable repository. If problems still persist, please make note of it in this bug report.
nfs-utils-1.1.2-10.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report.
Please do not close this bug yet. I strongly believe the way it is fixed here and upstream is incorrect. This is being discussed on upstream linux-nfs list.
(In reply to comment #18) > nfs-utils-1.1.4-7.fc10 has been pushed to the Fedora 10 stable repository. If > problems still persist, please make note of it in this bug report. The problem with client IPs w/o DNS PTR entries are still there.
This is largely undone in the resolution of Bug #480223, patch forthcoming. Closing duplicate. *** This bug has been marked as a duplicate of bug 480223 ***