Bug 480420 - [REGRESSION] nfs-utils-1.1.2-9 breaks NFS mounts where client reverse DNS lookup fails
[REGRESSION] nfs-utils-1.1.2-9 breaks NFS mounts where client reverse DNS loo...
Status: CLOSED DUPLICATE of bug 480223
Product: Fedora
Classification: Fedora
Component: nfs-utils (Show other bugs)
9
All Linux
urgent Severity urgent
: ---
: ---
Assigned To: Steve Dickson
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks: K12LTSP
  Show dependency treegraph
 
Reported: 2009-01-16 17:57 EST by Zoltan Boszormenyi
Modified: 2009-01-29 16:09 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-29 16:09:34 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
wireshark packet capture (3.43 KB, application/octet-stream)
2009-01-18 02:39 EST, Warren Togami
no flags Details

  None (edit)
Description Zoltan Boszormenyi 2009-01-16 17:57:22 EST
Description of problem:

Latest nfs-utils on my x86-64 server breaks ltsp on my i386 clients.

Version-Release number of selected component (if applicable):

nfs-utils-1.1.2-9

How reproducible:

Always.

Steps to Reproduce:
1. Have an uptodate x86-64 server
2. Setup ltsp with i386 clients
3. Boot a client
  
Actual results:

The clients can fetch vmlinuz and initrd off the server but bootup fails
because the client fails to mount its nfs root. The text copied manually
from the client screen:

...
Red Hat nash version 6.0.52 starting
mount: RPC: Authentication error: why = Failed (unspecified error)
nfsmount: error mounting 172.31.100.254:/opt/ltsp/i386 on /sysroot as nfs: Bad file descriptor
setuproot: moving /dev failed: No such file or directory
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
cp: error opening /sysroot/dev/.dhclient-eth0.leases: No such file or directory
Mount failed for selinuxfs on /selinux: No such file or directory
switchroot: mount failed: No such file or directory

Expected results:

Boot ltsp clients just fine.

Additional info:

Downgrading to nfs-utils-1.1.2-7.x86_64 on the server fixes it.
Upgrading to nfs-utils-1.1.2-9 breaks clients again.
I restarted nfs and nfslock after changing nfs-utils version and rebooted a client.
Comment 1 Brian Fristensky 2009-01-17 11:42:27 EST
I get precisely the same messages, again on an x84_64 server running FC9, with
the thin client running i386. It may be worth showing the output
of exportfs -v here:

/opt/ltsp     	<world>(ro,async,wdelay,no_root_squash,no_subtree_check)

I've been unable to test downgrading to nfs-utils-1.1.2-7.x86_64 because
it doesn't seem to be available anywhere I can find.
Comment 2 Zoltan Boszormenyi 2009-01-17 11:59:29 EST
I have the same nfs entry:

[root@db00 ~]# exportfs -v
/opt/ltsp     	<world>(ro,async,wdelay,no_root_squash,no_subtree_check)

I guess you also created it via ltsp-build-client. I found one non-up-to-date
Fedora mirror that still carries the previous version as of January 17, 2009:
http://fedora.patan.com.ar/linux/updates/testing/9/x86_64.newkey/

I saved both the x86_64 and src RPMs, I can attach them to this bug if needed.
Comment 3 Warren Togami 2009-01-18 01:46:34 EST
http://kojipkgs.fedoraproject.org/packages/nfs-utils/
You can find all previous versions of packages here.
Comment 4 Warren Togami 2009-01-18 02:00:07 EST
nfs-utils-1.1.4-6.fc10.x86_64 upgrade broke i386 client mounts as well.
Comment 5 Warren Togami 2009-01-18 02:12:18 EST
nfs-utils-1.1.4-6.fc10.i386 upgraded on the server broke i386 client mounts.  This update could have broken mounts from nash mount used by mkinitrd for NFS root boot.
Comment 6 Warren Togami 2009-01-18 02:39:19 EST
Created attachment 329297 [details]
wireshark packet capture

wireshark packet capture from initrd DHCP to nash NFS mount request.
Comment 7 Warren Togami 2009-01-18 03:46:39 EST
https://admin.fedoraproject.org/updates/F10/FEDORA-2009-0266
Axel Thimm pointed out that it seems to be denying mounts when the client IP address lacks a reverse DNS lookup.  This was not a requirement in the past.  This breaks working NFS client/server deployments.
Comment 8 Axel Thimm 2009-01-18 03:59:45 EST
Maybe this is happening due to

* Sat Dec 20 2008 Steve Dickson <steved@redhat.com> 1.1.2-8
- Re-enabled and fixed/enhanced tcp wrappers.

The clients' ips are not resolving, check the server's logs for lines like

mountd[6866]: Warning: Client IP address '192.168.100.10' not found in host lookup
mountd[6895]: connect from 192.168.100.10 to proc (0) in mountd: request from unauthorized host

This affects F9 and F10 servers the same.
Comment 9 Zoltan Boszormenyi 2009-01-18 06:27:46 EST
Yes, I have those messages when using 1.1.2-9.
Fortunately, I already run named. Interestingly, after adding
both my thin clients to DNS, only one of them started working.
The other started working only after I added this line to /etc/hosts.allow:
mountd:     172.31.100.
Comment 10 Steve Dickson 2009-01-19 08:21:51 EST
Yes... if mountd (or statd) cannot resolve an IP address (similar to sshd and
other system daemons work) mounts (and locks) will be denied. 

This code (the tcp wrapper code) has always been enabled, it just never
worked correctly. 

From Comment #9:
> Interestingly, after adding
> both my thin clients to DNS, only one of them started working

So the one client that didn't start working failed with the same
warning about the IP address not being found or was there some other
problem??

So is this truly a Regression? Do we really want mountd/statd accepting
connection from IP addresses it knows nothing about??
Comment 11 Markus Schade 2009-01-19 08:50:40 EST
(In reply to comment #10)
> So is this truly a Regression? Do we really want mountd/statd accepting
> connection from IP addresses it knows nothing about??

Absolutely. If exported without any restrictions, then DNS is absolutely irrelevant. Same if IP adresses are used in /etc/exports.

If have seen some pages saying that working DNS is required for NFS, but I cannot find such a MUST in the RFCs (1094, 1813).
Comment 12 Zoltan Boszormenyi 2009-01-19 08:59:48 EST
(In reply to comment #10)
> From Comment #9:
> > Interestingly, after adding
> > both my thin clients to DNS, only one of them started working
> 
> So the one client that didn't start working failed with the same
> warning about the IP address not being found or was there some other
> problem??

Yes:
...
Jan 18 11:15:00 db00 dhcpd: DHCPACK on 172.31.100.100 to 00:02:a5:59:88:24 via ltspbr0
Jan 18 11:15:02 db00 mountd[14303]: Warning: Client IP address '172.31.100.100' not found in host lookup
Jan 18 11:15:02 db00 mountd[14465]: connect from 172.31.100.100 to proc (1) in mountd: request from unauthorized host

The client printed the same errors as I wrote in the original error report.
Comment 13 Michael Osborne 2009-01-19 17:06:31 EST
(In reply to comment #4)
> nfs-utils-1.1.4-6.fc10.x86_64 upgrade broke i386 client mounts as well.

Works fine here. All ltsp terminals boot normally. We do have full reverse DNS for all terminals though.
Comment 14 Fedora Update System 2009-01-20 07:53:06 EST
nfs-utils-1.1.2-10.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/nfs-utils-1.1.2-10.fc9
Comment 15 Fedora Update System 2009-01-20 07:55:16 EST
nfs-utils-1.1.4-7.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/nfs-utils-1.1.4-7.fc10
Comment 16 Zoltan Boszormenyi 2009-01-20 12:02:52 EST
(In reply to comment #14)
> nfs-utils-1.1.2-10.fc9 has been submitted as an update for Fedora 9.
> http://admin.fedoraproject.org/updates/nfs-utils-1.1.2-10.fc9

I tried this update but it doesn't work for me without using a reverse DNS.
Same error occurs on the client and I got this message again:
Jan 20 17:50:47 db00 mountd[28284]: authenticated mount request from 172.31.100.100:601 for /opt/ltsp/i386 (/opt/ltsp)

It doesn't matter whether "mountd: 172.31.100." is in /etc/hosts.allow or not.
Comment 17 Warren Togami 2009-01-20 17:02:20 EST
Here is some background information of how this happened.  A long standing bug in NFS since August 2000 is that it did not support tcp wrappers /etc/hosts.deny and /etc/hosts.allow.  This was fixed upstream, but then they discovered an issue.

/etc/hosts.deny:
mountd: badhost.example.com

If DNS is down or the source IP address lacks a reverse lookup, then it has no way of knowing that it is meant to be denied.  It allows the connection.  This was considered to a be an unacceptable bug.  If someone specifies a hostname, from a security perspective they expect the failure case to err to the side of caution and deny by default if cannot affirm that the incoming client is not a listed denied hostname.  Thus "deny upon reverse DNS failure" became the default behavior in nfs-utils and this ticket.

I disagreed strongly.  I believe that that should be NOTABUG for several reasons:

* BAD POLICY and MISCONFIGURATION.
TCP wrappers is behaving exactly how it is defined in policy.  Hostname in hosts.deny (itself always a bad idea) is dependent on the DNS server to be properly configured and operating.  Failure due to hostnames in /etc/hosts.deny is MISCONFIGURATION.  If they are really concerned about unknown clients connecting to that service, then they should use a wildcard like "mountd: ALL" and allow specific hosts or IP ranges in /etc/hosts.allow.
* Host names in iptables or tcp wrappers rules are ALWAYS a bad idea because you are relying upon an unreliable and insecure external data source that is DNS.  DNS can be too easily spoofed, hijacked, denial of service attacked, and it lacks any semblance of authentication or encryption.
* It is wrong for a service wrapped by tcp wrappers to over-engineer itself to workaround misconfiguration on the part of the sysadmin. 
* It is inconsistent with the behavior of any other service wrapped by tcp wrappers.  "sshd: badhost.example.com" allows incoming connections if reverse DNS lookup fails.  Does this mean sshd and every other service wrapped by tcp wrappers should also be modified?
* This is inconsistent with iptables.  "iptables -A INPUT -p tcp --dport 22 -s badhost.example.com -j REJECT" might also fail to reject an incoming connection under similar DNS-related conditions.  It would be clearly wrong for sshd to second-guess and parse iptables rules, and make its own decision based its own reverse DNS query matching hostnames found in those iptables rules.  Why is it OK to second guess tcp wrappers but not iptables? 
* Even if we agree that this is a bug that should be fixed and not simply sysadmin misconfiguration, the correct layer to fix it is in tcp wrappers itself, not the individual wrapped services.

We had a lengthy discussion and have come to reluctant agreement on how to proceed from here.  It was decided by the filesystem people that even if it is only the result of misconfiguration, nfs-utils should protect and deny by default.

Steve is working on the next build of nfs-utils with slightly different logic.  It will behave exactly as it did before this change (no tcp wrappers or deny on reverse DNS lookup failure).  It will only enable tcp wrapper and the new denial behavior if any mountd, statd or wildcard lines appear in /etc/hosts.deny or /etc/hosts.allow.

We will hopefully have a new build tomorrow.

Workarounds until this is fixed...
==================================
1) Remove tcpwrappers (if you don't need quotas this might be OK)
2) Downgrade nfs-utils to these versions:
http://kojipkgs.fedoraproject.org/packages/nfs-utils/1.1.2/7.fc9/
http://kojipkgs.fedoraproject.org/packages/nfs-utils/1.1.4/4.fc10/ 
3) Stop using NFS entirely.  LTSP users can switch to NBD instead, which is a little more work, but clients boot a little faster.
https://fedorahosted.org/k12linux/NBDRootConfiguration
Comment 18 Fedora Update System 2009-01-23 21:38:14 EST
nfs-utils-1.1.4-7.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 19 Fedora Update System 2009-01-23 21:41:02 EST
nfs-utils-1.1.2-10.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 20 Warren Togami 2009-01-28 22:22:58 EST
Please do not close this bug yet.  I strongly believe the way it is fixed here and upstream is incorrect.  This is being discussed on upstream linux-nfs list.
Comment 21 Axel Thimm 2009-01-29 01:45:59 EST
(In reply to comment #18)
> nfs-utils-1.1.4-7.fc10 has been pushed to the Fedora 10 stable repository.  If
> problems still persist, please make note of it in this bug report.

The problem with client IPs w/o DNS PTR entries are still there.
Comment 22 Warren Togami 2009-01-29 16:09:34 EST
This is largely undone in the resolution of Bug #480223, patch forthcoming.  Closing duplicate.

*** This bug has been marked as a duplicate of bug 480223 ***

Note You need to log in before you can comment on or make changes to this bug.