Bug 742267 - RHEV 3 can no longer use an NFS export which was usable by RHEV 2.2
Summary: RHEV 3 can no longer use an NFS export which was usable by RHEV 2.2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: vdsm
Version: 6.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Dan Kenigsberg
QA Contact: yeylon@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-09-29 14:41 UTC by Matthew Booth
Modified: 2016-04-18 06:42 UTC (History)
15 users (show)

Fixed In Version: vdsm-4.9-106
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-06 07:29:01 UTC
Target Upstream Version:


Attachments (Terms of Use)
vdsm.log for attempt to mount NFS export domain (3.83 KB, text/plain)
2011-09-30 09:06 UTC, Matthew Booth
no flags Details
strace of vdsm during attempt to mount (17.04 KB, application/x-xz)
2011-09-30 09:08 UTC, Matthew Booth
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2011:1782 0 normal SHIPPED_LIVE new packages: vdsm 2011-12-06 11:55:51 UTC

Description Matthew Booth 2011-09-29 14:41:29 UTC
Description of problem:
I have an NFS export on my NAS in my lab setup which I have been using as an export storage domain in RHEV 2.2 for some time. It has default ownership, but is world writeable.

I installed RHEV 3 and attempted to add the same NFS export. This failed with the following error in rhevm.log:

The connection with details qnap.rhev.marston:/rhev-export1 failed because of error code 469 and error message is: permission settings on the specified path do not allow access to the storage.
verify permission settings on the specified storage path.

This is clearly wrong, as the export has been in use for some time by RHEV 2.2. Additionally, I manually confirmed that the hypervisor node can mount the nfs export, and that uid 36 can write to it. Despite this, changing the ownership of the export to 36:36 makes the above error go away. This suggests that something somewhere is checking ownership, and when finding that it is not 36:36 is assuming that it is not writeable.

Version-Release number of selected component (if applicable):
vdsm-4.9-96.1.el6.x86_64
rhevm-3.0.0_0001-41.el6.x86_64

Comment 2 Dan Kenigsberg 2011-09-29 18:54:06 UTC
what were the ownership and permissions of the export?
And please attach vdsm.log.

Comment 3 Matthew Booth 2011-09-30 09:04:29 UTC
(In reply to comment #2)
> what were the ownership and permissions of the export?

Ownership was 0:0. Permissions are 0777.

Comment 4 Matthew Booth 2011-09-30 09:06:16 UTC
Created attachment 525720 [details]
vdsm.log for attempt to mount NFS export domain

Comment 5 Matthew Booth 2011-09-30 09:08:33 UTC
Created attachment 525726 [details]
strace of vdsm during attempt to mount

This is the output of strace on the relevant vdsm process during a failed attempt to mount the NFS export storage domain.

Comment 6 Dan Kenigsberg 2011-10-02 12:53:10 UTC
Mathew, your analysis is correct - vdsm validates that the NFS export is owned by 36:36. I am a bit surprised that rhev-2.2 was ok with 0:0 ownership. I'm pretty sure that we had a fat release note about nfs exports having to be owned by 36:36.

I believe that the stricter behavior of 
http://git.fedorahosted.org/git/?p=vdsm.git;a=commitdiff;h=e9caf1f1f98b226a984a4f126a20440835cc7bad
should be maintained.

Why should we not CLOSE|NOTABUG?

Comment 7 Andrew Cathrow 2011-10-02 13:33:26 UTC
ACK - We've always required 36:36
Closing as not a bug

Comment 8 Matthew Booth 2011-10-03 09:20:19 UTC
(In reply to comment #7)
> ACK - We've always required 36:36
> Closing as not a bug

Andrew,

We should not ignore the fact that this change introduces a regression, actively misdirects the user, is not well thought through, and it is simple to fix. The regression: something which used to work no longer works. There is no possible benefit to this change from the user's perspective:

1. User has existing 36:36 export domain. Still works. NO CHANGE.
2. User has existing 0:0 export domain. No longer works. WORSE.

We should think extremely hard before making intentional software changes which can only possibly impact the user negatively. In this particular case the change is not technically essential: if the ownership check were removed, everything else would function normally.

This particular regression is compounded by the fact that the error message is incorrect. It says: 'permission settings on the specified path
do not allow access to the storage'. This is obviously wrong because:

1. It worked under 2.2.
2. I manually checked that it worked.

This led me to ignore it entirely, assuming the real issue was something unrelated. This wasted more than a day. I originally thought I was hitting another previously reported issue relating to NFS from RHEV-H, and ended up re-installing a RHEL hypervisor. A more accurate error message could have said: 'RHEV requires that NFS exports are owned by 36:36'. This may have been unnecessarily frustrating, but at least it would have given me confidence that what I was being told was correct and would have led me directly to the solution. It would have avoided the necessity of posting a Guru Meditation to rhev-beta :)

The real question is why the change was made. I can only think of 2 potential reasons.

If the intention of this change is force better security, it hasn't worked. My NFS export is still world writable for all the same reasons which made me set it up that way in the first place; I was just forced to unnecessarily change its ownership. If you want to check for secure permissions, you also have to check permissions! If you aren't checking permissions, you don't need to check ownership.

If the intention of this change is to ensure that the location is readable and writable, the documented way to achieve this is the open(2) system call (see access(2)).

Having personally wasted over a day on this, it's clear there's an issue to be addressed here. It's not difficult to fix, either:

If the purpose of the change is to enforce security policy, the ownership test needs to be combined with a permissions test, and both tests should have their own, separate error messages. However, there would be little purpose in doing this. NFS is notorious for having No F...... Security, and we can't test for the truly egregious NFS misconfiguration errors.

If the purpose of the change is to ensure that the export location is readable and writable, the test should be changed to use an open(2) call. This would be less code, would always be correct, and would not have generated an error in this case.

If the purpose of the change is to enforce what was written in the documentation... the wording of the documentation should be updated.

Please consider re-opening this bug. I've been hit over the head with it; the fact that 3 people were able to tell me on the list to check ownership, despite that making no sense, suggests that I'm not the first.

Comment 9 Dan Kenigsberg 2011-10-04 12:42:29 UTC
http://gerrit.usersys.redhat.com/983

Comment 10 Dan Kenigsberg 2011-10-04 12:51:55 UTC
Bug reopened per Itamar's email

On Mon, Oct 03, 2011 at 01:03:57PM +0200, Itamar Heim wrote:
...
> 
> 3.0.0
>       Beta3 is on RHEL 6.2
>      vdsm 105 to re-spin on nfs mount issues

Comment 11 Dan Kenigsberg 2011-10-09 08:25:22 UTC
*** Bug 740956 has been marked as a duplicate of this bug. ***

Comment 13 Tomas Dosek 2011-10-18 10:30:50 UTC
Verified - vdsm-4.9-108 - Owner and group validation was removed, 2.2. created NFS export with rights set to 0777, owned by root:root can now be used with no problem.

Comment 14 errata-xmlrpc 2011-12-06 07:29:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1782.html

Comment 15 Eldad Marciano 2013-12-18 15:41:19 UTC

(In reply to Tomas Dosek from comment #13)
> Verified - vdsm-4.9-108 - Owner and group validation was removed, 2.2.
> created NFS export with rights set to 0777, owned by root:root can now be
> used with no problem.

Hi,
Bug regression!
the bug stil reproduce on vdsm 4.13.2

i had the same issue, when the NFS ownership change to 36:36 the problem resolved.


Note You need to log in before you can comment on or make changes to this bug.