Bug 1261178 - oVirt mounts 10GigE NFS with UDP
oVirt mounts 10GigE NFS with UDP
Status: CLOSED WORKSFORME
Product: oVirt
Classification: Community
Component: ovirt-engine-core (Show other bugs)
3.5
Unspecified Unspecified
unspecified Severity medium
: m1
: 3.6.0
Assigned To: Allon Mureinik
Pavel Stehlik
storage
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-08 15:23 EDT by Markus Stockhausen
Modified: 2016-03-10 01:17 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-13 04:17:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Markus Stockhausen 2015-09-08 15:23:55 EDT
Description of problem:

We added a new NFS domain 2 to an Ovirt datacenter 2 weeks ago. The performance of the new domain is worse than other NFS domains. After further analysis we found out that for some reason that domain is mounted with protocol udp instead of tcp.

Version-Release number of selected component (if applicable):

OVirt 3.5.3

How reproducible:

dont know

Steps to Reproduce:

add NFS domain

Actual results:

domain is mounted with protocol udp

Expected results:

domain should be mounted with protocol tcp

Additional info:

This is an old mount point on an ovirt node. The domain was added a year ago.

10.10.30.252:/var/nas2/OVirtIB on /rhev/data-center/mnt/10.10.30.252:_var_nas2_OVirtIB type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.10.30.252,mountvers=3,mountport=60268,mountproto=udp,local_lock=none,addr=10.10.30.252)

This is the mount point of a domain we added 2 weeks ago (on the same node)

10.10.30.254:/var/nas4/OVirtIB on /rhev/data-center/mnt/10.10.30.254:_var_nas4_OVirtIB type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,nosharecache,proto=udp,timeo=600,retrans=6,sec=sys,mountaddr=10.10.30.254,mountvers=3,mountport=48383,mountproto=udp,local_lock=none,addr=10.10.30.254)


Not only mounting the data with the wrong protocol also the rsize/wsize parameters have beend dramatically reduced.
Comment 1 Markus Stockhausen 2015-09-08 15:25:38 EDT
In both cases the OVirt GUI dialogue for adding the NFS domain was used without specifiying additional parameters.
Comment 2 Allon Mureinik 2015-09-09 04:00:10 EDT
Also, note that they are BOTH using udp. 

Regardless, oVirt shouldn't be messing with parameter you don't specify, so I'm guessing the issue is somewhere else, but let's start the investigation from the startpoint. Markus, can you please attach the engine and VDSM logs?
Comment 3 Allon Mureinik 2015-09-09 04:08:33 EDT
Adam - as this week's QA contact, can you take a look please?
Comment 4 Markus Stockhausen 2015-09-09 04:18:04 EDT
(In reply to Allon Mureinik from comment #2)
> Also, note that they are BOTH using udp. 
> 
> Regardless, oVirt shouldn't be messing with parameter you don't specify, so
> I'm guessing the issue is somewhere else, but let's start the investigation
> from the startpoint. Markus, can you please attach the engine and VDSM logs?

Clarification:

mountproto=UDP has always been there

only proto=UDP has changed recently (from proto=TCP)

Markus
Comment 5 Markus Stockhausen 2015-09-09 04:34:29 EDT
Ok, we just destroyed the domain and wiped the contents. After reattaching it is mounted correctly. 

The process broke once during initial setup because of missing access rights. I'll try to search the logs where the error came from.
Comment 6 Allon Mureinik 2015-09-09 05:58:15 EDT
(In reply to Markus Stockhausen from comment #4)
> (In reply to Allon Mureinik from comment #2)
> > Also, note that they are BOTH using udp. 
> > 
> > Regardless, oVirt shouldn't be messing with parameter you don't specify, so
> > I'm guessing the issue is somewhere else, but let's start the investigation
> > from the startpoint. Markus, can you please attach the engine and VDSM logs?
> 
> Clarification:
> 
> mountproto=UDP has always been there
> 
> only proto=UDP has changed recently (from proto=TCP)
Missed that, thanks for clarifying!



(In reply to Markus Stockhausen from comment #5)
> Ok, we just destroyed the domain and wiped the contents. After reattaching
> it is mounted correctly. 
> 
> The process broke once during initial setup because of missing access
> rights. I'll try to search the logs where the error came from.
OK, thanks.
It seems there's nothing to do on oVirt's side in that case, but let's wait for the logs.
Comment 7 Markus Stockhausen 2015-09-09 06:56:04 EDT
If I got it right vdsm does not set mount options for protocol itself. At least the logs show the mount command:

...
Thread-148510::DEBUG::2015-09-02 11:18:22,176::mount::227::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.10.30.254:/var/nas4/OVirtIB /rhev/data-center/mnt/10.10.30.254:_var_nas4_OVirtIB (cwd None)
...

From reading the NFS documentation there exist a UDP fallback in case TCP does not connect.

Question: Will it make sense to enforce protocol TCP as the default?
Comment 8 Allon Mureinik 2015-09-09 09:27:08 EDT
(In reply to Markus Stockhausen from comment #7)
> If I got it right vdsm does not set mount options for protocol itself. At
> least the logs show the mount command:
> 
> ...
> Thread-148510::DEBUG::2015-09-02
> 11:18:22,176::mount::227::Storage.Misc.excCmd::(_runcmd) /usr/bin/sudo -n
> /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3
> 10.10.30.254:/var/nas4/OVirtIB
> /rhev/data-center/mnt/10.10.30.254:_var_nas4_OVirtIB (cwd None)
> ...
Indeed - not unless you explicitly set them as advanced mount options.

> 
> From reading the NFS documentation there exist a UDP fallback in case TCP
> does not connect.
> 
> Question: Will it make sense to enforce protocol TCP as the default?
Do you mean setting mountptoroto=tcp by default?
I don't think so. I'd like to keep VDSM as simple as possible, and not make assumptions on what we should or shouldn't do with the mount options.
Comment 9 Markus Stockhausen 2015-09-09 13:43:22 EDT
I thought it might be more deterministic if we set proto=tcp. mountproto seems to be not so important. I leave it up to you to decide if such a fix will be helpful.
Comment 10 Allon Mureinik 2015-09-13 04:17:42 EDT
Closing for now.
If we find a smoking gun to prove this is required, we can always reopen.

Note You need to log in before you can comment on or make changes to this bug.