Bug 1359265

My quick immediate hack was to do the mount manually:

# mkdir /rhev/data-center/mnt/nfs.localdomain:_ovirt_hosted-engine

# /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=4 \
  nfs.localdomain:/ovirt/hosted-engine \
  /rhev/data-center/mnt/nfs.localdomain:_ovirt_hosted-engine


A more permant fix, from Simone's email on the users list:

If you need a quick fix you can:
- fix the configuration of your storage server to allow it to be accessed
also over nfsv3

- edit the configuration of the storage connection in the engine DB on the
engine VM to add the missing parameter. Something like:
 # sudo -u postgresl psql
 \c engine;
 select * from storage_server_connections;
 UPDATE storage_server_connections SET nfs_version = '4' WHERE connection =
'nfs.localdomain:/ovirt/hosted-engine';
 commit;
 select * from storage_server_connections;

*** Bug 1435570 has been marked as a duplicate of this bug. ***

This should support Gluster and possibly other storage types as well.

*** Bug 1471026 has been marked as a duplicate of this bug. ***

Part of node zero I assume.

Addressed in node-zero for NFS and gluster

INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[No external trackers attached]

For more info please contact: infra

Simone, are there patches to attach to this bug to make the bot happy?

Can you please provide relevant documentation regarding usage of mounting options during deployment of Node Zero over NFSv3 and NFSv4 storage domains?

(In reply to Martin Sivák from comment #10)
> Simone, are there patches to attach to this bug to make the bot happy?

It's implicitly addressed by the new node zero flow, no specific patches on the old one (vdsm is still not reporting in use mount options to the engine at autoimport time).

(In reply to Nikolai Sednev from comment #11)
> Can you please provide relevant documentation regarding usage of mounting
> options during deployment of Node Zero over NFSv3 and NFSv4 storage domains?

Please try with "rsize=32768,wsize=32768" on this question:
          If needed, specify additional mount options for the connection to the hosted-engine storagedomain []: 

The additional mount options should be visible in the engine and in hosted-engine.conf on all the involved hosts.

I've tried with additional NFSv3 mount options from https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-nfs-client-config-options.html, such as rsize=32768,wsize=32768:

[ ERROR ]  [WARNING]: Failure using method (v2_runner_on_failed) in callback plugin
         
[ ERROR ] (<ansible.plugins.callback.1_otopi_json.CallbackModule object at 0x1f2d950>):
         
[ ERROR ] 'ascii' codec can't encode character u'\u2018' in position 520: ordinal not in
         
[ ERROR ] range(128)
         
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook

Components used during deployment:
ovirt-engine-appliance-4.2-20180108.1.el7.centos.noarch.rpm

http://pastebin.test.redhat.com/545655

Created attachment 1379602 [details]
sosreport from alma03

(In reply to Simone Tiraboschi from comment #13)
> (In reply to Nikolai Sednev from comment #11)
> > Can you please provide relevant documentation regarding usage of mounting
> > options during deployment of Node Zero over NFSv3 and NFSv4 storage domains?
> 
> Please try with "rsize=32768,wsize=32768" on this question:
>           If needed, specify additional mount options for the connection to
> the hosted-engine storagedomain []: 
> 
> The additional mount options should be visible in the engine and in
> hosted-engine.conf on all the involved hosts.

I've tried with additional NFSv3 mount options from https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-nfs-client-config-options.html, such as rsize=32768,wsize=32768:

[ INFO  ] changed: [localhost]
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: 
          Please specify the nfs version you would like to use (auto, v3, v4, v4_1)[auto]: v3
          Please specify the full shared storage connection path to use (example: host:/path): yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_1
          If needed, specify additional mount options for the connection to the hosted-engine storagedomain []: "rsize=32768,wsize=32768"
[ INFO  ] Creating Storage Domain

[ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Problem while trying to mount target]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Problem while trying to mount target]\". HTTP response code is 400."}
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: 

Components used during deployment:
ovirt-engine-appliance-4.2-20180108.1.el7.centos.noarch.rpm
ovirt-hosted-engine-setup-2.2.4-0.0.master.20180108132354.git2005b97.el7.centos.noarch
ovirt-hosted-engine-ha-2.2.3-0.0.master.20171218181916.20171218181911.git4c22b93.el7.centos.noarch
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Linux 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Dec 28 14:23:39 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

(In reply to Nikolai Sednev from comment #14)     
> [ ERROR ] 'ascii' codec can't encode character u'\u2018' in position 520:
> ordinal not in

The real issue is here:
2018-01-10 17:58:30,479+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:73 {u'_ansible_parsed': True, u'stderr_lines': [u'dd: failed to open \u2018/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_nsednev__he__2/000ccade-f52e-4bf6-9a11-d09f4c74a3a9/images/6ab7311c-3107-4afb-8d77-c290efdaabf6/71c22342-0ec9-43df-b756-1bc6ab8ef741\u2019: Permission denied'], u'cmd': [u'dd', u'bs=20480', u'count=1', u'oflag=direct', u'if=/var/tmp/localvmYRB8qf/71c22342-0ec9-43df-b756-1bc6ab8ef741', u'of=/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_nsednev__he__2/000ccade-f52e-4bf6-9a11-d09f4c74a3a9/images/6ab7311c-3107-4afb-8d77-c290efdaabf6/71c22342-0ec9-43df-b756-1bc6ab8ef741'], u'end': u'2018-01-10 17:58:30.337743', u'_ansible_no_log': False, u'stdout': u'', u'changed': True, u'start': u'2018-01-10 17:58:30.332305', u'delta': u'0:00:00.005438', u'stderr': u'dd: failed to open \u2018/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_nsednev__he__2/000ccade-f52e-4bf6-9a11-d09f4c74a3a9/images/6ab7311c-3107-4afb-8d77-c290efdaabf6/71c22342-0ec9-43df-b756-1bc6ab8ef741\u2019: Permission denied', u'rc': 1, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'_raw_params': u'dd bs=20480 count=1 oflag=direct if="/var/tmp/localvmYRB8qf/71c22342-0ec9-43df-b756-1bc6ab8ef741" of="/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_nsednev__he__2/000ccade-f52e-4bf6-9a11-d09f4c74a3a9/images/6ab7311c-3107-4afb-8d77-c290efdaabf6/71c22342-0ec9-43df-b756-1bc6ab8ef741"', u'removes': None, u'creates': None, u'chdir': None, u'stdin': None}}, u'stdout_lines': [], u'msg': u'non-zero return code'}


but we had an issue parsing the error message on:
[u'dd: failed to open \u2018/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_nsednev__he__2/000ccade-f52e-4bf6-9a11-d09f4c74a3a9/images/6ab7311c-3107-4afb-8d77-c290efdaabf6/71c22342-0ec9-43df-b756-1bc6ab8ef741\u2019: Permission denied']

And so:
2018-01-10 17:58:30,783+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:150 (<ansible.plugins.callback.1_otopi_json.CallbackModule object at 0x1f2d950>):
2018-01-10 17:58:30,783+0200 DEBUG otopi.plugins.otopi.dialog.human 
2018-01-10 17:58:30,783+0200 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:150 'ascii' codec can't encode character u'\u2018' in position 520: ordinal not in

(\u2018 is LEFT SINGLE QUOTATION MARK and \u2019 is RIGHT SINGLE QUOTATION MARK).

Could you please open a new bug on the node zero flow about correctly handling unicode chars in error messages?

(In reply to Nikolai Sednev from comment #16)
> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is
> "[Problem while trying to mount target]". HTTP response code is 400.
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault
> reason is \"Operation Failed\". Fault detail is \"[Problem while trying to
> mount target]\". HTTP response code is 400."}

Sorry, plaese try instead with v4 and 'rsize=8192,wsize=8192'

[ INFO  ] changed: [localhost]
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: 
          Please specify the nfs version you would like to use (auto, v3, v4, v4_1)[auto]: v4
          Please specify the full shared storage connection path to use (example: host:/path): 192.168.1.115:/storage/nfs/he1
          If needed, specify additional mount options for the connection to the hosted-engine storagedomain []: rsize=8192,wsize=8192

Why it did not worked for NFSv3?
According to what is written in documentation, NFSv3 should support changing of these valuse.
"rsize=num and wsize=num — These settings speed up NFS communication for reads (rsize) and writes (wsize) by setting a larger data block size, in bytes, to be transferred at one time. Be careful when changing these values; some older Linux kernels and network cards do not work well with larger block sizes. For NFSv2 or NFSv3, the default values for both parameters is set to 8192. For NFSv4, the default values for both parameters is set to 32768."

Created attachment 1379958 [details]
vdsm log from host

OK, the issue is just here:

2018-01-11 12:09:36,852+0200 INFO  (jsonrpc/2) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'mnt_options': u"'rsize=8192,wsize=8192'", u'id': u'00000000-0000-0000-0000-000000000000', u'connection': u'yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_1', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'4', u'password': '********', u'port': u''}], options=None) from=::ffff:192.168.122.167,47676, flow_id=f91f0ff7-069d-4d21-9cf3-bc104a63ece7, task_id=d9fb3a50-e7aa-44f5-98b6-b9bcad56b51c (api:46)
2018-01-11 12:09:36,856+0200 INFO  (jsonrpc/2) [storage.Mount] mounting yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_1 at /rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_nsednev__he__1 (mount:204)
2018-01-11 12:09:36,995+0200 ERROR (jsonrpc/2) [storage.HSM] Could not connect to storageServer (hsm:2407)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2404, in connectStorageServer
    conObj.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 406, in connect
    return self._mountCon.connect()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 179, in connect
    six.reraise(t, v, tb)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 171, in connect
    self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount
    cgroup=cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda>
    **kwargs)
  File "<string>", line 2, in mount
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
MountError: (32, ';mount.nfs: an incorrect mount option was specified\n')
2018-01-11 12:09:36,997+0200 INFO  (jsonrpc/2) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 477, 'id': u'00000000-0000-0000-0000-000000000000'}]} from=::ffff:192.168.122.167,47676, flow_id=f91f0ff7-069d-4d21-9cf3-bc104a63ece7, task_id=d9fb3a50-e7aa-44f5-98b6-b9bcad56b51c (api:52)

and indeed you entered: "'rsize=8192,wsize=8192'" while you should simply enter "rsize=8192,wsize=8192" (skipping also double quote characters, they are there just to keep it a bit more readable!)

>           If needed, specify additional mount options for the connection to
> the hosted-engine storagedomain []: 'rsize=8192,wsize=8192'


The issue was exactly the same also on the nfs v3 test:

>           Please specify the nfs version you would like to use (auto, v3,
> v4, v4_1)[auto]: v3
>           Please specify the full shared storage connection path to use
> (example: host:/path):
> yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_1
>           If needed, specify additional mount options for the connection to
> the hosted-engine storagedomain []: "rsize=32768,wsize=32768"

I did around 16:23 as follows:
  Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: 
          Please specify the nfs version you would like to use (auto, v3, v4, v4_1)[auto]: v4
          Please specify the full shared storage connection path to use (example: host:/path): yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_1
          If needed, specify additional mount options for the connection to the hosted-engine storagedomain []: rsize=32768,wsize=32768


Received at 16:31:
[ ERROR ] ConnectionError: Error while sending HTTP request: (7, 'Failed connect to nsednev-he-1.qa.lab.tlv.redhat.com:443; No route to host')
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": false, "msg": "Error while sending HTTP request: (7, 'Failed connect to nsednev-he-1.qa.lab.tlv.redhat.com:443; No route to host')"}
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: 

See the sosreport from host alma03 as appears within the attachment.

Created attachment 1380061 [details]
sosreport from alma03

Ok, the deployment worked for me with simple insert of parameters without quoting:
[ INFO  ] changed: [localhost]
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: 
          Please specify the nfs version you would like to use (auto, v3, v4, v4_1)[auto]: v4
          Please specify the full shared storage connection path to use (example: host:/path): yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_1
          If needed, specify additional mount options for the connection to the hosted-engine storagedomain []: rsize=32768,wsize=32768

[ INFO  ] Hosted Engine successfully deployed



alma04 ~]# cat  /etc/ovirt-hosted-engine/hosted-engine.conf
fqdn=nsednev-he-1.qa.lab.tlv.redhat.com
vm_disk_id=a3251750-f78e-4d19-986d-5d2d224bce3f
vm_disk_vol_id=96951442-d779-4761-a05c-5484eefa5457
vmid=ecf2ed4a-1b2b-46e8-bd6a-45aa3973d4a5
storage=yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_1
nfs_version=v4
mnt_options=rsize=32768,wsize=32768
conf=/var/run/ovirt-hosted-engine-ha/vm.conf
host_id=1
console=vnc
domainType=nfs
spUUID=00000000-0000-0000-0000-000000000000
sdUUID=b550d9b5-05e6-47ef-b72d-2aa7d9305218
connectionUUID=e29cf818-5ee5-46e1-85c1-8aeefa33e95d
ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem
ca_subject="C=EN, L=Test, O=Test, CN=Test"
vdsm_use_ssl=true
gateway=10.35.72.254
bridge=ovirtmgmt
metadata_volume_UUID=a8672425-5b56-450b-aa8f-85126b51017d
metadata_image_UUID=6614c002-abe1-4bd1-a8d3-6cbd98a5357a
lockspace_volume_UUID=600aa1fc-d0e4-436e-a348-8ffd2d69a054
lockspace_image_UUID=da5ecc31-ba4d-4958-b7d3-4ccda9cfa21c
conf_volume_UUID=08fef88e-d263-4ac6-bd44-b9baa3c1f5fb
conf_image_UUID=f7718321-1b3a-4963-bdad-bd198b645084

# The following are used only for iSCSI storage
iqn=
portal=
user=
password=
port=
You have new mail in /var/spool/mail/root

Also values shown in UI appeared correctly, please see the attached screenshot.

Moving to verified.

Worked for me on these components:
ovirt-hosted-engine-ha-2.2.3-0.0.master.20171218181916.20171218181911.git4c22b93.el7.centos.noarch
ovirt-hosted-engine-setup-2.2.4-0.0.master.20180109170856.gitb593776.el7.centos.noarch
ovirt-engine-appliance-4.2-20180110.1.el7.centos.noarch

Created attachment 1380101 [details]
screenshot from UI

This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.