Description of problem:

glusterfs is strict on versions for compatibility, if there's different glusterfs version on oVirt hosts and remote glusterfs replica 3 storage, there's an error which doesn't reveal much info and the storage fails to be used...

See https://bugzilla.redhat.com/show_bug.cgi?id=1356033

~~~
In RHEVM environment we deploy Self-Hosted Engine on glusterfs storage. SHE setup runs:

~~~
/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'Jiri', '--remote-host=<HOST>'
~~~

and this returns nothing with RC 2.

Although running this via ssh and without '--remote-host' it returns volume info:

~~~
# ssh  <HOST> "gluster --mode=script --xml volume info Jiri" | head
root@<HOST>'s password: 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
  <opRet>0</opRet>
  <opErrno>0</opErrno>
  <opErrstr/>
  <volInfo>
    <volumes>
      <volume>
        <name>Jiri</name>
        <id>538318f2-8d99-4111-af2c-247e570b7f1c</id>
~~~

And comment https://bugzilla.redhat.com/show_bug.cgi?id=1356033#c2

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-1.3.7.2-1.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1. have replica 3 glusterfs storage with older 3.7.x (eg. 3.7.5)
2. have recent ovirt-hosted-engine-setup from 3.6 which includes recent
   3.7.x glusterfs rpms (eg. glusterfs-server-0:3.7.13-1.el7.x86_64)
3. hosted-engine --deploy and use 'glusterfs' as storage bckend

Actual results:
failure with non-clear error

Expected results:
either warn about the need of equal glusterfs versions or at least document it somewhere

Additional info:


~~~
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: glusterfs
[ INFO  ] Please note that Replica 3 support is required for the shared storage.
          Please specify the full shared storage connection path to use (example: host:/path): xx.xx.xx.27:/Jiri
[ ERROR ] Cannot access storage connection xx.xx.xx.27:/Jiri: Command '/sbin/gluster' failed to execute
          Please specify the full shared storage connection path to use (example: host:/path):
~~~


It is something with querying glusterfs remotely:

~~~
[root@purple-vds2 ~]# gluster --mode=script --xml --remote-host=xx.xx.xx.27 volume info Jiri

[root@purple-vds2 ~]# ssh  xx.xx.xx.27 "gluster volume info Jiri"
root.xx.27's password:
 
Volume Name: Jiri
Type: Replicate
Volume ID: 538318f2-8d99-4111-af2c-247e570b7f1c
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: xx.xx.xx.25:/gluster_volumes/Jiri
Brick2: xx.xx.xx.26:/gluster_volumes/Jiri
Brick3: xx.xx.xx.27:/gluster_volumes/Jiri
Options Reconfigured:
storage.owner-uid: 36
storage.owner-gid: 36
performance.readdir-ahead: on

[root@purple-vds2 ~]# ssh  xx.xx.xx.27 "gluster --mode=script --xml volume info Jiri"
...snip...
~~~

^^ this also works OK

And the setup log:

~~~
2016-07-13 12:13:06 DEBUG otopi.plugins.otopi.dialog.human human.queryString:156 query OVEHOSTED_STORAGE_DOMAIN_CONNECTION
2016-07-13 12:13:06 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:SEND                 Please specify the full shared storage connection path to use (example: host:/path):
2016-07-13 12:13:14 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:RECEIVE    xx.xx.xx.27:/Jiri
2016-07-13 12:13:14 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.nfs plugin.executeRaw:828 execute: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'Jiri', '--remote-host=xx.xx.xx.27'), exe
cutable='None', cwd='None', env=None
2016-07-13 12:13:14 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.nfs plugin.executeRaw:878 execute-result: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'Jiri', '--remote-host=xx.xx.xx.27
'), rc=2
2016-07-13 12:13:14 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.nfs plugin.execute:936 execute-output: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'Jiri', '--remote-host=xx.xx.xx.27')
stdout:


2016-07-13 12:13:14 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.nfs plugin.execute:941 execute-output: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'Jiri', '--remote-host=xx.xx.xx.27')
stderr:


2016-07-13 12:13:14 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.nfs nfs._customization:395 exception
Traceback (most recent call last):
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/nfs.py", line 390, in _customization
    check_space=False,
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/nfs.py", line 302, in _validateDomain
    self._check_volume_properties(connection)
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/nfs.py", line 179, in _check_volume_properties
    raiseOnError=True
  File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 946, in execute
    command=args[0],
RuntimeError: Command '/sbin/gluster' failed to execute
2016-07-13 12:13:14 ERROR otopi.plugins.ovirt_hosted_engine_setup.storage.nfs nfs._customization:405 Cannot access storage connection 1
~~~

And... glusterfs cli.log:

~~~
[2016-07-13 09:13:14.395510] I [cli.c:722:main] 0-cli: Started running /sbin/gluster with version 3.7.13
[2016-07-13 09:13:14.395572] I [cli.c:608:cli_rpc_init] 0-cli: Connecting to remote glusterd at xx.xx.xx.27
[2016-07-13 09:13:14.408521] I [cli-cmd-volume.c:1831:cli_check_gsync_present] 0-: geo-replication not installed
[2016-07-13 09:13:14.408939] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-07-13 09:13:14.409022] I [socket.c:2472:socket_event_handler] 0-transport: disconnecting now
[2016-07-13 09:13:14.411062] I [cli-rpc-ops.c:805:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2016-07-13 09:13:14.411274] E [cli-rpc-ops.c:883:gf_cli_get_volume_cbk] 0-cli: Error outputting to xml
[2016-07-13 09:13:14.411389] I [input.c:36:cli_batch] 0-: Exiting with: -2
~~~