Bug 1404606

Summary: The initial checks on the gluster volume ignores additional mount options like backup-volfile-servers
Product: [oVirt] ovirt-hosted-engine-setup Reporter: SATHEESARAN <sasundar>
Component: Plugins.GlusterAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED CURRENTRELEASE QA Contact: SATHEESARAN <sasundar>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.1.0CC: bugs, khung, sabose, sasundar, stirabos, ylavi
Target Milestone: ovirt-4.2.2Flags: sabose: ovirt-4.2?
sasundar: planning_ack?
rule-engine: devel_ack+
sasundar: testing_ack+
Target Release: 2.2.10   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-10 06:23:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1455169    
Bug Blocks:    
Attachments:
Description Flags
hosted-engine-setup none

Description SATHEESARAN 2016-12-14 08:53:15 UTC 
Description of problem:
-----------------------
Self Hosted-Engine deployment with glusterfs backend fails to mount gluster volume with backup-volfile-servers option though provided with the deployment, and fails when the primary volfile server is down

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
ovirt-4.1-snapshot
ovirt-hosted-engine-setup-2.1.0-0.0.master.20161130101611.gitb3ad261.el7.centos.noarch
RHGS 3.2.0 ( nightly - glusterfs-3.8.4-8.el7rhgs )

How reproducible:
-----------------
Always

Steps to Reproduce:
--------------------
1. Create a gluster replica 3 volume, optimize the volume for virt store and start the volume
2. Run hosted engine deployment - 'hosted-engine --deploy' 
3. Choose 'glusterfs' storage, and provide storage path with IP of node1 (i.e) <NODE1_IP>:/<gluster_volume_name>
3. Provide additional mount options with backup-volfile-servers
4. Bring down the glusterd on the node NODE1
5. Continue with setup

Actual results:
---------------
Hosted-engine setup fails with error - "[ ERROR ] Cannot access storage connection 10.70.36.73:/engine: Command '/sbin/gluster' failed to execute"

Expected results:
-----------------
As backup-volfile-server option is provided as additional mount option, the mount should happen with the additional volfile servers
Comment 1 SATHEESARAN 2016-12-14 08:54:23 UTC 
Here is the snip from the hosted-engine deployment

<snip>

          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: glusterfs
[ INFO  ] Please note that Replica 3 support is required for the shared storage.
          Please specify the full shared storage connection path to use (example: host:/path): 10.70.36.73:/engine           
          If needed, specify additional mount options for the connection to the hosted-engine storage domain []: backup-volfile-servers=10.70.36.74:10.70.36.75
[ ERROR ] Cannot access storage connection 10.70.36.73:/engine: Command '/sbin/gluster' failed to execute

</snip>
Comment 2 SATHEESARAN 2016-12-14 08:55:36 UTC 
I have marked 'oVirt Team as Infra'. I am not sure on that. Please make suitable changes on this field, if I'm wrong
Comment 3 SATHEESARAN 2016-12-14 08:57:35 UTC 
Created attachment 1231520 [details]
hosted-engine-setup
Comment 4 Simone Tiraboschi 2016-12-14 09:47:23 UTC 
What fails here it's just the initial validation of gluster volume: we use '/sbin/gluster --mode=script volume info' to gather volume info in order to enforce it's in replica 3.

2016-12-14 12:49:40 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 If needed, specify additional mount options for the connection to the hosted-engine storage domain []: 
2016-12-14 12:50:41 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:RECEIVE    backup-volfile-servers=10.70.36.74:10.70.36.75
2016-12-14 12:50:41 DEBUG otopi.plugins.gr_he_setup.storage.nfs plugin.executeRaw:813 execute: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'engine', '--remote-host=10.70.36.73'), executable='None', cwd='None', env=None
2016-12-14 12:50:41 DEBUG otopi.plugins.gr_he_setup.storage.nfs plugin.executeRaw:863 execute-result: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'engine', '--remote-host=10.70.36.73'), rc=1
2016-12-14 12:50:41 DEBUG otopi.plugins.gr_he_setup.storage.nfs plugin.execute:921 execute-output: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'engine', '--remote-host=10.70.36.73') stdout:
Connection failed. Please check if gluster daemon is operational.

2016-12-14 12:50:41 DEBUG otopi.plugins.gr_he_setup.storage.nfs plugin.execute:926 execute-output: ('/sbin/gluster', '--mode=script', '--xml', 'volume', 'info', 'engine', '--remote-host=10.70.36.73') stderr:


2016-12-14 12:50:41 DEBUG otopi.plugins.gr_he_setup.storage.nfs nfs._customization:420 exception
Traceback (most recent call last):
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/storage/nfs.py", line 414, in _customization
    ohostedcons.StorageEnv.MNT_OPTIONS
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/storage/nfs.py", line 311, in _validateDomain
    self._check_volume_properties(connection)
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/storage/nfs.py", line 182, in _check_volume_properties
    raiseOnError=True
  File "/usr/lib/python2.7/site-packages/otopi/plugin.py", line 931, in execute
    command=args[0],
RuntimeError: Command '/sbin/gluster' failed to execute
Comment 5 Simone Tiraboschi 2016-12-14 09:52:20 UTC 
Satheesaran, do you know if
 /sbin/gluster --mode=script --xml volume info engine --remote-host=10.70.36.73
could support somehow the additional mount options?
Comment 6 SATHEESARAN 2016-12-15 04:10:24 UTC 
(In reply to Simone Tiraboschi from comment #5)
> Satheesaran, do you know if
>  /sbin/gluster --mode=script --xml volume info engine
> --remote-host=10.70.36.73
> could support somehow the additional mount options?

Hi Simone,

The command actually never supports additional mount options.
But it does accepts additional remote-hosts.

"/sbin/gluster --mode=script --xml volume info engine --remote-host=10.70.36.73 --remote-host=10.70.36.74" 

This works well as the command is executed on the other remote host until we have multiple remote hosts mentioned
Comment 7 SATHEESARAN 2016-12-15 06:32:35 UTC 
(In reply to SATHEESARAN from comment #6)
> (In reply to Simone Tiraboschi from comment #5)
> > Satheesaran, do you know if
> >  /sbin/gluster --mode=script --xml volume info engine
> > --remote-host=10.70.36.73
> > could support somehow the additional mount options?
> 
> Hi Simone,
> 
> The command actually never supports additional mount options.
> But it does accepts additional remote-hosts.
> 
> "/sbin/gluster --mode=script --xml volume info engine
> --remote-host=10.70.36.73 --remote-host=10.70.36.74" 
> 
> This works well as the command is executed on the other remote host until we
> have multiple remote hosts mentioned

Sorry that doesn't works. 
I stand corrected.
It takes only the last '--remote-host' value.
There is no multiple values of '--remote-host' accepted for the command.
Comment 8 Sahina Bose 2016-12-22 08:11:04 UTC 
*** Bug 1398769 has been marked as a duplicate of this bug. ***
Comment 9 Simone Tiraboschi 2017-12-19 17:11:10 UTC 
Fixed with node-zero
Comment 10 SATHEESARAN 2018-05-07 01:13:54 UTC 
Tested with RHV 4.2.3 and ovirt-hosted-engine-setup-2.2.18.

With node-zero deployment, this issue is no longer seen
Comment 11 Sandro Bonazzola 2018-05-10 06:23:46 UTC 
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.