Bug 1317699 - Hosted engine on Gluster prevents additional non-ha hosts being added
Summary: Hosted engine on Gluster prevents additional non-ha hosts being added
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 3.6.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-3.6.7
: 3.6.7.1
Assignee: Roy Golan
QA Contact: Artyom
URL:
Whiteboard:
: 1324075 1351203 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-14 22:23 UTC by Darryl Bond
Modified: 2016-07-04 12:31 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The hosted engine gluster storage domain was missing the vfstype attribute in the engine Consequence: Additional non-HE hosts that got the connect action by the engine just failed to mount it because the vdstype, "glusterfs" should be part of mount options for gluster. Fix: When importing the hosted storage, make sure to store "glusterfs" attribute for gluster domains. Result: A non-HE host is able to connect and monitor the hosted engine domain
Clone Of:
Environment:
Last Closed: 2016-07-04 12:31:05 UTC
oVirt Team: SLA
Embargoed:
rule-engine: ovirt-3.6.z+
rule-engine: blocker+
mgoldboi: planning_ack+
rgolan: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1327516 0 urgent CLOSED Infinite loop trying to fix remote path on glusterFS storage domain 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1351203 0 unspecified CLOSED VDSM fails to mount hosted-engine gluster volume 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 2423321 0 None None None 2016-07-01 17:44:26 UTC
oVirt gerrit 56455 0 master MERGED engine: Set the vfsType glusterfs when importing HE domain 2020-10-23 10:28:50 UTC
oVirt gerrit 57846 0 ovirt-engine-3.6 MERGED engine: Set the vfsType glusterfs when importing HE domain 2020-10-23 10:29:04 UTC
oVirt gerrit 57929 0 ovirt-engine-3.6.7 MERGED engine: Set the vfsType glusterfs when importing HE domain 2020-10-23 10:29:04 UTC

Internal Links: 1327516 1351203

Description Darryl Bond 2016-03-14 22:23:34 UTC
Description of problem: A 3 node hosted engine ha cluster with the engine storage on glusterfs cannot have additional non-ha hosts added. They fail due to attempting to mount the glusterfs hosted engine storage as NFS but with the gluster options which subsequently fails.


Version-Release number of selected component (if applicable): 3.6.3.4


How reproducible: Very


Steps to Reproduce:
1. Add a new (non-ha) host to the Datacentre that hosts the gluster hosted storage
2. 
3.

Actual results:
Host fails due to being unable to mount the hosted engine storage

Expected results:
Active host


Additional info:
jsonrpc.Executor/4::INFO::2016-03-09 16:05:02,002::logUtils::48::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=7, spUUID=u'00000001-0001-0001-0001-000000000229', conList=[{u'id': u'19fb9b3b-79c1-48e8-9300-d0d52ddce7b1', u'connection': u'ovirt36-h1:/hosted-engine', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '********', u'port': u''}], options=None)
 jsonrpc.Executor/6::DEBUG::2016-03-09 15:10:01,022::fileUtils::143::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/glusterSD/ovirt36-h1:_hosted-engine mode: None
jsonrpc.Executor/6::DEBUG::2016-03-09 15:10:01,022::storageServer::357::Storage.StorageServer.MountConnection::(_get_backup_servers_option) Using bricks: ['ovirt36-h1', 'ovirt36-h2', 'ovirt36-h3']
jsonrpc.Executor/6::DEBUG::2016-03-09 15:10:01,022::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset --cpu-list 0-11 /usr/bin/sudo -n /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount -o backup-volfile-servers=ovirt36-h2:ovirt36-h3 ovirt36-h1:/hosted-engine /rhev/data-center/mnt/glusterSD/ovirt36-h1:_hosted-engine (cwd None)
jsonrpc.Executor/6::ERROR::2016-03-09 15:10:01,042::hsm::2473::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2470, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 236, in connect
    six.reraise(t, v, tb)
  File "/usr/share/vdsm/storage/storageServer.py", line 228, in connect
    self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
  File "/usr/share/vdsm/storage/mount.py", line 225, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 241, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (32, ';Running scope as unit run-18808.scope.\nmount.nfs: an incorrect mount option was specified\n')

The '-t glusterfs' option is not being passed and therefore an nfs mount fails due to incorrect options.

Comment 1 Darryl Bond 2016-03-14 22:30:53 UTC
A suggested workaround was successful 
Set the vfs_type field to glusterfs in the storage_server_connections table in the engine database. 
(My engine gluster storage was called 'hosted-engine')

update storage_server_connections set vfs_type = 'glusterfs' where connection = 'hosted-engine'

Hosts can be added from there on.

Comment 2 Tal Nisan 2016-03-15 09:47:27 UTC
Roy, any reason the mount options are not sent correctly?

Comment 3 Doron Fediuck 2016-03-27 08:04:54 UTC
Darryl,
can you please explain how did you install HE with GlusterFS?
Current installer will not create the bricks and volume for you.

Comment 4 Darryl Bond 2016-03-27 10:46:28 UTC
I created the gluster volume manually.
3 hosts each with 2 disks, root filesystem mirrored on each with the remainder of the disk xfs with a 3way replica glusterfs across the 6 disks as bricks.

I used the recommended gluster settings for oVirt and set up the hosted engine as per the doco.

The 3 hosts are the HE hosts for the engine. The glusterfs has the storage for the engine, reports engine VM and a Cinder VM for the ceph main storage.

I know hyper-converged is not yet 'production ready' but this configuration seems to work well.

Comment 5 Roy Golan 2016-03-30 10:39:24 UTC
HE hosts are connected by the hosted engine agents so they know its -t gluster. 
non HE hosts are coneccted using the engine - but the domain in engine is missing that vfs type. 

The root cause is probably the auto-import which is not adding the vfs type if its gluster or the maybe the AddExistingStorageDomainCommand which is ignoreing the type.

Comment 6 Richard Neuboeck 2016-04-08 06:43:21 UTC
*** Bug 1324075 has been marked as a duplicate of this bug. ***

Comment 7 Yaniv Kaul 2016-05-04 18:26:19 UTC
A draft on master means it'll miss 3.6.6. Moving to 3.6.7. Let me know if it's not OK.

Comment 8 Artyom 2016-06-07 14:23:22 UTC
Verified on rhevm-3.6.7.2-0.1.el6.noarch

Comment 9 Simone Tiraboschi 2016-06-30 12:21:02 UTC
*** Bug 1351203 has been marked as a duplicate of this bug. ***

Comment 11 Marina Kalinin 2016-07-01 17:44:26 UTC
Simone, take a look:
https://access.redhat.com/solutions/2423321


Note You need to log in before you can comment on or make changes to this bug.