Bug 1670707

Summary:	[RHV 4.3]Hosted engine deployment fails with error storage format not supported
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	SATHEESARAN <sasundar>
Component:	rhhi	Assignee:	Gobinda Das <godas>
Status:	CLOSED ERRATA	QA Contact:	SATHEESARAN <sasundar>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	rhgs-3.4	CC:	rhs-bugs, sabose, sankarshan
Target Milestone:	---
Target Release:	RHHI-V 1.6
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	ovirt-engine-4.3.0.4 , appliance ovirt-engine-appliance-4.3-20190131.1.el7.x86_64	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1669606	Environment:
Last Closed:	2019-05-09 06:09:06 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1669606
Bug Blocks:	1672932

Description SATHEESARAN 2019-01-30 08:33:45 UTC

Description of problem:
------------------------
While performing HE deployment ( for RHHI-V ) using cockpit, it surpassed the creating the hostedenginelocal VM stage, and while creating the glusterfs based target storage domain, ends up in failure with error - 'Storage format not supported'

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
RHVH 4.3 ( RHEL 7.6 )
ovirt-hosted-engine-setup-2.3.2-1.el7ev.noarch

How reproducible:
-----------------
Always

Steps to Reproduce:
-------------------
1. Start with 'hyperconverged' deployment from cockpit
2. After gluster volume configuration, the deployment proceeds to configuring HE
3. Proceed with HE deployment

Actual results:
---------------
Configuration the HostedEngineLocal was successful, but while proceeding to next stage, deployment fails while create glusterfs based storage domain with error 'storage format not supported'

Expected results:
------------------
HE deployment should be successful

Comment 1 SATHEESARAN 2019-01-30 08:34:20 UTC

--- Additional comment from SATHEESARAN on 2019-01-25 18:56:27 UTC ---

Errors from the log file - ovirt-hosted-engine-setup-ansible-create_storage_domain-2019025212943-xqti5r.log

<snip>
2019-01-25 21:30:16,161+0530 DEBUG ansible on_any args TASK: oVirt.hosted-engine-setup : Add glusterfs storage domain kwargs is_conditional:False 
2019-01-25 21:30:19,134+0530 DEBUG var changed: host "localhost" var "otopi_storage_domain_details_gluster" type "<type 'dict'>" value: "{
    "changed": false, 
    "exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_ovirt_storage_domain_payload_RxcsV3/__main__.py\", line 682, in main\n    ret = storage_domains_module.create()\n  File \"/tmp/ansible_ovirt_storage_domain_payload_RxcsV3/ansible_ovirt_storage_domain_payload.zip/ansible/module_utils/ovirt.py\", line 587, in create\n    **kwargs\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\", line 25077, in add\n    return self._internal_add(storage_domain, headers, query, wait)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n    return future.wait() if wait else future\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 55, in wait\n    return self._code(response)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 229, in callback\n    self._check_fault(response)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n    self._raise_error(response, body)\n  File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n    raise error\nError: Fault reason is \"Operation Failed\". Fault detail is \"[Storage format is unsupported]\". HTTP response code is 400.\n", 
    "failed": true, 
    "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Storage format is unsupported]\". HTTP response code is 400."
}"
</snip>

--- Additional comment from Simone Tiraboschi on 2019-01-25 19:52:12 UTC ---

We are using ovirt_storage_domain ansible module with glusterfs parameter.

https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/blob/master/tasks/create_storage_domain.yml#L63

Ondra, any hint?

--- Additional comment from SATHEESARAN on 2019-01-25 20:22:15 UTC ---



--- Additional comment from Simone Tiraboschi on 2019-01-26 09:29:50 UTC ---

Do you also have engine.log?

--- Additional comment from SATHEESARAN on 2019-01-30 03:30:53 UTC ---

(In reply to Simone Tiraboschi from comment #4)
> Do you also have engine.log?

I have reprovisioned systems for RHV 4.2.8-1 testing. I have hit the problem with another set of systems.
Same issue persists. I have captured the engine.log as well.

Below is the snip from engine.log
<snip>
2019-01-30 07:23:42,382+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [646ed358] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM rhsqa-grafton8-nic2.lab.eng.blr.redhat.com command CreateStorageDomainVDS failed: Domain version `5` is unsupported by this version of VDSM: ''
2019-01-30 07:23:42,382+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [646ed358] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand' return value 'StatusOnlyReturn [status=Status [code=394, message=Domain version `5` is unsupported by this version of VDSM: '']]'
2019-01-30 07:23:42,382+05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [646ed358] HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com
2019-01-30 07:23:42,382+05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (default task-1) [646ed358] Command 'CreateStorageDomainVDSCommand(HostName = rhsqa-grafton8-nic2.lab.eng.blr.redhat.com, CreateStorageDomainVDSCommandParameters:{hostId='a8e8fdba-6353-48e6-8d17-68b01432d66e', storageDomain='StorageDomainStatic:{name='hosted_storage', id='97b88734-885c-4f62-be61-5ff87fee10de'}', args='rhsqa-grafton8.lab.eng.blr.redhat.com:/engine'})' execution failed: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Domain version `5` is unsupported by this version of VDSM: ''
</snip>

--- Additional comment from SATHEESARAN on 2019-01-30 03:33:23 UTC ---



--- Additional comment from SATHEESARAN on 2019-01-30 03:40:19 UTC ---

Resetting needinfo on Ondra

--- Additional comment from Simone Tiraboschi on 2019-01-30 08:15:04 UTC ---

(In reply to SATHEESARAN from comment #5)
> (In reply to Simone Tiraboschi from comment #4)
> > Do you also have engine.log?
> 
> I have reprovisioned systems for RHV 4.2.8-1 testing. I have hit the problem
> with another set of systems.
> Same issue persists. I have captured the engine.log as well.
> 
> Below is the snip from engine.log
> <snip>
> 2019-01-30 07:23:42,382+05 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-1) [646ed358] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802),
> VDSM rhsqa-grafton8-nic2.lab.eng.blr.redhat.com command
> CreateStorageDomainVDS failed: Domain version `5` is unsupported by this
> version of VDSM: ''
> 2019-01-30 07:23:42,382+05 INFO 
> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand]
> (default task-1) [646ed358] Command
> 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand'
> return value 'StatusOnlyReturn [status=Status [code=394, message=Domain
> version `5` is unsupported by this version of VDSM: '']]'

It seams that you mixed up a newer version of the engine with an older vdsm or something like that.
Nir?

--- Additional comment from Simone Tiraboschi on 2019-01-30 08:17:16 UTC ---

I see on that env:

vdsm.x86_64                                   4.30.8-2.el7ev
rhvm-appliance.x86_64                         2:4.3-20190124.1.el7

--- Additional comment from SATHEESARAN on 2019-01-30 08:20:46 UTC ---

(In reply to Simone Tiraboschi from comment #8)
> 
> It seams that you mixed up a newer version of the engine with an older vdsm
> or something like that.
> Nir?

Hi Simone, 

I have used the latest RHVH 4.3 & latest RHVH 4.3 appliance. 
There are no moving parts here.

--- Additional comment from SATHEESARAN on 2019-01-30 08:24:09 UTC ---

RHVH is installed from http://download-node-02.eng.bos.redhat.com/rhel-7/devel/candidate-trees/RHVH-4.3-20190127.1/compose/RHVH/x86_64/iso/RHVH-4.3-20190127.1-RHVH-x86_64-dvd1.iso

[root@rhsqa-grafton8 ~]# imgbase w
You are on rhvh-4.3.0.2-0.20190129.0+1

Appliance is from
http://download.eng.blr.redhat.com/brewroot/packages/rhvm-appliance/4.3/20190124.1.el7/x86_64/rhvm-appliance-4.3-20190124.1.el7.x86_64.rpm

--- Additional comment from Simone Tiraboschi on 2019-01-30 08:31:39 UTC ---

I think that we need a respin with https://gerrit.ovirt.org/#/c/97262/

Comment 4 SATHEESARAN 2019-02-26 12:07:46 UTC

The dependent bug is already VERIFIED and so moving this bug to ON_QA

Comment 5 SATHEESARAN 2019-03-08 05:30:49 UTC

Tested with RHVH 4.3 and ovirt-hosted-engine-setup-2.3.6-1.el7ev.noarch

HE deployment is successful over glusterfs storage domain

Comment 7 errata-xmlrpc 2019-05-09 06:09:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1121