Bug 1434423 - Mount options for the HE Gluster storage that will have the replica peers are not persisted in the database
Summary: Mount options for the HE Gluster storage that will have the replica peers are...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.1.1.2
Hardware: Unspecified
OS: Unspecified
unspecified
urgent vote
Target Milestone: ovirt-4.2.2
: 4.2.2
Assignee: Allon Mureinik
QA Contact: SATHEESARAN
URL:
Whiteboard:
Depends On: 1455169
Blocks: Gluster-HC-2
TreeView+ depends on / blocked
 
Reported: 2017-03-21 13:34 UTC by RamaKasturi
Modified: 2018-05-10 06:24 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-10 06:24:08 UTC
oVirt Team: Gluster
rule-engine: ovirt-4.2?
sasundar: planning_ack?
rule-engine: devel_ack+
sasundar: testing_ack+


Attachments (Terms of Use)
Attaching screenshot for hosted_storage (164.67 KB, image/png)
2017-03-21 13:35 UTC, RamaKasturi
no flags Details
hosted_storage domain screenshot (176.58 KB, image/png)
2017-04-10 07:33 UTC, RamaKasturi
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1402417 None CLOSED Mount options are lost when storage is imported to the engine, please block it from the setup. 2019-04-08 18:59:26 UTC
Red Hat Bugzilla 1426517 None CLOSED Additional mount option for glusterfs volume as backend for HE storage 2019-04-08 18:59:26 UTC

Internal Links: 1402417 1426517

Description RamaKasturi 2017-03-21 13:34:21 UTC
Description of problem:
I see that when hosted_storage is imported into UI mount options are not persisted in UI and database. Attaching the screenshot showing mount options not present in UI. 

Below is the query i ran to check in database and i do not see it their either.

root@hostedenginesm2 ~]# su - postgres 
Last login: Tue Mar 21 06:36:20 EDT 2017 on pts/0
-bash-4.2$ psql -d engine
psql (9.2.18)
Type "help" for help.

engine=# SELECT * FROM storage_server_connections;
                  id                  |     connection      | user_name | password | iqn | port | portal | storage_type |                 mount_options                  | vf
s_type  | nfs_version | nfs_timeo | nfs_retrans |          gluster_volume_id           
--------------------------------------+---------------------+-----------+----------+-----+------+--------+--------------+------------------------------------------------+---
--------+-------------+-----------+-------------+--------------------------------------
 228012f0-bc6e-47fe-922f-6fdbce532194 | 10.70.36.82:/data   |           |          |     |      | 1      |            7 | backup-volfile-servers=10.70.36.83:10.70.36.84 | gl
usterfs |             |           |             | c5f995d3-938d-4eed-9159-386b52ca20be
 d832dd3e-e67c-4d94-850c-ede4bd259b4c | 10.70.36.82:/engine |           |          |     |      | 1      |            7 |                                                | gl
usterfs |             |           |             | 
(2 rows)

Version-Release number of selected component (if applicable):
Red Hat Virtualization Manager Version: 4.1.1.2-0.1.el7

How reproducible:
Always

Steps to Reproduce:
1. Install latest RHV-H ISO
2. Deploy gluster using cockpit
3. Once deployment is done proceed to hosted engine deployment.
4. Create first master storage domain 

Actual results:
Once master storage domain is created hosted_storage is imported to the UI but mount options are not persisted in UI and Database.

Expected results:
when hosted_storage is imported to the UI mount options should be persisted in UI and DB.

Additional info:

Comment 1 RamaKasturi 2017-03-21 13:35:00 UTC
Created attachment 1265072 [details]
Attaching screenshot for hosted_storage

Comment 2 Simone Tiraboschi 2017-03-22 07:54:52 UTC
Yes, we know, that's why we hided the relevant query in ovirt-hosted-engine-setup. AFAIK for NFS storage domain, VDSM is neither reporting them back to the engine.

Please see:

https://bugzilla.redhat.com/show_bug.cgi?id=1402417 - Mount options are lost when storage is imported to the engine, please block it from the setup.

https://bugzilla.redhat.com/show_bug.cgi?id=1426517 - Additional mount option for glusterfs volume as backend for HE storage

Comment 3 RamaKasturi 2017-03-22 08:52:49 UTC
(In reply to Simone Tiraboschi from comment #2)
> Yes, we know, that's why we hided the relevant query in
> ovirt-hosted-engine-setup. AFAIK for NFS storage domain, VDSM is neither
> reporting them back to the engine.

Simone, but i have a question here. will this not effect restarting of engine if the primary server is down?
> 
> Please see:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1402417 - Mount options are lost
> when storage is imported to the engine, please block it from the setup.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1426517 - Additional mount
> option for glusterfs volume as backend for HE storage

In this bug i see that restarting HE setup causes the issue. But in my case there was no restart of HE setup.

Comment 4 Simone Tiraboschi 2017-03-22 10:47:40 UTC
(In reply to RamaKasturi from comment #3)
> (In reply to Simone Tiraboschi from comment #2)
> > Yes, we know, that's why we hided the relevant query in
> > ovirt-hosted-engine-setup. AFAIK for NFS storage domain, VDSM is neither
> > reporting them back to the engine.
> 
> Simone, but i have a question here. will this not effect restarting of
> engine if the primary server is down?

Yes, it will.
If the engine is down, ovirt-ha-agent will try to reconnect the storage according to how it's described under /etc/ovirt-hosted-engine/hosted-engine.conf and this includes the mnt_option field.
The point is that, for additional hosts, that file is wrote by host-deploy according to what the engine knows about the storage.

> > Please see:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1402417 - Mount options are lost
> > when storage is imported to the engine, please block it from the setup.
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1426517 - Additional mount
> > option for glusterfs volume as backend for HE storage
> 
> In this bug i see that restarting HE setup causes the issue. But in my case
> there was no restart of HE setup.

The root cause is not a restart of the hosted-engine-setup, the issue is VDSM not reporting back the mount options to the engine so the engine ignores them.

Comment 5 Yaniv Lavi 2017-03-29 12:56:32 UTC
Please test this flow:
  + Run gdeploy with two volumes (HE and data).
  + After first host setup of HE with the mount options for replica set.
  + Check that the first HE host conf contains the mount options.
  + Add the data volume as SD to engine.
  + Wait for the HE SD import to complete after the data SD is added.
  + Check that mount options for replica were added to engine (please check how much time this takes to happen).
  + Only then deploy the two additional hosts.
  + See if the conf of the addtional hosts contains the mount option

Please report on any issues.

Comment 6 RamaKasturi 2017-03-30 07:55:05 UTC
(In reply to Yaniv Dary from comment #5)
> Please test this flow:
>   + Run gdeploy with two volumes (HE and data).
>   + After first host setup of HE with the mount options for replica set.
>   + Check that the first HE host conf contains the mount options.
>   + Add the data volume as SD to engine.
>   + Wait for the HE SD import to complete after the data SD is added.
>   + Check that mount options for replica were added to engine (please check
> how much time this takes to happen).
>   + Only then deploy the two additional hosts.
>   + See if the conf of the addtional hosts contains the mount option
> 
> Please report on any issues.

Hi Yaniv,

   This is the same flow what we follow currently in HC. When we follow this flow we see that mount options are not persisted in DB and UI. I see the options in /etc/ovirt-hosted-engine/hosted-engine.conf file.

Thanks
kasturi

Comment 7 Yaniv Lavi 2017-03-30 09:12:37 UTC
(In reply to RamaKasturi from comment #6)
> 
> Hi Yaniv,
> 
>    This is the same flow what we follow currently in HC. When we follow this
> flow we see that mount options are not persisted in DB and UI. I see the
> options in /etc/ovirt-hosted-engine/hosted-engine.conf file.
> 
> Thanks
> kasturi

The engine queries the peers for the Gluster volume HCI or not.
You do not need get it from the HE setup. Please work in this flow and retest, it should provide the peers for the volume.

Comment 8 RamaKasturi 2017-04-10 07:33:13 UTC
Created attachment 1270404 [details]
hosted_storage domain screenshot

Comment 9 RamaKasturi 2017-04-10 07:36:06 UTC
just wanted to highlight , host deploy writes the mnt_options for the /etc/hosted-engine/hosted-engine.conf on all the nodes in the gluster cluster. problem here appears with hosted_storage domain imported into the UI does not have the mnt_options present

Comment 10 Sandro Bonazzola 2017-05-15 08:28:01 UTC
Re-targeting to 4.2.0 since it depends on bug #1373930 which has been targeted to 4.2.0

Comment 11 Simone Tiraboschi 2017-05-15 08:34:28 UTC
The mnt_options value is wrote by host-deploy on /etc/hosted-engine/hosted-engine.conf on all the hosts since the initial value is saved in the answer file of the first host on the hosted-engine configuration volume and the engine extracts it.

That value is not saved in the engine DB (and so it's also not visibile in the UI) since VDSM doesn't report it back to the engine as https://bugzilla.redhat.com/show_bug.cgi?id=1373930

Always relying on the initial value from the answerfile is wrong since in that way we are always going to consume just the initial value (the set of three initial hosts) regardless of host that have been decommissioned and so on.

Comment 12 Sahina Bose 2017-11-21 06:29:42 UTC
Kasturi, is this bug still relevant? If so, what's the impact?

Comment 13 Simone Tiraboschi 2018-02-21 17:50:15 UTC
(In reply to Sahina Bose from comment #12)
> Kasturi, is this bug still relevant? If so, what's the impact?

Implicitly not relevant for node 0

Comment 14 Simone Tiraboschi 2018-02-22 16:16:25 UTC
It's implicitly fixed by the new node-zero design. Moving to QE for testing.

Comment 15 SATHEESARAN 2018-05-07 01:32:26 UTC
Tested with RHV 4.2.3 and ovirt-hosted-engine-setup-2.2.18

After HE deployment, the mount options are available after importing HostedEngine Storage

Comment 16 Sandro Bonazzola 2018-05-10 06:24:08 UTC
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.