Bug 1263602

Summary: [RFE] Ability to set different mount options for hosted_engine nfs storage than the default
Product: Red Hat Enterprise Virtualization Manager Reporter: Alexandros Gkesos <agkesos>
Component: RFEsAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED ERRATA QA Contact: Nikolai Sednev <nsednev>
Severity: medium Docs Contact:
Priority: high    
Version: 3.5.4CC: achareka, apinnick, lsurette, mavital, nsoffer, ppostler, pstehlik, rbalakri, sasundar, sbonazzo, srevivo, stirabos, tnisan, trichard, ykaul, ylavi
Target Milestone: ovirt-4.2.1Keywords: FutureFeature, Triaged
Target Release: ---Flags: lsvaty: testing_plan_complete-
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Additional mount options for the self-hosted engine storage domain, such as NFS mount, are available in the Manager.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-15 17:36:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1359265    
Bug Blocks:    

Description Alexandros Gkesos 2015-09-16 09:38:43 UTC
3. What is the nature and description of the request?  

- It should be possible to use other mount options then the default one.
      
4. Why does the customer need this? (List the business requirements here)  
5. How would the customer like to achieve this? (List the functional requirements here)  

- IT security requires mounts to be nodev,noexec,nosuid
      
6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.  

- Set up self-hosted-engine (RHEV) on NFS storage domain.
      
7. Is there already an existing RFE upstream or in Red Hat Bugzilla?  

- None that i am aware of.
      
8. Does the customer have any specific timeline dependencies and which release would they like to target 
      
- RHEL6 + RHEL7

9. Is the sales team involved in this request and do they have any additional input?  

- No
      
10. List any affected packages or components.  

- ovirt-hosted-engine-ha
      
11. Would the customer be able to assist in testing this functionality if implemented?  

- Yes

Comment 7 Sandro Bonazzola 2016-07-06 13:11:37 UTC
Simone, please create an upstream bug blocking this one for 4.1.

Comment 8 Simone Tiraboschi 2016-07-06 13:18:04 UTC
Now in /etc/ovirt-hosted-engine/hosted-engine.conf we already have a mnt_options= line for this kind of options.
It's already possible to pass a value for it via answerfile.
We just have to let the user optionally customize that value from the text UI; the question is if we want also to parse/validate that options somehow.

Comment 9 Yaniv Lavi 2016-07-06 13:30:46 UTC
(In reply to Simone Tiraboschi from comment #8)
> Now in /etc/ovirt-hosted-engine/hosted-engine.conf we already have a
> mnt_options= line for this kind of options.
> It's already possible to pass a value for it via answerfile.
> We just have to let the user optionally customize that value from the text
> UI; the question is if we want also to parse/validate that options somehow.

How does that work with the shared conf and adding hosts via UI\API?
Is this shared? Can you set it via UI?

Comment 10 Simone Tiraboschi 2016-07-06 14:10:15 UTC
(In reply to Yaniv Dary from comment #9)
> How does that work with the shared conf and adding hosts via UI\API?
> Is this shared? Can you set it via UI?

It should be part of the auto-import procedure: the auto-import procedure will import the hosted-engine storage domain with its mount options and it should also pass that value to host-deploy to write it in hosted-engine.conf on each host.

Probably we miss the last step to close the gap. Roy?

Comment 11 Roy Golan 2016-07-11 10:23:21 UTC
(In reply to Simone Tiraboschi from comment #10)
> (In reply to Yaniv Dary from comment #9)
> > How does that work with the shared conf and adding hosts via UI\API?
> > Is this shared? Can you set it via UI?
> 
> It should be part of the auto-import procedure: the auto-import procedure
> will import the hosted-engine storage domain with its mount options and it
> should also pass that value to host-deploy to write it in hosted-engine.conf
> on each host.
> 
> Probably we miss the last step to close the gap. Roy?

The engine will pass whatever there in the hosted-engine.conf. 

So if that would be updated then for the HE hosts, we are covered.

The problem is with non HE hosts which are connected from the backend to that storage domain, does it matter there even though those host will never read/write to that domain because the hosted engine vm will not run there? (and in 4.1 we will prevent creating other disks there)

Comment 12 Simone Tiraboschi 2016-07-11 10:52:46 UTC
(In reply to Roy Golan from comment #11)
> (In reply to Simone Tiraboschi from comment #10)
> > (In reply to Yaniv Dary from comment #9)
> > > How does that work with the shared conf and adding hosts via UI\API?
> > > Is this shared? Can you set it via UI?
> > 
> > It should be part of the auto-import procedure: the auto-import procedure
> > will import the hosted-engine storage domain with its mount options and it
> > should also pass that value to host-deploy to write it in hosted-engine.conf
> > on each host.
> > 
> > Probably we miss the last step to close the gap. Roy?
> 
> The engine will pass whatever there in the hosted-engine.conf. 
> 
> So if that would be updated then for the HE hosts, we are covered.

Yes, it's mnt_options under hosted-engine.conf 

But I see a possible race conditions here: if I correctly understood what you are saying, we are not fetching the mount options when we are importing the hosted-engine storage domain into the engine; the trick is just that ovirt-ha-agent will connect the storage domain before the engine, and so the engine connection (without the mnt_options) will be not effective since the hosted-engine storage domain is already connected.

This will work for sure for the first host and for additional hosts deployed via CLI.
I'm not really sure we will not have race conditions when we deploy an host from the engine since in this case the engine and ovirt-ha-agent are running independently and only the first one will mount.
See: https://bugzilla.redhat.com/show_bug.cgi?id=1349532#c18

Then there is another possible issue: 
setting an host into maintenance mode from the engine will also disconnect the hosted-engine storage domain; the agent will try to consume it every 35 second and if not available it will reconnect.
If the user exits maintenance mode before that, the engine could reconnect before the agent loosing mnt_options.
see https://bugzilla.redhat.com/show_bug.cgi?id=1353608#c5


> The problem is with non HE hosts which are connected from the backend to
> that storage domain, does it matter there even though those host will never
> read/write to that domain because the hosted engine vm will not run there?
> (and in 4.1 we will prevent creating other disks there)

In general I think that correctly importing it with the right mnt_options will be a safer design.

Comment 14 Roy Golan 2016-07-11 12:33:29 UTC
The auto-import doesn't have the mnt options in the context and we can't pull the hosted-engine.conf disk at that point. We need vdsm to expose those mount options it possible and that would be the best. Using the hosted-engine.conf should be used by the ha-agent. 

Allon, exposing mount options on getStorageDomainInfo is on your plan? if not can we consider it for hosted engine?

Comment 15 Allon Mureinik 2016-07-11 16:29:33 UTC
(In reply to Roy Golan from comment #14)
> The auto-import doesn't have the mnt options in the context and we can't
> pull the hosted-engine.conf disk at that point. We need vdsm to expose those
> mount options it possible and that would be the best. Using the
> hosted-engine.conf should be used by the ha-agent. 
> 
> Allon, exposing mount options on getStorageDomainInfo is on your plan? if
> not can we consider it for hosted engine?

My instinctive answer would be to say no, as domains are logical entities and the mount options are StorageConnection details.
BUT... I see it does return the remove path for some reason, so it makes me rethink things.

Nir - your two cents?

Comment 16 Nir Soffer 2016-08-29 15:34:28 UTC
(In reply to Allon Mureinik from comment #15)
> (In reply to Roy Golan from comment #14)
> > The auto-import doesn't have the mnt options in the context and we can't
> > pull the hosted-engine.conf disk at that point. We need vdsm to expose those
> > mount options it possible and that would be the best. Using the
> > hosted-engine.conf should be used by the ha-agent. 
> > 
> > Allon, exposing mount options on getStorageDomainInfo is on your plan? if
> > not can we consider it for hosted engine?
> 
> My instinctive answer would be to say no, as domains are logical entities
> and the mount options are StorageConnection details.
> BUT... I see it does return the remove path for some reason, so it makes me
> rethink things.
> 
> Nir - your two cents?

I don't see a reason to hide the mount options, and it can be useful also
for DR.

The information is available in /proc/mounts and can be easily exposed in
vdsm apis as file storage domain specific info.

Comment 17 Roy Golan 2016-08-30 14:17:50 UTC
Should I open a bug against VDSM for that?

Comment 18 Nir Soffer 2016-08-30 14:57:11 UTC
(In reply to Roy Golan from comment #17)
> Should I open a bug against VDSM for that?

Seems like a good idea for tracking this.

Comment 19 Allon Mureinik 2016-08-30 15:02:11 UTC
(In reply to Roy Golan from comment #17)
> Should I open a bug against VDSM for that?

Yes please.

Comment 20 Sandro Bonazzola 2016-09-01 07:52:27 UTC
Roy, please open it and reference it here.

Comment 23 Nikolai Sednev 2017-11-05 15:31:56 UTC
Simone, may you kindly provide more info regarding reproduction steps, what was the solution exactly (to get additional nfs opions within the otopi during deployment, or to fetch these using /etc/ovirt-hosted-engine/hosted-engine.conf)?
Is there any documentation covering this new capability?

Comment 24 Simone Tiraboschi 2017-11-13 10:14:38 UTC
You have to see a question like this: 'If needed, specify additional mount options for the connection to the hosted-engine storage domain'
and you can enter additional mount options there.

The options added by the user should be saved under mnt_options field in /etc/ovirt-hosted-engine/hosted-engine.conf on the host itself and also on additional hosts.

The engine should care them as well importing the hosted-engine storage domain (you could see them trying editing the storage domain) and they should be effective on all the hosts of the storage pool (check the result of mount command on all the hosts).

Mount options should also be tried checking the NFS mount at customization stage.
If the mount fails hosted-engine-setup should report and ask again.
So please try also negative flow with dumb options.

Comment 25 Simone Tiraboschi 2017-11-13 10:17:08 UTC
(In reply to Alexandros Gkesos from comment #0)
> 8. Does the customer have any specific timeline dependencies and which
> release would they like to target 
>       
> - RHEL6 + RHEL7

This bug is targeted to 4.2 which covers only RHEL 7.

Comment 26 Nikolai Sednev 2017-11-13 13:05:14 UTC
(In reply to Simone Tiraboschi from comment #24)
> You have to see a question like this: 'If needed, specify additional mount
> options for the connection to the hosted-engine storage domain'
> and you can enter additional mount options there.
> 
> The options added by the user should be saved under mnt_options field in
> /etc/ovirt-hosted-engine/hosted-engine.conf on the host itself and also on
> additional hosts.
> 
> The engine should care them as well importing the hosted-engine storage
> domain (you could see them trying editing the storage domain) and they
> should be effective on all the hosts of the storage pool (check the result
> of mount command on all the hosts).
> 
> Mount options should also be tried checking the NFS mount at customization
> stage.
> If the mount fails hosted-engine-setup should report and ask again.
> So please try also negative flow with dumb options.

During latest deployment Today, I didn't seen expected message, e.g. "If needed, specify additional mount options for the connection to the hosted-engine storage domain".

Tested on these components:
ovirt-hosted-engine-setup-2.2.0-0.0.master.20171110120732.git35143a6.el7.centos.noarch

See deployment flow here: http://pastebin.test.redhat.com/531649

Moving back to assigned.

Comment 27 Simone Tiraboschi 2017-11-13 13:08:27 UTC
(In reply to Nikolai Sednev from comment #26)
> Moving back to assigned.

Yes, sorry, fault of mine:
https://github.com/oVirt/ovirt-hosted-engine-setup/blob/master/src/plugins/gr-he-setup/storage/nfs.py#L55

We blocked it on hosted-engine-setup side till https://bugzilla.redhat.com/show_bug.cgi?id=1359265 get correctly fixed on engine-side

Comment 28 Sandro Bonazzola 2017-11-20 08:52:29 UTC
I understand from Yaniv lavi that POSIX domain can be used for setting custom properties here.

Comment 29 Allon Mureinik 2017-11-21 11:00:48 UTC
(In reply to Sandro Bonazzola from comment #28)
> I understand from Yaniv lavi that POSIX domain can be used for setting
> custom properties here.

Both NFS and POSIX support custom mount options from engine/vdsm side.
If there's something missing on HE setup's side, we should fix it.
In any event, we should not use POSIX type for NFS domains (even though it should work) - there are several optimizations and special treatments for NFS

Comment 30 Red Hat Bugzilla Rules Engine 2017-11-21 11:00:57 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 32 Nikolai Sednev 2017-11-29 14:08:13 UTC
During latest deployment Today, I didn't seen expected message, e.g. "If needed, specify additional mount options for the connection to the hosted-engine storage domain".
Tested on these components:
ovirt-hosted-engine-setup-2.2.0-0.0.master.20171127213100.git92f0f9a.el7.centos.noarch
ovirt-hosted-engine-ha-2.2.0-0.0.master.20171122155227.20171122155225.gitbc3ec09.el7.centos.noarch
See deployment flow here: http://pastebin.test.redhat.com/536131
Moving back to assigned.

Comment 33 Simone Tiraboschi 2017-11-29 17:51:23 UTC
We have two distinct flows here:

a. in the old hosted-engine flow, ovirt-hosted-engine-setup directly creates the storage domain trough vdsm. We are not asking about custom mount_options there since vdsm will not report them back to the engine and so they will be absolutely ignored by the engine at auto import time. See: https://bugzilla.redhat.com/1359265

b. in the new ansible based flow (Node 0) instead, ovirt-hosted-engine-setup will create the NFS storage domain trough the engine running on the local VM and so all the custom mount options will be in the DB by design. In this case ovirt-hosted-engine-setup is asking about custom mount option since we know that they reach the engine.

Comment 34 Simone Tiraboschi 2018-01-15 09:39:06 UTC
Fixed on node zero

Comment 36 Nikolai Sednev 2018-01-21 06:47:25 UTC
Moving to verified forth to https://bugzilla.redhat.com/show_bug.cgi?id=1359265#c25.

Comment 39 errata-xmlrpc 2018-05-15 17:36:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1488

Comment 40 Franta Kust 2019-05-16 13:09:21 UTC
BZ<2>Jira Resync