Bug 1491597

Summary: Volume create fails, when heat template configures nfs for both Cinder & Glance.
Product: Red Hat OpenStack Reporter: Tzach Shefi <tshefi>
Component: openstack-tripleo-heat-templatesAssignee: Alan Bishop <abishop>
Status: CLOSED EOL QA Contact: Gurenko Alex <agurenko>
Severity: medium Docs Contact:
Priority: medium    
Version: 11.0 (Ocata)CC: aschultz, mburns, pgrist, rhel-osp-director-maint
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-22 12:33:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Logs
none
nfs11_GlanceCinder.yaml
none
nfs11Cinder.yaml
none
Logs none

Description Tzach Shefi 2017-09-14 08:58:01 UTC 
Created attachment 1325843 [details]
Logs

Description of problem: When I used attached (nfs11_GlanceCinder.yaml) heat template to configure both Cinder and Glance to mount to NFS share, Glance mount works but Cinder's isn't created.

When I used this template nfs11Cinder.yaml, just Cinder without Glance, create volume works fine. 

Version-Release number of selected component (if applicable):
rhel7.4

openstack-tripleo-heat-templates-6.2.0-3.el7ost.noarch
puppet-tripleo-6.5.0-8.el7ost.noarch
puppet-cinder-10.3.1-1.el7ost.noarch


How reproducible:
Every time (twice already) 

Steps to Reproduce:
1. Use attached nfs yaml and deploy overcloud
2. Try to create a volume. 
3.

Actual results:
Volume in error state as Cinder nfs mount is broke.

Expected results:
Volume be available. 

Additional info:

I hit this bug while working on another heat storage nfs bz
https://bugzilla.redhat.com/show_bug.cgi?id=1440700#c12

While debugging ^ bug, Eric/Alan determined it's a side effect of SELinux.
Grep for "mount" in controller's /var/log/messages and see "SELinux: mount invalid. 
Same superblock, different security settings for (dev 0:41, type nfs4)"
Suggested add "nosharecache" to both the cinder and glance nfs mount options

See https://bugzilla.redhat.com/show_bug.cgi?id=253530
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/SELinux_Users_and_Administrators_Guide/sect-Security-Enhanced_Linux-Working_with_SELinux-Mounting_File_Systems.html#sect-Security-Enhanced_Linux-Mounting_File_Systems-Multiple_NFS_Mounts

Suggested add "nosharecache" to both the cinder and glance nfs mount options
Comment 1 Tzach Shefi 2017-09-14 08:59:48 UTC 
Created attachment 1325844 [details]
nfs11_GlanceCinder.yaml
Comment 2 Tzach Shefi 2017-09-14 09:00:06 UTC 
Created attachment 1325846 [details]
nfs11Cinder.yaml
Comment 3 Tzach Shefi 2017-09-14 11:46:42 UTC 
Created attachment 1325941 [details]
Logs

Some mode debugging info, deleted overcloud 
Attempted with this template 

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: ''     --> removed the retry=1, which was here before. 
  CinderNfsServers: '10.35.160.111:/export/ins_cinder'

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsShare: '10.35.160.111:/export/ins_glance'


Cinder create still fails, Cinder mount still missing.
Comment 4 Tzach Shefi 2017-09-14 12:51:51 UTC 
OK looks like setting mount option->  nosharecache resolves the issue. 

On this last attempt used this heat template:  

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'nosharecache'
  CinderNfsServers: '10.35.160.111:/export/ins_cinder'

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsMountOptions: 'nosharecache'
  GlanceNfsShare: '10.35.160.111:/export/ins_glance'


Volume is available
$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID                                   | Status    | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 08945b6f-83f9-442c-bbd5-b19bc2a6dc13 | available | -    | 1    | -           | false    |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+

Not sure what the outcome of this bug is:
1. Maybe check if we should add nosharecache by default? 

2. At least document this issue.
Comment 6 Paul Grist 2018-03-21 17:19:23 UTC 
Is this one worth fixing or do we want to set the fields to just make sure we document it?
Comment 7 Scott Lewis 2018-06-22 12:33:31 UTC 
OSP11 is now retired, see details at https://access.redhat.com/errata/product/191/ver=11/rhel---7/x86_64/RHBA-2018:1828