Bug 1433404 - Creating a Cinder Volume using NFS Backend fails
Summary: Creating a Cinder Volume using NFS Backend fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: z4
: 10.0 (Newton)
Assignee: Alan Bishop
QA Contact: Tzach Shefi
URL:
Whiteboard:
: 1327616 (view as bug list)
Depends On:
Blocks: 1381612
TreeView+ depends on / blocked
 
Reported: 2017-03-17 15:07 UTC by David Peacock
Modified: 2021-12-10 14:59 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.2.0-24.el7ost, puppet-tripleo-5.6.0-4.el7ost, puppet-cinder-9.5.0-2.el7ost
Doc Type: Bug Fix
Doc Text:
Cause: The NFS backend driver for Cinder implements enhanced NAS security features that default to being enabled. However, the features require non-standard configuration changes in Nova's libvirt, and without those changes some cinder volume operations fail. Consequence: Some cinder volume operations fail when using the NFS backend. Fix: Add TripleO settings to control the NFS driver's NAS secure features, and disable the features by default. Result: Cinder volume operations no longer fail when using the NFS backend.
Clone Of:
Environment:
Last Closed: 2017-09-06 17:09:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
cinder volume log and others (29.55 KB, application/x-gzip)
2017-03-30 19:54 UTC, David Peacock
no flags Details
Attached example cinder.conf (169.99 KB, text/plain)
2017-03-31 03:57 UTC, Tzach Shefi
no flags Details
Custom templates from customer with Cinder / NFS issues (14.88 MB, application/x-gzip)
2017-04-13 20:08 UTC, David Peacock
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1688332 0 None None None 2017-05-04 15:56:04 UTC
OpenStack gerrit 462663 0 None MERGED Add support for Cinder "NAS secure" driver params 2021-01-04 01:15:36 UTC
OpenStack gerrit 462665 0 None MERGED Add support for Cinder "NAS secure" driver params 2021-01-04 01:15:36 UTC
OpenStack gerrit 462667 0 None MERGED Add support for Cinder "NAS secure" driver params 2021-01-04 01:15:36 UTC
Red Hat Bugzilla 1371911 0 unspecified CLOSED Cinder create volume from snapshot chmod permission error 2022-08-16 14:09:37 UTC
Red Hat Issue Tracker OSP-11278 0 None None None 2021-12-10 14:59:57 UTC
Red Hat Product Errata RHBA-2017:2654 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 director Bug Fix Advisory 2017-09-06 20:55:36 UTC

Internal Links: 1371911

Description David Peacock 2017-03-17 15:07:13 UTC
Description of problem:

Attempting to create an NFS-backed cinder volume fails; we need help with the custom template that allows this to work please.

Version-Release number of selected component (if applicable):

Red Hat OpenStack Platform 10

[stack@btrhlagrnce-h-ucld-01 ~]$ openstack volume list
+--------------------------------------+--------------+--------+------+-------------+
| ID                                   | Display Name | Status | Size | Attached to |
+--------------------------------------+--------------+--------+------+-------------+
| e26cfe2e-ad93-423a-a72b-519d29430c21 | test_nfs     | error  |    1 |             |
+--------------------------------------+--------------+--------+------+-------------+
Here is how I create the cinder type and volume
openstack volume type create NFS
openstack volume type set NFS --property volume_backend_name=tripleo_nfs
openstack volume create --size 1 --type NFS test_nfs
I troubleshot the issue and found a workaround for it but I want to make sure my template customization is right so I don’t need to use the workaround, below is my template customization for cinder
parameter_defaults:

  #### BACKEND SELECTION ####

  ## Whether to enable iscsi backend for Cinder.
  CinderEnableIscsiBackend: false
  ## Whether to enable rbd (Ceph) backend for Cinder.
  CinderEnableRbdBackend: false
  ## Cinder Backup backend can be either 'ceph' or 'swift'.
  CinderBackupBackend: swift
  ## Whether to enable NFS backend for Cinder.
  CinderEnableNfsBackend: true
  ## Whether to enable rbd (Ceph) backend for Nova ephemeral storage.
  NovaEnableRbdBackend: false
  ## Glance backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GlanceBackend: file
  ## Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GnocchiBackend: file


  #### CINDER NFS SETTINGS ####

  ## NFS mount options
  # CinderNfsMountOptions: ''
  ## NFS mount point, e.g. '192.168.122.1:/export/cinder'
  CinderNfsServers: '192.168.11.234:/cinder'
  CinderNfsMountOptions: 'nfsvers=3,intr,context=system_u:object_r:cinder_var_lib_t:s0'

From my troubleshooting the problem seems to be that the mountpoint is created with root ownership 
192.168.11.234:/cinder   99G  640K   99G   1% /var/lib/cinder/mnt/d524dc90a27972a8d2651fbc1f3b3e79
[root@btrhlagrnce-h-pe1dloc-003 ~]# cd /var/lib/cinder/mnt/
[root@btrhlagrnce-h-pe1dloc-003 mnt]# ll
total 8
drwxr-xr-x. 5 root root 1024 Feb 27 16:40 d524dc90a27972a8d2651fbc1f3b3e79
[root@btrhlagrnce-h-pe1dloc-003 mnt]# ls -la
total 8
drwxr-xr-x. 3 cinder cinder   46 Feb 28 00:13 .
drwxr-xr-x. 6 cinder cinder  139 Feb 28 00:21 ..
drwxr-xr-x. 5 root   root   1024 Feb 27 16:40 d524dc90a27972a8d2651fbc1f3b3e79
[root@btrhlagrnce-h-pe1dloc-003 mnt]#
And it looks like the driver is not running as root when tries to create the volume.
Can you please provide direction on how to customize the template to have cinder working without any workaround?

Comment 1 David Peacock 2017-03-23 18:00:53 UTC
Ping -

Any traction on this?  Customer is Verizon, and I believe this problem is starting to be a blocker for them.

Please let me know if there's anything I can do to answer any questions or provide more data.

Thank you,
David Peacock

Comment 2 David Peacock 2017-03-24 16:15:04 UTC
Hi engineering,

Is there anything you need here?

Thanks,
David

Comment 3 Eric Harney 2017-03-24 17:20:40 UTC
Please set the following settings in the nfs backend section of cinder.conf:

nas_secure_file_permissions=False
nas_secure_file_operations=False

and restart the cinder volume service.  This should get the NFS driver working.

Comment 4 David Peacock 2017-03-24 17:48:14 UTC
Thanks Eric, I'll give that a go. :-)

Comment 5 David Peacock 2017-03-27 20:02:56 UTC
I'm afraid this wasn't the special sauce.

Please let me know what you need from me; are we looking at a legitimate bug here?

stack@btrhlagrnce-h-ucld-01 ~]$ . overcloudrc
[stack@btrhlagrnce-h-ucld-01 ~]$ openstack volume type create NFS
+---------------------------------+--------------------------------------+
| Field                           | Value                                |
+---------------------------------+--------------------------------------+
| description                     | None                                 |
| id                              | 0e2568a2-453f-45a3-ad00-efcf85b08bbc |
| is_public                       | True                                 |
| name                            | NFS                                  |
| os-volume-type-access:is_public | True                                 |
+---------------------------------+--------------------------------------+
[stack@btrhlagrnce-h-ucld-01 ~]$ openstack volume type set NFS --property volume_backend_name=tripleo_nfs
[stack@btrhlagrnce-h-ucld-01 ~]$ openstack volume create --size 1 --type NFS nfs_workaround
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2017-03-27T18:12:21.580279           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | 4cc6102b-c79a-4b78-935a-a69cbea99704 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | nfs_workaround                       |
| properties          |                                      |
| replication_status  | disabled                             |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | NFS                                  |
| updated_at          | None                                 |
| user_id             | f2383e92683a4615a165641b3c4ca69f     |
+---------------------+--------------------------------------+
[stack@btrhlagrnce-h-ucld-01 ~]$ openstack volume list
+--------------------------------------+----------------+--------+------+-------------+
| ID                                   | Display Name   | Status | Size | Attached to |
+--------------------------------------+----------------+--------+------+-------------+
| 4cc6102b-c79a-4b78-935a-a69cbea99704 | nfs_workaround | error  |    1 |             |
+--------------------------------------+----------------+--------+------+-------------+
[stack@btrhlagrnce-h-ucld-01 ~]$

[root@btrhlagrnce-h-pe4dloc-003 ~]# cat /etc/cinder/cinder.conf | grep 'nas_secure_file'
#nas_secure_file_operations = auto
#nas_secure_file_permissions = auto
nas_secure_file_permissions=False
nas_secure_file_operations=False
[root@btrhlagrnce-h-pe4dloc-003 ~]#

[root@btrhlagrnce-h-pe4dloc-003 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: btrhlagrnce-h-pe4dloc-003 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Mon Mar 27 18:11:11 2017          Last change: Mon Mar 27 18:10:24 2017 by root via crm_resource on btrhlagrnce-h-pe4dloc-003

3 nodes and 19 resources configured

Online: [ btrhlagrnce-h-pe4dloc-001 btrhlagrnce-h-pe4dloc-002 btrhlagrnce-h-pe4dloc-003 ]

Full list of resources:

 ip-192.168.11.254      (ocf::heartbeat:IPaddr2):       Started btrhlagrnce-h-pe4dloc-001
 Clone Set: haproxy-clone [haproxy]
     Started: [ btrhlagrnce-h-pe4dloc-001 btrhlagrnce-h-pe4dloc-002 btrhlagrnce-h-pe4dloc-003 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ btrhlagrnce-h-pe4dloc-001 btrhlagrnce-h-pe4dloc-002 btrhlagrnce-h-pe4dloc-003 ]
 ip-192.168.12.9        (ocf::heartbeat:IPaddr2):       Started btrhlagrnce-h-pe4dloc-002
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ btrhlagrnce-h-pe4dloc-001 btrhlagrnce-h-pe4dloc-002 btrhlagrnce-h-pe4dloc-003 ]
 ip-192.168.8.5 (ocf::heartbeat:IPaddr2):       Started btrhlagrnce-h-pe4dloc-003
 ip-192.168.12.7        (ocf::heartbeat:IPaddr2):       Started btrhlagrnce-h-pe4dloc-001
 Master/Slave Set: redis-master [redis]
     Masters: [ btrhlagrnce-h-pe4dloc-003 ]
     Slaves: [ btrhlagrnce-h-pe4dloc-001 btrhlagrnce-h-pe4dloc-002 ]
 ip-2001.4888.a42.3101.420.fe0.0.2000   (ocf::heartbeat:IPaddr2):       Started btrhlagrnce-h-pe4dloc-002
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started btrhlagrnce-h-pe4dloc-003
 ip-10.217.162.138      (ocf::heartbeat:IPaddr2):       Started btrhlagrnce-h-pe4dloc-001

Comment 6 Eric Harney 2017-03-27 20:05:27 UTC
(In reply to David Peacock from comment #5)
> [root@btrhlagrnce-h-pe4dloc-003 ~]# cat /etc/cinder/cinder.conf | grep
> 'nas_secure_file'
> #nas_secure_file_operations = auto
> #nas_secure_file_permissions = auto
> nas_secure_file_permissions=False
> nas_secure_file_operations=False

Can you confirm that these settings are in the driver backend section of cinder.conf and not the default section?  I can't tell from this output.

(Just grabbing cinder.conf may be easiest.)

Comment 7 Eric Harney 2017-03-27 20:06:14 UTC
We should also get a cinder volume log to see what's really going on.

Comment 8 David Peacock 2017-03-27 20:07:57 UTC
I'll check that Eric.  Thanks.

Comment 9 David Peacock 2017-03-30 19:17:29 UTC
Working through some logistical issues with the customer in getting these details.

Can you confirm the specific section of cinder.conf you're talking about, Eric?

`[BACKEND]`?  Or does it have another name?

Thank you,
David

Comment 10 David Peacock 2017-03-30 19:53:37 UTC
The customer advises me that the lines were added in the [DEFAULT] section of the cinder.conf.

Please find attached the log files from the last test.

Thank you,
David

Comment 11 David Peacock 2017-03-30 19:54:10 UTC
Created attachment 1267667 [details]
cinder volume log and others

Comment 12 Tzach Shefi 2017-03-31 03:57:22 UTC
Created attachment 1267711 [details]
Attached example cinder.conf

I've attached an example cinder.conf (nfs) file from my RFE mentioned below. 

Look at Eric's #6 on this bug.

Review a recent NFS RFE: 
https://bugzilla.redhat.com/show_bug.cgi?id=1161413#c16

Notice initially I also changed stuff under default section - not good :) 

The correct method later on #19. 
Under Cinder.conf's default section you should only change this:
enabled_backend=nfs    (nfs was my chosen name, use anything you like)

Then at the bottom of cinder.conf create a new section

[nfs]                   (same name as ^)
volume_backend_name=nfs  
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/
..


NFS mount tips I used suggest you try:

If you run the #mount command on controller, does the cinder mount show up?  

If it doesn't show up, can you successfully mount the share manually?

Can you R/W a temp file on that mount ?

Another NFS issue hit on my RFE, not sure it's relevant to your case probably isn't but just in case. If your NFS clients (controller compute nodes) are behind a NAT before they reach the NFS server you need to allow insecure nfs share option.  
/export/ins_cinder  *(rw,insecure,no_root_squash)

Comment 13 David Peacock 2017-04-11 12:10:22 UTC
@Tzach, Thank you for this information; I'm working with our customer to see how this works out for them.  I'll be back in touch.

David

Comment 14 David Peacock 2017-04-13 20:05:43 UTC
Hi guys,

I have confirmation back that with the settings in the correct section of the cinder.conf, the mountpoints do work correctly.

What I need from engineering next is two fold:

1) An understanding of why when the customer configures the templates as attached, which looks at face-value to be the idiomatic way as recommended by comments, these crucial cinder settings aren't put in the right place in the ultimate cinder.conf

and

2) Advise on how best to templatize this so that it does work out of the box on a fresh deployment without any post-deployment workaround.

Thank you very much indeed for your help; I (and our customer) really appreciate it.

David

Comment 15 David Peacock 2017-04-13 20:08:13 UTC
Created attachment 1271537 [details]
Custom templates from customer with Cinder / NFS issues

These are the customer's current templates; they look to me correct but result in our known issue as has been worked in this BZ.

I'd like to know the best practise for modifying these so that the crucial settings are introduced in the correct (non-default) section of the resulting cinder.conf

Comment 16 Alan Bishop 2017-04-13 22:39:52 UTC
Hi David, we hope to fully resolve this soon. See bug #1393924. It's a now a priority for OSP-11.

Comment 17 David Peacock 2017-04-18 13:57:19 UTC
Thank you Alan.

What's the timeline for OSP-11 at this point?  I have advised Verizon to continue with their work-around for now pending a bugfix.

Thank you,
David

Comment 18 Alan Bishop 2017-04-18 16:18:18 UTC
The current OSP-11 schedule is RC on Apr-27 and GA on May-18. However, I do not yet know the timeline for bug fix. That is, I cannot say whether it will be pre or post GA.

Comment 19 David Peacock 2017-04-18 16:28:11 UTC
That's good enough as a guesstimate.  Thanks a lot Alan; much appreciated.

David

Comment 20 Paul Grist 2017-05-17 02:46:44 UTC
*** Bug 1327616 has been marked as a duplicate of this bug. ***

Comment 24 Tzach Shefi 2017-08-16 20:00:22 UTC
Verified on:

openstack-tripleo-heat-templates-5.3.0-2.el7ost.noarch
puppet-tripleo-5.6.1-1.el7ost.noarch
puppet-cinder-9.5.0-2.el7ost.noarch

Configured Cinder with NFS backend via THT template
Cinder create worked

[stack@undercloud-0 ~]$ cinder show 11209f55-5cbf-41f5-b0c7-af059d43f000
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2017-08-16T19:50:02.000000           |
..                               |
| encrypted                      | False                                |
| id                             | 11209f55-5cbf-41f5-b0c7-af059d43f000 |
..                              |
| multiattach                    | False                                |
| name                           | one                                  |
| os-vol-host-attr:host          | hostgroup@tripleo_nfs#tripleo_nfs    |
                                 |
...
| status                         | available                            |



storage yaml
_______________________
## Whether to enable iscsi backend for Cinder.
  CinderEnableIscsiBackend: false
  ## Whether to enable rbd (Ceph) backend for Cinder.
  CinderEnableRbdBackend: false
  ## Cinder Backup backend can be either 'ceph' or 'swift'.
  CinderBackupBackend: false
  ## Whether to enable NFS backend for Cinder.
  CinderEnableNfsBackend: true
  ## Whether to enable rbd (Ceph) backend for Nova ephemeral storage.
  NovaEnableRbdBackend: false
  ## Glance backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GlanceBackend: swift
  ## Gnocchi backend can be either 'rbd' (Ceph), 'swift' or 'file'.
  GnocchiBackend: swift


  #### CINDER NFS SETTINGS ####

  ## NFS mount options
  CinderNfsMountOptions: ''
  ## NFS mount point, e.g. '192.168.122.1:/export/cinder'
  CinderNfsServers: '10.35.160.111:/export/ins_cinder'
_____________________

Cinder nfs config section
enabled_backends = tripleo_nfs

[tripleo_nfs]
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nas_secure_file_permissions=False
nfs_shares_config=/etc/cinder/shares-nfs.conf
nfs_mount_options=
nas_secure_file_operations=False
volume_backend_name=tripleo_nfs

Comment 27 errata-xmlrpc 2017-09-06 17:09:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2654


Note You need to log in before you can comment on or make changes to this bug.