Cinder's NFS driver supports snapshots, but the feature is disabled by default. This BZ is a backport of the fix that allows the director to enable the NFS driver's snapshot feature. +++ This bug was initially created as a clone of Bug #1633146 +++
According to our records, this should be resolved by openstack-tripleo-heat-templates-9.3.1-0.20190513171738.el7ost. This build is available now.
According to our records, this should be resolved by puppet-tripleo-9.4.1-0.20190508182403.89735a1.el7ost. This build is available now.
Verified on: openstack-tripleo-heat-templates-9.3.1-0.20190513171738.el7ost.noarch 1. At first I'll test the default value, should be snapshot enabled true. (overcloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock1.rdu2.redhat.com \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/storage/cinder-nfs.yaml \ -> Added Cinder nfs yaml -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e /home/stack/virt/extra_templates.yaml \ -> here lay the cinder parameters, see below --log-file overcloud_deployment_55.log (overcloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true CinderNfsMountOptions: 'retry=1' CinderNfsServers: 'x.y.w.z:/export/ins_cinder' -> w.x.y.z replace internal IPs, notice no mention of snapshot param so as to check default THT setting. 2. Once overcloud is deployed we see Cinder has an NFS backend: (overcloud) [stack@undercloud-0 ~]$ cinder service-list +------------------+-----------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+-----------------------+------+---------+-------+----------------------------+-----------------+ | cinder-backup | controller-1 | nova | enabled | up | 2019-08-19T12:44:05.000000 | - | | cinder-scheduler | controller-0 | nova | enabled | up | 2019-08-19T12:44:03.000000 | - | | cinder-scheduler | controller-1 | nova | enabled | up | 2019-08-19T12:44:03.000000 | - | | cinder-scheduler | controller-2 | nova | enabled | up | 2019-08-19T12:44:03.000000 | - | | cinder-volume | hostgroup@tripleo_nfs | nova | enabled | up | 2019-08-19T12:44:05.000000 | - | +------------------+-----------------------+------+---------+-------+----------------------------+-----------------+ Which is great Cinder runs over NFS. The resulting Cinder backend section as expected from the default includes snapshot support. [root@controller-0 ~]# tail /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf [tripleo_nfs] backend_host=hostgroup volume_backend_name=tripleo_nfs volume_driver=cinder.volume.drivers.nfs.NfsDriver nfs_shares_config=/etc/cinder/shares-nfs.conf nfs_mount_options=retry=1 nfs_snapshot_support=True --> Yay default true works! nas_secure_file_operations=False nas_secure_file_permissions=False 3. Create a cinder volume: (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --name vol_nfs +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2019-08-19T12:45:05.000000 | | description | None | | encrypted | False | | id | 415f711a-7269-4032-bbbf-d3832a0c1246 | | metadata | {} | | migration_status | None | | multiattach | False | | name | vol_nfs | | os-vol-host-attr:host | hostgroup@tripleo_nfs#tripleo_nfs | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 3cf2c871282447a1b617fd6e38a4f309 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | 2019-08-19T12:45:06.000000 | | user_id | 1cdf4bf222c94be088f8b2ed68986abd | | volume_type | tripleo | +--------------------------------+--------------------------------------+ Volume created on NFS backend (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+---------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------+------+-------------+----------+-------------+ | 415f711a-7269-4032-bbbf-d3832a0c1246 | available | vol_nfs | 1 | tripleo | false | | +--------------------------------------+-----------+---------+------+-------------+----------+-------------+ 4. Create a snapshot of said volume: (overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create 415f711a-7269-4032-bbbf-d3832a0c1246 --name Snap +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2019-08-19T12:46:18.096248 | | description | None | | id | 1aadf298-7e98-4cd3-8a66-0385c0c81657 | | metadata | {} | | name | Snap | | size | 1 | | status | creating | | updated_at | None | | volume_id | 415f711a-7269-4032-bbbf-d3832a0c1246 | +-------------+--------------------------------------+ A while later snap on an NFS volume is successfully created, so we know THT's default of enabled/true works! (overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list +--------------------------------------+--------------------------------------+-----------+------+------+ | ID | Volume ID | Status | Name | Size | +--------------------------------------+--------------------------------------+-----------+------+------+ | 1aadf298-7e98-4cd3-8a66-0385c0c81657 | 415f711a-7269-4032-bbbf-d3832a0c1246 | available | Snap | 1 | +--------------------------------------+--------------------------------------+-----------+------+------+ Now lets cleanup snap/vol and delete overcloud, this time wish snapshot disabled Cinder snapshot-delete .. cinder delete .. (undercloud) [stack@undercloud-0 ~]$ openstack stack delete overcloud --wait --yes .. Now we edit/add last line (overcloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml parameter_defaults: CinderEnableIscsiBackend: false CinderEnableRbdBackend: false CinderEnableNfsBackend: true CinderNfsMountOptions: 'retry=1' CinderNfsServers: 'x.y.w.z:/export/ins_cinder' CinderNfsSnapshotSupport: false -> add false to disable snapshot on NFS. Redeploy overcloud wait, wait for it.. :) We have an overcloud with Cinder using NFS as backend: (overcloud) [stack@undercloud-0 ~]$ cinder service-list +------------------+-----------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+-----------------------+------+---------+-------+----------------------------+-----------------+ | cinder-backup | controller-1 | nova | enabled | up | 2019-08-19T14:07:43.000000 | - | | cinder-scheduler | controller-0 | nova | enabled | up | 2019-08-19T14:07:44.000000 | - | | cinder-scheduler | controller-1 | nova | enabled | up | 2019-08-19T14:07:43.000000 | - | | cinder-scheduler | controller-2 | nova | enabled | up | 2019-08-19T14:07:44.000000 | - | | cinder-volume | hostgroup@tripleo_nfs | nova | enabled | up | 2019-08-19T14:07:49.000000 | - | +------------------+-----------------------+------+---------+-------+----------------------------+-----------------+ The resulting Cinder backend with disabled snapshot support: [root@controller-0 ~]# tail /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf [tripleo_nfs] backend_host=hostgroup volume_backend_name=tripleo_nfs volume_driver=cinder.volume.drivers.nfs.NfsDriver nfs_shares_config=/etc/cinder/shares-nfs.conf nfs_mount_options=retry=1 nfs_snapshot_support=False -> great as expected. nas_secure_file_operations=False nas_secure_file_permissions=False Create a Cinder vol: (overcloud) [stack@undercloud-0 ~]$ cinder create 1 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2019-08-19T15:19:09.000000 | | description | None | | encrypted | False | | id | fe867488-bb38-4251-aada-b086f78a8a74 | | metadata | {} | | migration_status | None | | multiattach | False | | name | None | | os-vol-host-attr:host | hostgroup@tripleo_nfs#tripleo_nfs | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | f2d2d1df82a445588e56da683691caa7 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | 2019-08-19T15:19:10.000000 | | user_id | 604b395c1968478ab39b2fab7382557b | | volume_type | tripleo | +--------------------------------+--------------------------------------+ Try to create a vol snapshot, it should fail: (overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create fe867488-bb38-4251-aada-b086f78a8a74 +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2019-08-19T15:20:01.401011 | | description | None | | id | 8d6024f9-b98d-425d-b416-0b12ecbaf0ef | | metadata | {} | | name | None | | size | 1 | | status | creating | | updated_at | None | | volume_id | fe867488-bb38-4251-aada-b086f78a8a74 | +-------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list +--------------------------------------+--------------------------------------+--------+------+------+ | ID | Volume ID | Status | Name | Size | +--------------------------------------+--------------------------------------+--------+------+------+ | 8d6024f9-b98d-425d-b416-0b12ecbaf0ef | fe867488-bb38-4251-aada-b086f78a8a74 | error | - | 1 | +--------------------------------------+--------------------------------------+--------+------+------+ c-vol expected error: on during message handling: VolumeDriverException: Volume driver reported an error: NFS driver snapshot support is disabled in cinder.conf. 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_snapshot 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1072, in create_snapshot 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server snapshot.save() 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1064, in create_snapshot 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server model_update = self.driver.create_snapshot(snapshot) 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_snapshot 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 565, in create_snapshot 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server self._check_snapshot_support() 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 550, in _check_snapshot_support 2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server VolumeDriverException: Volume driver reported an error: NFS driver snapshot support is disabled in cinder.conf. 2019-08-19 15:20:33.035 61 DEBUG oslo_concurrency.processutils [req-b9e83187-cc74-4798-83c0-0f8d59286ef8 - - - - -] Running cmd (subprocess): du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/47266020eacec99097bdec49f2451d38 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372 Good to verify.