Bug 1698682
| Summary: | [OSP14] Cannot configure nfs_snapshot_support properly in director | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alan Bishop <abishop> | |
| Component: | openstack-tripleo-heat-templates | Assignee: | Alan Bishop <abishop> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Tzach Shefi <tshefi> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 14.0 (Rocky) | CC: | abishop, cschwede, knoha, lmarsh, mburns, rheslop, scohen, tenobreg, tkajinam, tshefi | |
| Target Milestone: | z3 | Keywords: | TestOnly, Triaged, ZStream | |
| Target Release: | 14.0 (Rocky) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-heat-templates-9.3.1-0.20190513171736.9995be9.el7ost.noarch.rpm puppet-tripleo-9.4.1-0.20190508182403.89735a1.el7ost.noarch.rpm | Doc Type: | Enhancement | |
| Doc Text: |
Red Hat OpenStack Platform director now has the ability to control Block Storage service (Cinder) snapshots on NFS back ends. A new director parameter, CinderNfsSnapshotSupport, has a default value of True.
|
Story Points: | --- | |
| Clone Of: | 1633146 | |||
| : | 1698683 (view as bug list) | Environment: | ||
| Last Closed: | 2019-08-20 10:45:13 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1633146 | |||
| Bug Blocks: | 1698683 | |||
|
Description
Alan Bishop
2019-04-11 00:28:35 UTC
According to our records, this should be resolved by openstack-tripleo-heat-templates-9.3.1-0.20190513171738.el7ost. This build is available now. According to our records, this should be resolved by puppet-tripleo-9.4.1-0.20190508182403.89735a1.el7ost. This build is available now. Verified on:
openstack-tripleo-heat-templates-9.3.1-0.20190513171738.el7ost.noarch
1. At first I'll test the default value, should be snapshot enabled true.
(overcloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh
#!/bin/bash
openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock1.rdu2.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage/cinder-nfs.yaml \ -> Added Cinder nfs yaml
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/extra_templates.yaml \ -> here lay the cinder parameters, see below
--log-file overcloud_deployment_55.log
(overcloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml
parameter_defaults:
CinderEnableIscsiBackend: false
CinderEnableRbdBackend: false
CinderEnableNfsBackend: true
CinderNfsMountOptions: 'retry=1'
CinderNfsServers: 'x.y.w.z:/export/ins_cinder' -> w.x.y.z replace internal IPs, notice no mention of snapshot param so as to check default THT setting.
2. Once overcloud is deployed we see Cinder has an NFS backend:
(overcloud) [stack@undercloud-0 ~]$ cinder service-list
+------------------+-----------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-----------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | controller-1 | nova | enabled | up | 2019-08-19T12:44:05.000000 | - |
| cinder-scheduler | controller-0 | nova | enabled | up | 2019-08-19T12:44:03.000000 | - |
| cinder-scheduler | controller-1 | nova | enabled | up | 2019-08-19T12:44:03.000000 | - |
| cinder-scheduler | controller-2 | nova | enabled | up | 2019-08-19T12:44:03.000000 | - |
| cinder-volume | hostgroup@tripleo_nfs | nova | enabled | up | 2019-08-19T12:44:05.000000 | - |
+------------------+-----------------------+------+---------+-------+----------------------------+-----------------+
Which is great Cinder runs over NFS.
The resulting Cinder backend section as expected from the default includes snapshot support.
[root@controller-0 ~]# tail /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf
[tripleo_nfs]
backend_host=hostgroup
volume_backend_name=tripleo_nfs
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/shares-nfs.conf
nfs_mount_options=retry=1
nfs_snapshot_support=True --> Yay default true works!
nas_secure_file_operations=False
nas_secure_file_permissions=False
3. Create a cinder volume:
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --name vol_nfs
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-08-19T12:45:05.000000 |
| description | None |
| encrypted | False |
| id | 415f711a-7269-4032-bbbf-d3832a0c1246 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | vol_nfs |
| os-vol-host-attr:host | hostgroup@tripleo_nfs#tripleo_nfs |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 3cf2c871282447a1b617fd6e38a4f309 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2019-08-19T12:45:06.000000 |
| user_id | 1cdf4bf222c94be088f8b2ed68986abd |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Volume created on NFS backend
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| 415f711a-7269-4032-bbbf-d3832a0c1246 | available | vol_nfs | 1 | tripleo | false | |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
4. Create a snapshot of said volume:
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create 415f711a-7269-4032-bbbf-d3832a0c1246 --name Snap
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2019-08-19T12:46:18.096248 |
| description | None |
| id | 1aadf298-7e98-4cd3-8a66-0385c0c81657 |
| metadata | {} |
| name | Snap |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | 415f711a-7269-4032-bbbf-d3832a0c1246 |
+-------------+--------------------------------------+
A while later snap on an NFS volume is successfully created, so we know THT's default of enabled/true works!
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+-----------+------+------+
| 1aadf298-7e98-4cd3-8a66-0385c0c81657 | 415f711a-7269-4032-bbbf-d3832a0c1246 | available | Snap | 1 |
+--------------------------------------+--------------------------------------+-----------+------+------+
Now lets cleanup snap/vol and delete overcloud, this time wish snapshot disabled
Cinder snapshot-delete ..
cinder delete ..
(undercloud) [stack@undercloud-0 ~]$ openstack stack delete overcloud --wait --yes
..
Now we edit/add last line
(overcloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml
parameter_defaults:
CinderEnableIscsiBackend: false
CinderEnableRbdBackend: false
CinderEnableNfsBackend: true
CinderNfsMountOptions: 'retry=1'
CinderNfsServers: 'x.y.w.z:/export/ins_cinder'
CinderNfsSnapshotSupport: false -> add false to disable snapshot on NFS.
Redeploy overcloud wait, wait for it.. :)
We have an overcloud with Cinder using NFS as backend:
(overcloud) [stack@undercloud-0 ~]$ cinder service-list
+------------------+-----------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-----------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | controller-1 | nova | enabled | up | 2019-08-19T14:07:43.000000 | - |
| cinder-scheduler | controller-0 | nova | enabled | up | 2019-08-19T14:07:44.000000 | - |
| cinder-scheduler | controller-1 | nova | enabled | up | 2019-08-19T14:07:43.000000 | - |
| cinder-scheduler | controller-2 | nova | enabled | up | 2019-08-19T14:07:44.000000 | - |
| cinder-volume | hostgroup@tripleo_nfs | nova | enabled | up | 2019-08-19T14:07:49.000000 | - |
+------------------+-----------------------+------+---------+-------+----------------------------+-----------------+
The resulting Cinder backend with disabled snapshot support:
[root@controller-0 ~]# tail /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf
[tripleo_nfs]
backend_host=hostgroup
volume_backend_name=tripleo_nfs
volume_driver=cinder.volume.drivers.nfs.NfsDriver
nfs_shares_config=/etc/cinder/shares-nfs.conf
nfs_mount_options=retry=1
nfs_snapshot_support=False -> great as expected.
nas_secure_file_operations=False
nas_secure_file_permissions=False
Create a Cinder vol:
(overcloud) [stack@undercloud-0 ~]$ cinder create 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-08-19T15:19:09.000000 |
| description | None |
| encrypted | False |
| id | fe867488-bb38-4251-aada-b086f78a8a74 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | None |
| os-vol-host-attr:host | hostgroup@tripleo_nfs#tripleo_nfs |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | f2d2d1df82a445588e56da683691caa7 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2019-08-19T15:19:10.000000 |
| user_id | 604b395c1968478ab39b2fab7382557b |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Try to create a vol snapshot, it should fail:
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create fe867488-bb38-4251-aada-b086f78a8a74
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2019-08-19T15:20:01.401011 |
| description | None |
| id | 8d6024f9-b98d-425d-b416-0b12ecbaf0ef |
| metadata | {} |
| name | None |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | fe867488-bb38-4251-aada-b086f78a8a74 |
+-------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list
+--------------------------------------+--------------------------------------+--------+------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+--------+------+------+
| 8d6024f9-b98d-425d-b416-0b12ecbaf0ef | fe867488-bb38-4251-aada-b086f78a8a74 | error | - | 1 |
+--------------------------------------+--------------------------------------+--------+------+------+
c-vol expected error:
on during message handling: VolumeDriverException: Volume driver reported an error: NFS driver snapshot support is disabled in cinder.conf.
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_snapshot
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1072, in create_snapshot
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server snapshot.save()
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1064, in create_snapshot
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server model_update = self.driver.create_snapshot(snapshot)
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_snapshot
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 565, in create_snapshot
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server self._check_snapshot_support()
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 550, in _check_snapshot_support
2019-08-19 15:20:01.740 61 ERROR oslo_messaging.rpc.server VolumeDriverException: Volume driver reported an error: NFS driver snapshot support is disabled in cinder.conf.
2019-08-19 15:20:33.035 61 DEBUG oslo_concurrency.processutils [req-b9e83187-cc74-4798-83c0-0f8d59286ef8 - - - - -] Running cmd (subprocess): du -sb --apparent-size --exclude *snapshot* /var/lib/cinder/mnt/47266020eacec99097bdec49f2451d38 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372
Good to verify.
|