Bug 1331945

Summary:	NFSDriver: unable to backup volumes which are attached to the instance
Product:	Red Hat OpenStack	Reporter:	Pratik Pravin Bandarkar <pbandark>
Component:	openstack-cinder	Assignee:	Eric Harney <eharney>
Status:	CLOSED WONTFIX	QA Contact:	nlevinki <nlevinki>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	8.0 (Liberty)	CC:	egafford, eharney, sgotliv, srevivo
Target Milestone:	---
Target Release:	11.0 (Ocata)
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-09-15 15:14:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Pratik Pravin Bandarkar 2016-04-30 11:37:14 UTC

Description of problem:
unable to backup volumes which are attached to the instance.


Version-Release number of selected component (if applicable):
RHOS8

How reproducible:
100%

Steps to Reproduce:
1. configure nfs backend for cinder backup service.
2. create volume.
3. attach to the instance.
4. try to take backup of volume. It will fail. 
<snip>
[root@dhcp200-223 ~(keystone_bootcamp)]# nova volume-attach ade934be-42b7-4412-be06-42d3344ee8e1 74b10200-2b7f-424a-9c9e-d3aa8b100bfd 
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 74b10200-2b7f-424a-9c9e-d3aa8b100bfd |
| serverId | ade934be-42b7-4412-be06-42d3344ee8e1 |
| volumeId | 74b10200-2b7f-424a-9c9e-d3aa8b100bfd |
+----------+--------------------------------------+

[root@dhcp200-223 ~(keystone_bootcamp)]# cinder backup-create 74b10200-2b7f-424a-9c9e-d3aa8b100bfd --name test3_vol_backup --force
+-----------+--------------------------------------+
|  Property |                Value                 |
+-----------+--------------------------------------+
|     id    | 4bf00dc9-24a4-4079-a6c7-26776b9af089 |
|    name   |           test3_vol_backup           |
| volume_id | 74b10200-2b7f-424a-9c9e-d3aa8b100bfd |
+-----------+--------------------------------------+

[root@dhcp200-223 ~(keystone_bootcamp)]# cinder backup-show 4bf00dc9-24a4-4079-a6c7-26776b9af089
+-----------------------+------------------------------------------------------------+
|        Property       |                           Value                            |
+-----------------------+------------------------------------------------------------+
|   availability_zone   |                            None                            |
|       container       |                            None                            |
|       created_at      |                 2016-04-30T19:07:25.000000                 |
|      description      |                            None                            |
|      fail_reason      | 'NfsDriver' object has no attribute 'create_cloned_volume' |
| has_dependent_backups |                           False                            |
|           id          |            4bf00dc9-24a4-4079-a6c7-26776b9af089            |
|     is_incremental    |                           False                            |
|          name         |                      test3_vol_backup                      |
|      object_count     |                             0                              |
|          size         |                             1                              |
|         status        |                           error                            |
|       volume_id       |            74b10200-2b7f-424a-9c9e-d3aa8b100bfd            |
+-----------------------+------------------------------------------------------------+


2016-04-30 15:07:25.826 15419 INFO cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Create backup started, backup: 4bf00dc9
-24a4-4079-a6c7-26776b9af089 volume: 74b10200-2b7f-424a-9c9e-d3aa8b100bfd.

2016-04-30 15:07:25.834 15419 DEBUG cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Checking hostname 'dhcp200-223@product
ion' for backend info. _get_volume_backend /usr/lib/python2.7/site-packages/cinder/backup/manager.py:116
2016-04-30 15:07:25.835 15419 DEBUG cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Got backend 'production'. _get_volume_
backend /usr/lib/python2.7/site-packages/cinder/backup/manager.py:120
2016-04-30 15:07:25.855 15419 DEBUG cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Driver requested for volume_backend 'N
one'. _get_driver /usr/lib/python2.7/site-packages/cinder/backup/manager.py:146
2016-04-30 15:07:25.855 15419 DEBUG cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Fetching default backend. _get_driver 
/usr/lib/python2.7/site-packages/cinder/backup/manager.py:148
2016-04-30 15:07:25.856 15419 INFO cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Backend not found in hostname (None) so
 using default.
2016-04-30 15:07:25.856 15419 DEBUG cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Manager requested for volume_backend '
lvm'. _get_manager /usr/lib/python2.7/site-packages/cinder/backup/manager.py:134

2016-04-30 15:07:25.857 15419 DEBUG oslo_concurrency.processutils [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Running cmd (subprocess): moun
t execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:230
2016-04-30 15:07:25.870 15419 DEBUG oslo_concurrency.processutils [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] CMD "mount" returned: 0 in 0.0
13s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:260
2016-04-30 15:07:25.871 15419 INFO os_brick.remotefs.remotefs [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Already mounted: /var/lib/cinder/b
ackup_mount/86644d4ea62953499aa05c14c20d6b1e
2016-04-30 15:07:25.871 15419 DEBUG cinder.backup.drivers.nfs [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Using NFS backup repository: /var/lib/cinder/backup_mount/86644d4ea62953499aa05c14c20d6b1e __init__ /usr/lib/python2.7/site-packages/cinder/backup/drivers/nfs.py:58
2016-04-30 15:07:25.873 15419 DEBUG cinder.backup.drivers.posix [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Using backup repository: /var/lib/cinder/backup_mount/86644d4ea62953499aa05c14c20d6b1e __init__ /usr/lib/python2.7/site-packages/cinder/backup/drivers/posix.py:84
2016-04-30 15:07:25.874 15419 DEBUG cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Driver requested for volume_backend 'production'. _get_driver /usr/lib/python2.7/site-packages/cinder/backup/manager.py:146
2016-04-30 15:07:25.874 15419 DEBUG cinder.backup.manager [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Manager requested for volume_backend 'production'. _get_manager /usr/lib/python2.7/site-packages/cinder/backup/manager.py:134
2016-04-30 15:07:25.906 15419 DEBUG oslo_db.sqlalchemy.engines [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:256
2016-04-30 15:07:25.935 15419 DEBUG cinder.volume.driver [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Creating a new backup for volume volume-74b10200-2b7f-424a-9c9e-d3aa8b100bfd. _backup_volume_temp_volume /usr/lib/python2.7/site-packages/cinder/volume/driver.py:1103
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher [req-213d4460-a93d-480c-960e-a1085f10df5f 9fbade688f294569b040a223495c7825 7274fd8b20254b80a08f55f31eea3f99 - - -] Exception during message handling: 'NfsDriver' object has no attribute 'create_cloned_volume'
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 399, in create_backup
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     self._update_backup_error(backup, context, six.text_type(err))
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 393, in create_backup
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     backup_service)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1094, in backup_volume
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     backup_service)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1114, in _backup_volume_temp_volume
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     context, volume)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1264, in _create_temp_cloned_volume
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     self.db.volume_destroy(context, temp_vol_ref['id'])
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1261, in _create_temp_cloned_volume
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher     self.create_cloned_volume(temp_vol_ref, volume)
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher AttributeError: 'NfsDriver' object has no attribute 'create_cloned_volume'
2016-04-30 15:07:26.111 15419 ERROR oslo_messaging.rpc.dispatcher 

</snip>

- If I detach the volume from instance then I was able to create backup.

Actual results:
unable to backup volumes which are attached to the instance

Expected results:
backup operation should be successful even if volume is attached to the instance.

Additional info:

Comment 2 Sergey Gotliv 2016-05-02 22:35:53 UTC

I suspect this functionality depends on snapshotting which is also missing in the NFSDriver. This upstream patch [1] should fix both NFSDriver snapshotting and disruptive backups issues.

[1] https://review.openstack.org/#/c/147186/

Comment 4 Elise Gafford 2016-09-15 15:14:32 UTC

In discussion with Cinder team, it is believed that this is unlikely to be targeted for 11 upstream. Please reopen this bug if it becomes a customer priority; otherwise it is not a current priority.

Thanks,
Elise