Description of problem: backup-create fails if I mount the volume on instance Version-Release number of selected component (if applicable): RHOS8 How reproducible: 100% Steps to Reproduce: 1. create volume. 2. attach it the instance. 3. mount it on the instance. 4. unmount it 5. detach it from the instance. 6. try to create backup. [root@rhos8-beta9 ~(keystone_admin)]# cinder create 1 --name pbandark_test_vol6 +---------------------------------------+--------------------------------------+ | Property | Value | +---------------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2016-04-15T20:40:57.000000 | | description | None | | encrypted | False | | id | 1dc0f384-5f81-4995-8ccd-71b2bc566997 | | metadata | {} | | migration_status | None | | multiattach | False | | name | pbandark_test_vol6 | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | af930979147d4729975c2e58bb891936 | | os-volume-replication:driver_data | None | | os-volume-replication:extended_status | None | | replication_status | disabled | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | user_id | 0bd72a79b869497f88619377c1300478 | | volume_type | None | +---------------------------------------+--------------------------------------+ [root@rhos8-beta9 ~(keystone_admin)]# nova volume-attach 0d385694-99b4-4476-bd07-8e15730a2232 1dc0f384-5f81-4995-8ccd-71b2bc566997 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 1dc0f384-5f81-4995-8ccd-71b2bc566997 | | serverId | 0d385694-99b4-4476-bd07-8e15730a2232 | | volumeId | 1dc0f384-5f81-4995-8ccd-71b2bc566997 | +----------+--------------------------------------+ - mounted/unmounted the volume from horizon console. [root@rhos8-beta9 ~(keystone_admin)]# nova volume-detach 0d385694-99b4-4476-bd07-8e15730a2232 1dc0f384-5f81-4995-8ccd-71b2bc566997 [root@rhos8-beta9 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+------------------+-----------------------------------------------------+------+-------------+----------+-------------+-------------+ | ID | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to | +--------------------------------------+-----------+------------------+-----------------------------------------------------+------+-------------+----------+-------------+-------------+ | 05fa3d9c-a83e-48ae-b83b-4fc04afea38c | error | - | | 20 | - | false | False | | | 1dc0f384-5f81-4995-8ccd-71b2bc566997 | available | - | pbandark_test_vol6 | 1 | - | false | False | | | be27fa48-4032-4ba9-8d7e-c6c7eea4eba4 | available | - | nfs1 | 1 | nfs | false | False | | | d774045c-c3aa-4e65-ab1f-0addb7ee85ae | available | - | restore_backup_6be0bc01-ea4c-447b-bb9f-315b596fd23f | 1 | - | false | False | | +--------------------------------------+-----------+------------------+-----------------------------------------------------+------+-------------+----------+-------------+-------------+ [root@rhos8-beta9 ~(keystone_admin)]# cinder backup-create 1dc0f384-5f81-4995-8ccd-71b2bc566997 +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 6d7ad466-8d01-4b7d-80c0-b5d3f80a8ed3 | | name | None | | volume_id | 1dc0f384-5f81-4995-8ccd-71b2bc566997 | +-----------+--------------------------------------+ 2016-04-15 16:57:11.280 6880 INFO cinder.backup.manager [req-cf67a0a9-38df-450b-b5dd-aecee90930b4 0bd72a79b869497f88619377c1300478 af930979147d4729975c2e58bb891936 - - -] Create backup started, backup: 6f54b609-db49-4fc0-adb7-1bd413b4f734 volume: 1dc0f384-5f81-4995-8ccd-71b2bc566997. [...] 2016-04-15 16:57:11.636 6880 DEBUG oslo_concurrency.processutils [req-cf67a0a9-38df-450b-b5dd-aecee90930b4 0bd72a79b869497f88619377c1300478 af930979147d4729975c2e58bb891936 - - -] Running cmd (subprocess): dd if=/var/lib/cinder/mnt/2bb9c378dd673a3dbaeaa00e037e1c7a/volume-1dc0f384-5f81-4995-8ccd-71b2bc566997 of=/dev/null count=1 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:230 2016-04-15 16:57:11.645 6880 DEBUG oslo_concurrency.processutils [req-cf67a0a9-38df-450b-b5dd-aecee90930b4 0bd72a79b869497f88619377c1300478 af930979147d4729975c2e58bb891936 - - -] CMD "dd if=/var/lib/cinder/mnt/2bb9c378dd673a3dbaeaa00e037e1c7a/volume-1dc0f384-5f81-4995-8ccd-71b2bc566997 of=/dev/null count=1" returned: 1 in 0.009s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:260 2016-04-15 16:57:11.647 6880 DEBUG oslo_concurrency.processutils [req-cf67a0a9-38df-450b-b5dd-aecee90930b4 0bd72a79b869497f88619377c1300478 af930979147d4729975c2e58bb891936 - - -] u'dd if=/var/lib/cinder/mnt/2bb9c378dd673a3dbaeaa00e037e1c7a/volume-1dc0f384-5f81-4995-8ccd-71b2bc566997 of=/dev/null count=1' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:308 2016-04-15 16:57:11.650 6880 DEBUG cinder.volume.driver [req-cf67a0a9-38df-450b-b5dd-aecee90930b4 0bd72a79b869497f88619377c1300478 af930979147d4729975c2e58bb891936 - - -] Device for volume 1dc0f384-5f81-4995-8ccd-71b2bc566997 is unavailable but did attach, detaching it. _attach_volume /usr/lib/python2.7/site-packages/cinder/volume/driver.py:976 2016-04-15 16:57:11.651 6880 DEBUG cinder.volume.driver [req-cf67a0a9-38df-450b-b5dd-aecee90930b4 0bd72a79b869497f88619377c1300478 af930979147d4729975c2e58bb891936 - - -] volume 1dc0f384-5f81-4995-8ccd-71b2bc566997: removing export _detach_volume /usr/lib/python2.7/site-packages/cinder/volume/driver.py:404 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher [req-cf67a0a9-38df-450b-b5dd-aecee90930b4 0bd72a79b869497f88619377c1300478 af930979147d4729975c2e58bb891936 - - -] Exception during message handling: The device in the path /var/lib/cinder/mnt/2bb9c378dd673a3dbaeaa00e037e1c7a/volume-1dc0f384-5f81-4995-8ccd-71b2bc566997 is unavailable: Unable to access the backend storage via the path /var/lib/cinder/mnt/2bb9c378dd673a3dbaeaa00e037e1c7a/volume-1dc0f384-5f81-4995-8ccd-71b2bc566997. 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher executor_callback)) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher executor_callback) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 399, in create_backup 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher self._update_backup_error(backup, context, six.text_type(err)) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 393, in create_backup 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher backup_service) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1094, in backup_volume 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher backup_service) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1119, in _backup_volume_temp_volume 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher self._backup_device(context, backup, backup_service, device_to_backup) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1172, in _backup_device 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher properties) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 968, in _attach_volume 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher attach_info = self._connect_device(conn) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1076, in _connect_device 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher {'path': host_device})) 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher DeviceUnavailable: The device in the path /var/lib/cinder/mnt/2bb9c378dd673a3dbaeaa00e037e1c7a/volume-1dc0f384-5f81-4995-8ccd-71b2bc566997 is unavailable: Unable to access the backend storage via the path /var/lib/cinder/mnt/2bb9c378dd673a3dbaeaa00e037e1c7a/volume-1dc0f384-5f81-4995-8ccd-71b2bc566997. 2016-04-15 16:57:11.804 6880 ERROR oslo_messaging.rpc.dispatcher Actual results: Expected results: Additional info:
This is the same issue as with https://bugzilla.redhat.com/show_bug.cgi?id=1360424 The problem is caused by an inconsistent behavior of nas_secure_file_operations working on auto mode. A workaround for this is setting disabling it in the specific driver configuration: [mydriversection] nas_secure_file_operations=false
No recent progress on this issue. Moving to RHOS 11 for triage.
Closing this as a duplicate of the current bug we are using to track the default setting fix. *** This bug has been marked as a duplicate of bug 1433404 ***