Bug 987305 - cinder: creating a backup for a volume in external nfs storage will leave the volume and backup in creating state
cinder: creating a backup for a volume in external nfs storage will leave the...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
unspecified
x86_64 Linux
high Severity urgent
: beta
: 4.0
Assigned To: Flavio Percoco
Yogev Rabl
storage
:
Depends On: 966124
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-23 04:16 EDT by Dafna Ron
Modified: 2016-04-26 22:17 EDT (History)
6 users (show)

See Also:
Fixed In Version: openstack-cinder-2013.2-0.9.b3.el6ost
Doc Type: Bug Fix
Doc Text:
If an attempt was made to create backup volumes to an external NFS storage that was not available, the creation process would not terminate. This resulted in Block Storage backups getting stuck in a "creating" state. This has been fixed by setting Cinder to detect these cases and create an error. As a result, backups for Block Storage volumes on NFS storage fail, rather than hanging in a "creating" state. Note that backups are not supported for NFS-backed Cinder volumes, so they will never succeed.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 19:14:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (19.00 KB, application/x-gzip)
2013-07-23 04:16 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1200040 None None None Never

  None (edit)
Description Dafna Ron 2013-07-23 04:16:57 EDT
Created attachment 777221 [details]
logs

Description of problem:

i tried creating a backup to volumes in external nfs storage. 
the volumes get stuck in creating forever, we cannot delete the backup or the volumes because of invalid state. 

Version-Release number of selected component (if applicable):

openstack-cinder-2013.1.2-3.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. install AIO and change the local tgt storage to external nfs
2. create a volume: cinder create 10 --display-name bla
3. create a backup to the volume :  cinder backup-create <vol_id> --display-name <name>  --display-description <description>

Actual results:

the volume is stuck in backing-up state forever and the backup is stuck in creating state forever. 
we cannot delete or stop the process. 


**I created 2 backups and left it for a whole night but they are still creating and the volumes are stuck - adding list of all backups**

Expected results:

if we do not support this for nfs we need to block this. 
if we do support this for nfs we need to complete the backup create

Additional info:logs attached

[root@opens-vdsb ~(keystone_admin)]# cinder list
+--------------------------------------+------------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status   | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+------------+--------------+------+-------------+----------+-------------+
| 3a888ac1-395a-4082-973e-139be9de97d9 | backing-up |     bla1     |  10  |     bla     |  false   |             |
| 58adbbd4-628b-4137-a609-abca2cfffd38 | backing-up |    dafna     |  10  |     None    |  false   |             |
| 5fb0c74a-9540-4793-b386-d45ab5e4765a | available  |    blaaa     |  11  |    dafna    |  false   |             |
| b4b10bd5-fa66-41a5-a206-91c72dfd258b | available  |     bla      |  10  |     None    |  false   |             |
| fb596f0d-30ff-4f74-b68e-41d14b791d9b | backing-up |     bla      |  10  |    dafna    |  false   |             |
+--------------------------------------+------------+--------------+------+-------------+----------+-------------+
[root@opens-vdsb ~(keystone_admin)]# cinder delete fb596f0d-30ff-4f74-b68e-41d14b791d9b
ERROR: Invalid volume: Volume status must be available or error
[root@opens-vdsb ~(keystone_admin)]# cinder backup-list 
+--------------------------------------+--------------------------------------+----------+-------------+------+--------------+-----------+
|                  ID                  |              Volume ID               |  Status  |     Name    | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+----------+-------------+------+--------------+-----------+
| 38a27122-4019-4f29-9ebd-701b35ef45ac | 3a888ac1-395a-4082-973e-139be9de97d9 | creating | backup_bla1 |  10  |     None     |    None   |
| 7c4acfd5-d91a-4aad-84db-6fedbf968dbb | fb596f0d-30ff-4f74-b68e-41d14b791d9b | creating |     None    |  10  |     None     |    None   |
| bc8b6b86-2b61-48da-b18d-b47caf7f806f | 58adbbd4-628b-4137-a609-abca2cfffd38 | creating |    BACKUP   |  10  |     None     |    None   |
+--------------------------------------+--------------------------------------+----------+-------------+------+--------------+-----------+
[root@opens-vdsb ~(keystone_admin)]# cinder backup-delete bc8b6b86-2b61-48da-b18d-b47caf7f806f
ERROR: Invalid backup: Backup status must be available or error
Comment 1 Eric Harney 2013-07-23 07:29:11 EDT
Maybe related to bug 966124.
Comment 2 Flavio Percoco 2013-08-09 11:39:59 EDT
Hi,

This was fixed by this patch here https://review.openstack.org/#/c/36612/

I tested it and it works as expected now.
Comment 3 Flavio Percoco 2013-08-09 11:46:59 EDT
Actually, that's not the right patch, however, I'm still unable to replicate this bug.

[fedora@cinder-987305 devstack]$ grep volume_driver /etc/cinder/cinder.conf 
volume_driver=cinder.volume.drivers.nfs.NfsDriver


[fedora@cinder-987305 devstack]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 90426225-6b66-4802-bdf8-12e24fb7e3fd | available |     None     |  1   |     None    |  False   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
[fedora@cinder-987305 devstack]$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
|                  ID                  |              Volume ID               | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
| c96b77c6-c9f5-45e0-bcbf-3739444badf3 | 90426225-6b66-4802-bdf8-12e24fb7e3fd | error  | None |  1   |     None     |    None   |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
[fedora@cinder-987305 devstack]$
Comment 5 Dafna Ron 2013-08-19 07:09:09 EDT
(In reply to Flavio Percoco from comment #3)
> Actually, that's not the right patch, however, I'm still unable to replicate
> this bug.
> 
> [fedora@cinder-987305 devstack]$ grep volume_driver /etc/cinder/cinder.conf 
> volume_driver=cinder.volume.drivers.nfs.NfsDriver
> 
> 
> [fedora@cinder-987305 devstack]$ cinder list
> +--------------------------------------+-----------+--------------+------+---
> ----------+----------+-------------+
> |                  ID                  |   Status  | Display Name | Size |
> Volume Type | Bootable | Attached to |
> +--------------------------------------+-----------+--------------+------+---
> ----------+----------+-------------+
> | 90426225-6b66-4802-bdf8-12e24fb7e3fd | available |     None     |  1   |  
> None    |  False   |             |
> +--------------------------------------+-----------+--------------+------+---
> ----------+----------+-------------+
> [fedora@cinder-987305 devstack]$ cinder backup-list
> +--------------------------------------+-------------------------------------
> -+--------+------+------+--------------+-----------+
> |                  ID                  |              Volume ID             
> | Status | Name | Size | Object Count | Container |
> +--------------------------------------+-------------------------------------
> -+--------+------+------+--------------+-----------+
> | c96b77c6-c9f5-45e0-bcbf-3739444badf3 |
> 90426225-6b66-4802-bdf8-12e24fb7e3fd | error  | None |  1   |     None     |
> None   |
> +--------------------------------------+-------------------------------------
> -+--------+------+------+--------------+-----------+
> [fedora@cinder-987305 devstack]$


I reproduce 100%. 
perhaps its the nfs version on your remote server? 
can you confirm the configurations on /etc/cinder/shares.conf and on target server?
Comment 6 Scott Lewis 2013-11-19 11:54:19 EST
Auto adding >= MODIFIED bugs to beta
Comment 8 Yogev Rabl 2013-12-03 10:09:39 EST
verified on: 
python-cinderclient-1.0.6-2.el6ost.noarch
openstack-cinder-2013.2-2.el6ost.noarch
python-cinder-2013.2-2.el6ost.noarch
Comment 11 errata-xmlrpc 2013-12-19 19:14:58 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.