RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/
Bug 1038537 - Cannot create snapshot of a "in-use" cinder volume. Snapshot fails with "error" status.
Summary: Cannot create snapshot of a "in-use" cinder volume. Snapshot fails with "erro...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: RDO
Classification: Community
Component: openstack-nova
Version: unspecified
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: Havana
Assignee: Eric Harney
QA Contact: Ami Jeain
URL:
Whiteboard:
Depends On: 1038260
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-05 09:55 UTC by shilpa
Modified: 2015-02-17 15:59 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
virt rhos cinder integration
Last Closed: 2015-02-17 15:59:35 UTC
Embargoed:


Attachments (Terms of Use)

Description shilpa 2013-12-05 09:55:31 UTC
Description of problem:
While creating a snaphsot of a volume that is attached to an instance, the snapshot fails with "error" status. 


Version-Release number of selected component (if applicable):
RHOS 4.0 rdo-release-havana-7.noarch
RHS 2.1 U2 glusterfs-3.4.0.44.1u2rhs-1.el6rhs.x86_64

How reproducible:
Always


Steps to Reproduce:
 1. Created 2 6x2 Distributed-Replicate volumes for cinder and glance

2. Tagged the volumes with group virt
   (i.e) gluster volume set cinder-vol group virt
         gluster volume set glance-vol group virt

3. Set the storage.owner-uid and storage.owner-gid of glance-vol to 161
         gluster volume set glance-vol storage.owner-uid 161
         gluster volume set glance-vol storage.owner-gid 161

4. Set the storage.owner-uid and storage.owner-gid of cinder-vol to 165
         gluster volume set cinder-vol storage.owner-uid 165
         gluster volume set cinder-vol storage.owner-gid 165


Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: 8b20ce62-3606-4c52-b36e-567f97ebff7f
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.37.168:/rhs/brick2/c1
Brick2: 10.70.37.214:/rhs/brick2/c2
Brick3: 10.70.37.181:/rhs/brick2/c3
Brick4: 10.70.37.164:/rhs/brick2/c4
Brick5: 10.70.37.168:/rhs/brick2/c5
Brick6: 10.70.37.214:/rhs/brick2/c6
Brick7: 10.70.37.181:/rhs/brick2/c7
Brick8: 10.70.37.164:/rhs/brick2/c8
Brick9: 10.70.37.181:/rhs/brick2/c11
Brick10: 10.70.37.164:/rhs/brick2/c12
Brick11: 10.70.37.168:/rhs/brick2/c9
Brick12: 10.70.37.214:/rhs/brick2/c10
Options Reconfigured:
server.allow-insecure: on
storage.owner-uid: 165
storage.owner-gid: 165
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

5. On RHOS machine Configured cinder to use glusterfs volume

  a. 
      # openstack-config --set /etc/cinder/cinder.conf DEFAULT volume_driver cinder.volume.drivers.glusterfs.GlusterfsDriver
      # openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_shares_config /etc/cinder/shares.conf
      # openstack-config --set /etc/cinder/cinder.conf DEFAULT glusterfs_mount_point_base /var/lib/cinder/volumes
  
  b. # cat /etc/cinder/shares.conf
     10.70.x.x:/cinder-vol

  c. for i in api scheduler volume; do sudo service openstack-cinder-${i} restart; done

7. Fuse mount the RHS glance volume on /mnt/gluster/glance/images

8.Upload an image and boot a VM instance. 

9. Created a cinder volume called test-vol and attach it to the running VM. 

10. Tried to create snapshot of the attached volume using:

# cinder snapshot-create --force True --display-name snap test-vol


Actual results:

The snapshot shows "error" status:

# cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+--------------+------+
|                  ID                  |              Volume ID               |   Status  | Display Name | Size |
+--------------------------------------+--------------------------------------+-----------+--------------+------+

| 96c7ffb9-195f-4e64-bc64-86c199d7ba35 | 8f382c93-3b7f-43a3-8c22-c3c2105c53ca |   error   |     snap     |  10  |
+--------------------------------------+--------------------------------------+-----------+--------------+------+

Expected results:

The snapshot creation of a volume in-use should be successful with "--force True" option in the "cinder snapshot-create" command.


Additional info:

from cinder/volume.log, found errors pointing to /cinder/volume/drivers/glusterfs.py

2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 371, in create_snapshot
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error'})
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     self.gen.next()
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 362, in create_snapshot
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     model_update = self.driver.create_snapshot(snapshot_ref)
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 414, in create_snapshot
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     raise exception.GlusterfsException(msg)
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp GlusterfsException: Nova returned "error" status while creating snapshot.

Comment 1 shilpa 2013-12-05 10:03:28 UTC
Adding errors from nova/compute.log:


2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_da
ta
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     **args)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispa
tch
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 439, in inner
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     return catch_client_exception(exceptions, func, *args, **kwargs)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 420, in catch_cli
ent_exception
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     return func(*args, **kwargs)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2351, in volume_snapshot_crea
te
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     create_info)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1661, in volume_snapshot_
create
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     context, snapshot_id, 'error')
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1654, in volume_snapshot_
create
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     create_info['new_file'])
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1611, in _volume_snapshot
_create
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     domain.snapshotCreateXML(snapshot_xml, snap_flags)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     rv = execute(f,*args,**kwargs)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     rv = meth(*args,**kwargs)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp     if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
2013-12-05 15:16:55.201 26673 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error missing name from disk snapshot element

Comment 3 shilpa 2013-12-06 15:35:07 UTC
Tested this on a pure RHOS4.0 set up. Without glusterfs. Snapshots of a volume in-use *can* be created with the option "--force True" without any errors.

Comment 4 Vivek Agarwal 2013-12-10 17:32:07 UTC
glusterfs is not supporting snapshots as yet, snapshot will be supported in Denali

Comment 5 shilpa 2013-12-11 05:22:01 UTC
(In response to comment#4) This issue is about RHOS4.0 cinder snapshots. It is supported by Glusterfs.

Comment 6 crisbud@redhat.com 2013-12-11 11:03:49 UTC
cinder/volume.log indicating that there was some error from glusterfs for which it raised an exception. 

2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/glusterfs.py", line 414, in create_snapshot
2013-12-05 15:16:56.175 17101 TRACE cinder.openstack.common.rpc.amqp     raise exception.GlusterfsException(msg)

An exception raised because of some errors in "glusterfs". That needs to be triaged. 

I will triage to see what caused an exception and update the bug.

Comment 7 crisbud@redhat.com 2013-12-11 11:51:56 UTC
I looked at the NOVA logs at SHilpa's test setup. I could see that there are two different types of snapshot creation failures reported all over the logs. 

/var/log/nova/compute.log-20131208



2013-12-05 15:09:55.824 19134 ERROR nova.virt.libvirt.driver [req-eba43a97-798b-418f-b641-c16bd1d223d4 575f0769ede540bc8bb279f568336f1e ccb8126a56b9408c9a1895c71cdf3ea9] Error occurred during volume_snapshot_create, sending error status to Cinder.
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver Traceback (most recent call last):
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1654, in volume_snapshot_create
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver     create_info['new_file'])
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1563, in _volume_snapshot_create
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver     new_file_path = os.path.join(os.path.dirname(current_file),
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver   File "/usr/lib64/python2.6/posixpath.py", line 119, in dirname
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver     i = p.rfind('/') + 1
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver AttributeError: 'NoneType' object has no attribute 'rfind'
2013-12-05 15:09:55.824 19134 TRACE nova.virt.libvirt.driver
2013-12-05 15:09:55.839 19134 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.70.36.32
2013-12-05 15:09:55.948 19134 ERROR nova.openstack.common.rpc.amqp [req-eba43a97-798b-418f-b641-c16bd1d223d4 575f0769ede540bc8bb279f568336f1e ccb8126a56b9408c9a1895c71cdf3ea9] Exception during message handling
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     **args)
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 439, in inner
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     return catch_client_exception(exceptions, func, *args, **kwargs)
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/common.py", line 420, in catch_client_exception
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     return func(*args, **kwargs)
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2351, in volume_snapshot_create
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     create_info)
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1661, in volume_snapshot_create
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     context, snapshot_id, 'error')
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1654, in volume_snapshot_create
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     create_info['new_file'])
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1563, in _volume_snapshot_create
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     new_file_path = os.path.join(os.path.dirname(current_file),
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/posixpath.py", line 119, in dirname
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp     i = p.rfind('/') + 1
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp AttributeError: 'NoneType' object has no attribute 'rfind'
2013-12-05 15:09:55.948 19134 TRACE nova.openstack.common.rpc.amqp








=====================


2013-12-05 15:15:08.365 26673 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.70.36.32
2013-12-05 15:15:08.434 26673 WARNING nova.compute.manager [req-831e67f8-4f06-4f2a-8ec0-234a5566c617 None None] Found 2 in the database and 1 on the hypervisor.
2013-12-05 15:15:08.578 26673 INFO nova.compute.manager [req-831e67f8-4f06-4f2a-8ec0-234a5566c617 None None] Updating bandwidth usage cache
2013-12-05 15:16:09.762 26673 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2013-12-05 15:16:09.928 26673 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000000e: [Errno 2] No such file or directory: '/var/lib/nova/instances/7e5e0171-940e-4d86-97cc-01a061390f50/disk'
2013-12-05 15:16:09.933 26673 ERROR nova.virt.libvirt.driver [-] Getting disk size of instance-0000000f: [Errno 2] No such file or directory: '/var/lib/nova/instances/93aaef55-3fe0-4f03-93c7-931c662410f5/disk'
2013-12-05 15:16:10.347 26673 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 11312
2013-12-05 15:16:10.347 26673 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 9
2013-12-05 15:16:10.348 26673 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 22
2013-12-05 15:16:10.472 26673 INFO nova.compute.resource_tracker [-] Compute_service record updated for rhs-client8.lab.eng.blr.redhat.com:rhs-client8.lab.eng.blr.redhat.com
2013-12-05 15:16:55.080 26673 ERROR nova.virt.libvirt.driver [req-a33e054b-8b1b-4401-a836-6bbb4b6c2654 575f0769ede540bc8bb279f568336f1e ccb8126a56b9408c9a1895c71cdf3ea9] Unable to create quiesced VM snapshot, attempting again with quiescing disabled.
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver Traceback (most recent call last):
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1602, in _volume_snapshot_create
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver     snap_flags | QUIESCE)
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 187, in doit
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 147, in proxy_call
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver     rv = execute(f,*args,**kwargs)
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver     rv = meth(*args,**kwargs)
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver     if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver libvirtError: internal error missing name from disk snapshot element
2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver
2013-12-05 15:16:55.095 26673 ERROR nova.virt.libvirt.driver [req-a33e054b-8b1b-4401-a836-6bbb4b6c2654 575f0769ede540bc8bb279f568336f1e ccb8126a56b9408c9a1895c71cdf3ea9] Unable to create VM snapshot, failing volume_snapshot operation

Comment 8 crisbud@redhat.com 2013-12-11 11:56:03 UTC
It looks to me that #2 of above failure is related to uncleared old nova instances. Such instances are just uncleared but still a snapshot creation is being attempted. 

#1 is of interest of us and which falls in the same timezone as mentioned by tester. "2013-12-05 15:09:55.824". Also that is the only distinct snapshot creation failure may need attention from "Libvirt". 

It raised traceback in NOVA code seeing some attribute related errors. Goes to Libvirt in both cases. 


Thanks,
Chetan Risbud.

Comment 9 Gowrishankar Rajaiyan 2013-12-13 18:37:39 UTC
Per 12/13 triage, moving this bug to RHOS product for analysis based on comment #8.

Comment 11 Eric Harney 2013-12-17 00:05:46 UTC
Can you confirm the version of libvirt being used on the Nova platform when this error occurred?

2013-12-05 15:16:55.080 26673 TRACE nova.virt.libvirt.driver libvirtError: internal error missing name from disk snapshot element

Comment 12 Eric Harney 2013-12-17 00:08:17 UTC
Also, if you can reproduce that failure, Nova debug level output would be very helpful as well.

Comment 13 shilpa 2013-12-17 10:31:16 UTC
Libvirt version beng used: libvirt-0.10.2-29.el6.1.x86_64

Nova compute logs attached. Reproduced the failure on volume vol7. Snap name is vol7-snap1 (6ab5a9e5-4641-4e89-bf51-163a0abe8c97) created at 16:00 on 2013-12-17 in the logs. 

[1] 
# cinder snapshot-list

| 6ab5a9e5-4641-4e89-bf51-163a0abe8c97 | fe61fb62-4043-468a-b55f-d3c6a2b3d74a |     error      |  vol7-snap1  |  2   |


[2]
# cinder snapshot-show 6ab5a9e5-4641-4e89-bf51-163a0abe8c97
+--------------------------------------------+--------------------------------------+
|                  Property                  |                Value                 |
+--------------------------------------------+--------------------------------------+
|                 created_at                 |      2013-12-17T10:30:23.000000      |
|            display_description             |                 None                 |
|                display_name                |              vol7-snap1              |
|                     id                     | 6ab5a9e5-4641-4e89-bf51-163a0abe8c97 |
|                  metadata                  |                  {}                  |
|  os-extended-snapshot-attributes:progress  |                 90%                  |
| os-extended-snapshot-attributes:project_id |   ccb8126a56b9408c9a1895c71cdf3ea9   |
|                    size                    |                  2                   |
|                   status                   |                error                 |
|                 volume_id                  | fe61fb62-4043-468a-b55f-d3c6a2b3d74a |
+--------------------------------------------+--------------------------------------+

[3]
# cinder list

| fe61fb62-4043-468a-b55f-d3c6a2b3d74a |   in-use  |     vol7     |  2   |     None    |  false   | cfa0a944-4f8e-47b5-be0e-9831087a7bf1 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+

Comment 14 shilpa 2013-12-17 11:46:58 UTC
Could not attach logs as they were too big. Uploaded compute logs in http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1038537/

Comment 15 Eric Harney 2014-01-09 20:02:19 UTC
It looks like three separate things were happening here:

1) libvirt snapshotCreateXML failed with "libvirtError: internal error missing name from disk snapshot element"

I'm not sure what environment this was running on exactly -- the initial report indicates "RHOS 4.0 rdo-release-havana-7.noarch" ?  What platform and libvirt version was this with?

None of the logs reveal what was passed in to cause this particular error.


2) snapshot creation failed due to qemu not supporting it

2013-12-17 12:52:25.615 22645 TRACE nova.virt.libvirt.driver   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1636, in snapshotCreateXML
2013-12-17 12:52:25.615 22645 TRACE nova.virt.libvirt.driver     if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
2013-12-17 12:52:25.615 22645 TRACE nova.virt.libvirt.driver libvirtError: unsupported configuration: reuse is not supported with this QEMU binary

This occurs if you don't have the correct qemu package installed.  Which distribution and qemu packages were being used here?


3) Errors stemming from incorrect share configuration for the Cinder Gluster driver.  The Cinder side of this is being fixed in bug 1038260.  A patch to make Nova behave more cleanly when passed malformed information is also appropriate.

Comment 16 shilpa 2014-01-13 06:41:30 UTC
Here are the versions of qemu and libvirt on RHEl 6.5

1. libvirt-0.10.2-29.el6_5.2.x86_64
2. qemu-img-rhev-0.12.1.2-2.415.el6_5.3.x86_64
   qemu-kvm-rhev-0.12.1.2-2.415.el6_5.3.x86_64

3. I ran this test after correcting the cinder share configuration as per 1038260. 

Do you require a new set of logs?

Comment 17 Eric Harney 2014-01-13 20:21:54 UTC
(In reply to shilpa from comment #16)
> Here are the versions of qemu and libvirt on RHEl 6.5
> 
> 1. libvirt-0.10.2-29.el6_5.2.x86_64
> 2. qemu-img-rhev-0.12.1.2-2.415.el6_5.3.x86_64
>    qemu-kvm-rhev-0.12.1.2-2.415.el6_5.3.x86_64
> 

I don't understand why you mentioned rdo-release-havana-7.noarch.  Was this on RHOS or RDO?

Are the Cinder packages el6 or el6ost?

> 3. I ran this test after correcting the cinder share configuration as per
> 1038260. 
> 
> Do you require a new set of logs?

Possibly but I'd like to understand enough about the environment to reproduce this myself.  I'm not sure how to hit these two libvirt errors you showed.

Comment 18 shilpa 2014-01-14 14:23:12 UTC
In reply to comment#17

My bad it was RDO.. Tested on a RHOS4.0 set up and it works fine. Cannot reproduce the issue.

Comment 19 Kashyap Chamarthy 2015-02-17 15:59:35 UTC
Closing per comment #18

Also, Havana is End-Of-Life as of Sep 22, 2014

  https://wiki.openstack.org/wiki/Releases


Note You need to log in before you can comment on or make changes to this bug.