Bug 1049864

Summary: Creating a 99G volume on 100G cinder-volume succeeded, but deleting it failed (no space left)
Product: [Community] RDO Reporter: Meital Bourvine <mbourvin>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED NOTABUG QA Contact: Dafna Ron <dron>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: eharney, nlevinki, yeylon, zaitcev
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-18 18:21:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
cinder volume log none

Description Meital Bourvine 2014-01-08 11:47:11 UTC 
Description of problem:
I managed to create a 99G volume on 100G cinder-volume.
When I try to delete it (or basically do any other action - like `cinder list` or login to horizon, I get this error:

Jan  8 13:40:42 master-vds12 object-replicator: Exception dumping recon cache: #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1948, in dump_recon_cache#012    raise#012  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012    self.gen.throw(type, value, traceback)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1320, in lock_file#012    yield file_obj#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1941, in dump_recon_cache#012    tf.write(json.dumps(cache_entry) + '\n')#012  File "/usr/lib64/python2.6/tempfile.py", line 411, in __exit__#012    result = self.file.__exit__(exc, value, tb)#012IOError: [Errno 28] No space left on device

I didn't find any way to revert this situation. Now my system is not functional anymore.

Version-Release number of selected component (if applicable):
openstack-cinder-2014.1-0.2.b1.el6.noarch

How reproducible:
100%

Steps to Reproduce:
1. Create cinder-volume with size 100G
2. Create a volume with size 99G
3. Delete the volume

Actual results:
Deletion fails, system is not functional.

Expected results:
Deletion will succeed.
Or creation will not be allowed if there is not enough space left on device.

Additional info:
Comment 1 Meital Bourvine 2014-01-08 11:52:55 UTC 
Created attachment 847084 [details]
cinder volume log
Comment 2 Eric Harney 2014-01-08 14:53:58 UTC 
This could be due to disk space constraints on the system.  Can you post the output of "df", "vgs", "lvs", and "losetup -a"?

The delete process for Cinder LVM-backed volumes writes over the full amount of data during deletion, so it will write 99G of data in this case.

If the VG is backed by a loopback file it is possible that the file system filled up, causing the Swift error seen above.
Comment 3 Meital Bourvine 2014-01-08 14:59:49 UTC 
I re-provisioned my machine, so I can't provide this info.

What you wrote is probably the case, but I don't understand something - shouldn't it write the zeros on the existing volume? We already have 99G for this volume.
Comment 4 Eric Harney 2014-01-08 15:05:50 UTC 
Yes, but it is possible to have a 99G volume that is backed by a file on a file system smaller than 99G.  Packstack will do this by default I believe (with a 20G configuration).

An empty file is created which is mounted as a loopback device, and then the volume group is created on top of that.  But the VG can be larger than the underlying file system, which is why this can occur when lots of data is written to it.
Comment 6 Eric Harney 2015-03-18 18:21:39 UTC 
Seems to be explained by comment #4.