Description of problem: I managed to create a 99G volume on 100G cinder-volume. When I try to delete it (or basically do any other action - like `cinder list` or login to horizon, I get this error: Jan 8 13:40:42 master-vds12 object-replicator: Exception dumping recon cache: #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1948, in dump_recon_cache#012 raise#012 File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012 self.gen.throw(type, value, traceback)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1320, in lock_file#012 yield file_obj#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1941, in dump_recon_cache#012 tf.write(json.dumps(cache_entry) + '\n')#012 File "/usr/lib64/python2.6/tempfile.py", line 411, in __exit__#012 result = self.file.__exit__(exc, value, tb)#012IOError: [Errno 28] No space left on device I didn't find any way to revert this situation. Now my system is not functional anymore. Version-Release number of selected component (if applicable): openstack-cinder-2014.1-0.2.b1.el6.noarch How reproducible: 100% Steps to Reproduce: 1. Create cinder-volume with size 100G 2. Create a volume with size 99G 3. Delete the volume Actual results: Deletion fails, system is not functional. Expected results: Deletion will succeed. Or creation will not be allowed if there is not enough space left on device. Additional info:
Created attachment 847084 [details] cinder volume log
This could be due to disk space constraints on the system. Can you post the output of "df", "vgs", "lvs", and "losetup -a"? The delete process for Cinder LVM-backed volumes writes over the full amount of data during deletion, so it will write 99G of data in this case. If the VG is backed by a loopback file it is possible that the file system filled up, causing the Swift error seen above.
I re-provisioned my machine, so I can't provide this info. What you wrote is probably the case, but I don't understand something - shouldn't it write the zeros on the existing volume? We already have 99G for this volume.
Yes, but it is possible to have a 99G volume that is backed by a file on a file system smaller than 99G. Packstack will do this by default I believe (with a 20G configuration). An empty file is created which is mounted as a loopback device, and then the volume group is created on top of that. But the VG can be larger than the underlying file system, which is why this can occur when lots of data is written to it.
Seems to be explained by comment #4.