Created attachment 957372[details]
logs from engine and host
Description of problem:
Snapshot creation fails sometimes with a Qemu image error on vdsm. Happened to me several times, all of them while trying to create a live snapshot with memory saved for a VM that was cloned out of another VM which also had memory images attached to it.
Version-Release number of selected component (if applicable):
rhev 3.5 vt10
vdsm-4.16.7.4-1.el6ev.x86_64
iscsi-initiator-utils-6.2.0.873-13.el6.x86_64
How reproducible:
Not always
Steps to Reproduce:
Happened to me several times when I tried to create a live snapshot with memory saved
1. Create a VM with a disk attaced (block domain)
2. Start the VM, take live snapshot with memory saved
3. Stop the VM, clone it
4. Start the cloned VM, create a live snapshot with memory saved
Actual results:
Operation fails on vdsm with the following error message:
67e6483f-a7f6-4d91-b1b0-a2b471d25d52::ERROR::2014-11-14 00:58:23,974::volume::264::Storage.Volume::(clone) cannot clone image 3785a814-7dea-4fdf-b66f-6eb67a724d78 volume 773001e
b-1d3d-47b9-b364-413eaa4bb24f to /rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724d78/9c65c93b
-c456-4cc8-8a80-0fa5137074eb
Traceback (most recent call last):
File "/usr/share/vdsm/storage/volume.py", line 260, in clone
backingFormat=fmt2str(self.getFormat()))
File "/usr/lib/python2.6/site-packages/vdsm/qemuimg.py", line 137, in create
raise QImgError(rc, out, err)
QImgError: ecode=1, stdout=["Formatting '/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724d78/
9c65c93b-c456-4cc8-8a80-0fa5137074eb', fmt=qcow2 size=1073741824 backing_file='../3785a814-7dea-4fdf-b66f-6eb67a724d78/773001eb-1d3d-47b9-b364-413eaa4bb24f' backing_fmt='raw' en
cryption=off cluster_size=65536 "], stderr=['/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724
d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb: error while creating qcow2: Permission denied'], message=None
67e6483f-a7f6-4d91-b1b0-a2b471d25d52::DEBUG::2014-11-14 00:58:29,327::lvm::454::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mute
x
67e6483f-a7f6-4d91-b1b0-a2b471d25d52::ERROR::2014-11-14 00:58:29,328::volume::476::Storage.Volume::(create) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/volume.py", line 435, in create
srcVolUUID, volPath)
File "/usr/share/vdsm/storage/blockVolume.py", line 151, in _create
volParent.clone(volPath, volFormat)
File "/usr/share/vdsm/storage/volume.py", line 269, in clone
raise se.CannotCloneVolume(self.volumePath, dstPath, str(e))
CannotCloneVolume: Cannot clone volume: u'src=/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a72
4d78/773001eb-1d3d-47b9-b364-413eaa4bb24f, dst=/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a7
24d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb: ecode=1, stdout=["Formatting \'/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/378
5a814-7dea-4fdf-b66f-6eb67a724d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb\', fmt=qcow2 size=1073741824 backing_file=\'../3785a814-7dea-4fdf-b66f-6eb67a724d78/773001eb-1d3d-47b9-b36
4-413eaa4bb24f\' backing_fmt=\'raw\' encryption=off cluster_size=65536 "], stderr=[\'/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb: error while creating qcow2: Permission denied\'], message=None'
Reproduced ~6 times out of 10 attampts.
Also happened 1 time while creating live snapshot with memory for a newly created VM (not cloned).
Expected results:
Live snapshot creation should succeed
Additional info:
logs from engine and host
Created attachment 957372 [details] logs from engine and host Description of problem: Snapshot creation fails sometimes with a Qemu image error on vdsm. Happened to me several times, all of them while trying to create a live snapshot with memory saved for a VM that was cloned out of another VM which also had memory images attached to it. Version-Release number of selected component (if applicable): rhev 3.5 vt10 vdsm-4.16.7.4-1.el6ev.x86_64 iscsi-initiator-utils-6.2.0.873-13.el6.x86_64 How reproducible: Not always Steps to Reproduce: Happened to me several times when I tried to create a live snapshot with memory saved 1. Create a VM with a disk attaced (block domain) 2. Start the VM, take live snapshot with memory saved 3. Stop the VM, clone it 4. Start the cloned VM, create a live snapshot with memory saved Actual results: Operation fails on vdsm with the following error message: 67e6483f-a7f6-4d91-b1b0-a2b471d25d52::ERROR::2014-11-14 00:58:23,974::volume::264::Storage.Volume::(clone) cannot clone image 3785a814-7dea-4fdf-b66f-6eb67a724d78 volume 773001e b-1d3d-47b9-b364-413eaa4bb24f to /rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724d78/9c65c93b -c456-4cc8-8a80-0fa5137074eb Traceback (most recent call last): File "/usr/share/vdsm/storage/volume.py", line 260, in clone backingFormat=fmt2str(self.getFormat())) File "/usr/lib/python2.6/site-packages/vdsm/qemuimg.py", line 137, in create raise QImgError(rc, out, err) QImgError: ecode=1, stdout=["Formatting '/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724d78/ 9c65c93b-c456-4cc8-8a80-0fa5137074eb', fmt=qcow2 size=1073741824 backing_file='../3785a814-7dea-4fdf-b66f-6eb67a724d78/773001eb-1d3d-47b9-b364-413eaa4bb24f' backing_fmt='raw' en cryption=off cluster_size=65536 "], stderr=['/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724 d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb: error while creating qcow2: Permission denied'], message=None 67e6483f-a7f6-4d91-b1b0-a2b471d25d52::DEBUG::2014-11-14 00:58:29,327::lvm::454::Storage.OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mute x 67e6483f-a7f6-4d91-b1b0-a2b471d25d52::ERROR::2014-11-14 00:58:29,328::volume::476::Storage.Volume::(create) Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/volume.py", line 435, in create srcVolUUID, volPath) File "/usr/share/vdsm/storage/blockVolume.py", line 151, in _create volParent.clone(volPath, volFormat) File "/usr/share/vdsm/storage/volume.py", line 269, in clone raise se.CannotCloneVolume(self.volumePath, dstPath, str(e)) CannotCloneVolume: Cannot clone volume: u'src=/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a72 4d78/773001eb-1d3d-47b9-b364-413eaa4bb24f, dst=/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a7 24d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb: ecode=1, stdout=["Formatting \'/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/378 5a814-7dea-4fdf-b66f-6eb67a724d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb\', fmt=qcow2 size=1073741824 backing_file=\'../3785a814-7dea-4fdf-b66f-6eb67a724d78/773001eb-1d3d-47b9-b36 4-413eaa4bb24f\' backing_fmt=\'raw\' encryption=off cluster_size=65536 "], stderr=[\'/rhev/data-center/00000002-0002-0002-0002-0000000001d3/e58ba9a4-9598-42b1-ad22-7830aa86144b/images/3785a814-7dea-4fdf-b66f-6eb67a724d78/9c65c93b-c456-4cc8-8a80-0fa5137074eb: error while creating qcow2: Permission denied\'], message=None' Reproduced ~6 times out of 10 attampts. Also happened 1 time while creating live snapshot with memory for a newly created VM (not cloned). Expected results: Live snapshot creation should succeed Additional info: logs from engine and host