Description:
On rhevm server with iscsi storage, "HSMGetAllTasksStatusesVDS failed: Cannot create Logical Volume" while live storage migration for vm
Versions:
*vdsm-4.19.18-1.el7ev.x86_64*
libvirt-3.2.0-9.el7.x86_64
qemu-kvm-rhev-2.9.0-10.el7.x86_64
How reproducible:
100%
Steps:
1. Prepare rhevm env
1 host:hp
2 iscsi storages:IS1,IS2
1 vm:h
2.start the vm
3.select vm h, click on 'disk' tab, select the disk h_Disk1 and click on 'Move' to do live storage migration:
Operation failed with following errors:
(1)On rhevm server:
Jun 14, 2017 3:06:35 AM VDSM hp command HSMGetAllTasksStatusesVDS failed: Cannot create Logical Volume
Jun 14, 2017 3:06:42 AM User admin@internal-authz have failed to move disk h_Disk1 to domain IS1.
...
Jun 14, 2017 3:06:49 AM VM h has been paused.
...
(2)vdsm.log on host:
2017-06-14 03:06:35,484-0400 ERROR (tasks/2) [storage.Volume] Failed to create volume /rhev/data-center/ad36c80c-ff51-464b-8927-887c70d4880e/49858257-ecc7-4ce7-86b4-6ed8410a9e17/images/ede0bc84-9c25-47c1-b884-3cb0e32b0f8a/5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5') (volume:1070)
2017-06-14 03:06:35,484-0400 ERROR (tasks/2) [storage.Volume] Unexpected error (volume:1107)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/volume.py", line 1067, in create
initialSize=initialSize)
File "/usr/share/vdsm/storage/blockVolume.py", line 528, in _create
initialTags=(sc.TAG_VOL_UNINIT,))
File "/usr/share/vdsm/storage/lvm.py", line 1158, in createLV
raise se.CannotCreateLogicalVolume(vgName, lvName)
CannotCreateLogicalVolume: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5')
2017-06-14 03:06:35,485-0400 ERROR (tasks/2) [storage.Image] Unexpected error (image:467)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/image.py", line 450, in _createTargetImage
srcVolUUID=volParams['parent'])
File "/usr/share/vdsm/storage/sd.py", line 758, in createVolume
initialSize=initialSize)
File "/usr/share/vdsm/storage/volume.py", line 1067, in create
initialSize=initialSize)
File "/usr/share/vdsm/storage/blockVolume.py", line 528, in _create
initialTags=(sc.TAG_VOL_UNINIT,))
File "/usr/share/vdsm/storage/lvm.py", line 1158, in createLV
raise se.CannotCreateLogicalVolume(vgName, lvName)
CannotCreateLogicalVolume: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5')
2017-06-14 03:06:35,486-0400 ERROR (tasks/2) [storage.TaskManager.Task] (Task='0600a365-fa4f-4324-acfc-69d91b291324') Unexpected error (task:870)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 877, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/storage/task.py", line 333, in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
return method(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 1712, in cloneImageStructure
img.cloneStructure(sdUUID, imgUUID, dstSdUUID)
File "/usr/share/vdsm/storage/image.py", line 676, in cloneStructure
self._createTargetImage(sdCache.produce(dstSdUUID), sdUUID, imgUUID)
File "/usr/share/vdsm/storage/image.py", line 450, in _createTargetImage
srcVolUUID=volParams['parent'])
File "/usr/share/vdsm/storage/sd.py", line 758, in createVolume
initialSize=initialSize)
File "/usr/share/vdsm/storage/volume.py", line 1067, in create
initialSize=initialSize)
File "/usr/share/vdsm/storage/blockVolume.py", line 528, in _create
initialTags=(sc.TAG_VOL_UNINIT,))
File "/usr/share/vdsm/storage/lvm.py", line 1158, in createLV
raise se.CannotCreateLogicalVolume(vgName, lvName)
CannotCreateLogicalVolume: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5')
2017-06-14 03:06:48,814-0400 ERROR (periodic/0) [virt.vm] (vmId='7947a8ba-8a9c-4588-9781-b67fd4dcf6aa') Improbable extension request for volume 7615613b-da59-4df0-b726-e9a7c03d3769 on domain d5a4bab3-315c-4814-8ffd-12997537d4c5, pausing the VM to avoid corruptions (capacity: 8589934592, allocated: 8589934592, physical: 1073741824, next physical size: 2147483648) (vm:873)
2017-06-14 03:07:40,919-0400 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='aa5477bd-398d-4bf6-a891-977a380702d2') Unexpected error (task:870)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 877, in _run
return fn(*args, **kargs)
File "<string>", line 2, in getVolumeInfo
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
ret = func(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 3046, in getVolumeInfo
volUUID=volUUID).getInfo()
File "/usr/share/vdsm/storage/sd.py", line 743, in produceVolume
volUUID)
File "/usr/share/vdsm/storage/blockVolume.py", line 479, in __init__
manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID)
File "/usr/share/vdsm/storage/blockVolume.py", line 75, in __init__
volUUID)
File "/usr/share/vdsm/storage/volume.py", line 86, in __init__
self.validate()
File "/usr/share/vdsm/storage/blockVolume.py", line 168, in validate
raise se.VolumeDoesNotExist(self.volUUID)
VolumeDoesNotExist: Volume does not exist: (u'7615613b-da59-4df0-b726-e9a7c03d3769',)
2017-06-14 03:07:40,920-0400 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getVolumeInfo error=Volume does not exist: (u'7615613b-da59-4df0-b726-e9a7c03d3769',) (dispatcher:78)
Actual result
As in step3, "HSMGetAllTasksStatusesVDS failed: Cannot create Logical Volume" while live storage migration for vm with iscsi storage
Expected result
Should succeed to finish live storage migration for vm with iscsi storage without error.
Additional info:
Ala, doesn't this look like a dupe of something you're already working on?
Comment 3Red Hat Bugzilla Rules Engine
2017-06-15 12:13:35 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Description: On rhevm server with iscsi storage, "HSMGetAllTasksStatusesVDS failed: Cannot create Logical Volume" while live storage migration for vm Versions: *vdsm-4.19.18-1.el7ev.x86_64* libvirt-3.2.0-9.el7.x86_64 qemu-kvm-rhev-2.9.0-10.el7.x86_64 How reproducible: 100% Steps: 1. Prepare rhevm env 1 host:hp 2 iscsi storages:IS1,IS2 1 vm:h 2.start the vm 3.select vm h, click on 'disk' tab, select the disk h_Disk1 and click on 'Move' to do live storage migration: Operation failed with following errors: (1)On rhevm server: Jun 14, 2017 3:06:35 AM VDSM hp command HSMGetAllTasksStatusesVDS failed: Cannot create Logical Volume Jun 14, 2017 3:06:42 AM User admin@internal-authz have failed to move disk h_Disk1 to domain IS1. ... Jun 14, 2017 3:06:49 AM VM h has been paused. ... (2)vdsm.log on host: 2017-06-14 03:06:35,484-0400 ERROR (tasks/2) [storage.Volume] Failed to create volume /rhev/data-center/ad36c80c-ff51-464b-8927-887c70d4880e/49858257-ecc7-4ce7-86b4-6ed8410a9e17/images/ede0bc84-9c25-47c1-b884-3cb0e32b0f8a/5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5') (volume:1070) 2017-06-14 03:06:35,484-0400 ERROR (tasks/2) [storage.Volume] Unexpected error (volume:1107) Traceback (most recent call last): File "/usr/share/vdsm/storage/volume.py", line 1067, in create initialSize=initialSize) File "/usr/share/vdsm/storage/blockVolume.py", line 528, in _create initialTags=(sc.TAG_VOL_UNINIT,)) File "/usr/share/vdsm/storage/lvm.py", line 1158, in createLV raise se.CannotCreateLogicalVolume(vgName, lvName) CannotCreateLogicalVolume: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5') 2017-06-14 03:06:35,485-0400 ERROR (tasks/2) [storage.Image] Unexpected error (image:467) Traceback (most recent call last): File "/usr/share/vdsm/storage/image.py", line 450, in _createTargetImage srcVolUUID=volParams['parent']) File "/usr/share/vdsm/storage/sd.py", line 758, in createVolume initialSize=initialSize) File "/usr/share/vdsm/storage/volume.py", line 1067, in create initialSize=initialSize) File "/usr/share/vdsm/storage/blockVolume.py", line 528, in _create initialTags=(sc.TAG_VOL_UNINIT,)) File "/usr/share/vdsm/storage/lvm.py", line 1158, in createLV raise se.CannotCreateLogicalVolume(vgName, lvName) CannotCreateLogicalVolume: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5') 2017-06-14 03:06:35,486-0400 ERROR (tasks/2) [storage.TaskManager.Task] (Task='0600a365-fa4f-4324-acfc-69d91b291324') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "/usr/share/vdsm/storage/task.py", line 333, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 1712, in cloneImageStructure img.cloneStructure(sdUUID, imgUUID, dstSdUUID) File "/usr/share/vdsm/storage/image.py", line 676, in cloneStructure self._createTargetImage(sdCache.produce(dstSdUUID), sdUUID, imgUUID) File "/usr/share/vdsm/storage/image.py", line 450, in _createTargetImage srcVolUUID=volParams['parent']) File "/usr/share/vdsm/storage/sd.py", line 758, in createVolume initialSize=initialSize) File "/usr/share/vdsm/storage/volume.py", line 1067, in create initialSize=initialSize) File "/usr/share/vdsm/storage/blockVolume.py", line 528, in _create initialTags=(sc.TAG_VOL_UNINIT,)) File "/usr/share/vdsm/storage/lvm.py", line 1158, in createLV raise se.CannotCreateLogicalVolume(vgName, lvName) CannotCreateLogicalVolume: Cannot create Logical Volume: ('49858257-ecc7-4ce7-86b4-6ed8410a9e17', '5d2fef3d-ceb9-4876-9c2d-a06f2a2f5ac5') 2017-06-14 03:06:48,814-0400 ERROR (periodic/0) [virt.vm] (vmId='7947a8ba-8a9c-4588-9781-b67fd4dcf6aa') Improbable extension request for volume 7615613b-da59-4df0-b726-e9a7c03d3769 on domain d5a4bab3-315c-4814-8ffd-12997537d4c5, pausing the VM to avoid corruptions (capacity: 8589934592, allocated: 8589934592, physical: 1073741824, next physical size: 2147483648) (vm:873) 2017-06-14 03:07:40,919-0400 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='aa5477bd-398d-4bf6-a891-977a380702d2') Unexpected error (task:870) Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 877, in _run return fn(*args, **kargs) File "<string>", line 2, in getVolumeInfo File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 3046, in getVolumeInfo volUUID=volUUID).getInfo() File "/usr/share/vdsm/storage/sd.py", line 743, in produceVolume volUUID) File "/usr/share/vdsm/storage/blockVolume.py", line 479, in __init__ manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/share/vdsm/storage/blockVolume.py", line 75, in __init__ volUUID) File "/usr/share/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/share/vdsm/storage/blockVolume.py", line 168, in validate raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'7615613b-da59-4df0-b726-e9a7c03d3769',) 2017-06-14 03:07:40,920-0400 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH getVolumeInfo error=Volume does not exist: (u'7615613b-da59-4df0-b726-e9a7c03d3769',) (dispatcher:78) Actual result As in step3, "HSMGetAllTasksStatusesVDS failed: Cannot create Logical Volume" while live storage migration for vm with iscsi storage Expected result Should succeed to finish live storage migration for vm with iscsi storage without error. Additional info: