Created attachment 754756 [details] logs Description of problem: In iscsi storage with 2 hosts cluster, I created a vm on the master domain and than started the vm, when vm was up I started a LSM for the disk and blocked the master domain (src domain only) from both hosts using iptables. after master domain reconstruct, src domain was put in inactive and vm paused and failed the LSM, I restored the connectivity to the storage and once the domain was activated I resumed the vm and than powered it off. when I tried to move the vm's disk off line (when the vm was down) we get an error from vdsm. ImagePathError: Image path does not exist or cannot be accessed/created: ('/rhev/data-center/7fd33b43-a9f4-4eb7-a885-e9583a929ceb/7414f930-bbdb-4ec6-8132-4640cbb3c722/images/aecc0c48-1174-4400-a0e8-002f6d3784a6',) and than ResourceAcqusitionFailed: Could not acquire resource. Probably resource factory threw an exception.: () 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::DEBUG::2013-05-30 12:47:49,264::task::869::TaskManager.Task::(_run) Task=`101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd`::Task._run: 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd () {} failed - stopping task Version-Release number of selected component (if applicable): sf17.2 vdsm-4.10.2-22.0.el6ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. create two iscsi storage domains located on different servers 2. create a template and copy it to both domains 3. create a vm from the template on the master domain 4. run the vm and start a LSM on its disk 5. when the task is sent to the vdsm block the master domain in both hosts using iptables 6. once the reconstruct finished, vm is paused and src domain becomes inactive, restore the connectivity to the master domain from both hosts. 7. once the domain becomes active resume the vm 8. power off the vm and try to move the disk offline Actual results: we fail to move the disk off line with resource error from vdsm Expected results: we should be able to move the vm disk Additional info: logs opening this bug as medium since vdsm restart will solve the issue. 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::ERROR::2013-05-30 12:47:49,259::blockVolume::403::Storage.Volume::(validateImagePath) Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/blockVolume.py", line 401, in validateImagePath os.mkdir(imageDir, 0755) OSError: [Errno 2] No such file or directory: '/rhev/data-center/7fd33b43-a9f4-4eb7-a885-e9583a929ceb/7414f930-bbdb-4ec6-8132-4640cbb3c722/images/aecc0c48-1174-4400-a0e8-002f6d 3784a6' 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::WARNING::2013-05-30 12:47:49,260::resourceManager::520::ResourceManager::(registerResource) Resource factory failed to create resource '74 14f930-bbdb-4ec6-8132-4640cbb3c722_imageNS.aecc0c48-1174-4400-a0e8-002f6d3784a6'. Canceling request. Traceback (most recent call last): File "/usr/share/vdsm/storage/resourceManager.py", line 518, in registerResource obj = namespaceObj.factory.createResource(name, lockType) File "/usr/share/vdsm/storage/resourceFactories.py", line 193, in createResource lockType) File "/usr/share/vdsm/storage/resourceFactories.py", line 122, in __getResourceCandidatesList imgUUID=resourceName) File "/usr/share/vdsm/storage/image.py", line 316, in getChain srcVol = volclass(self.repoPath, sdUUID, imgUUID, uuidlist[0]) File "/usr/share/vdsm/storage/blockVolume.py", line 80, in __init__ volume.Volume.__init__(self, repoPath, sdUUID, imgUUID, volUUID) File "/usr/share/vdsm/storage/volume.py", line 128, in __init__ self.validate() File "/usr/share/vdsm/storage/blockVolume.py", line 89, in validate volume.Volume.validate(self) File "/usr/share/vdsm/storage/volume.py", line 140, in validate self.validateImagePath() File "/usr/share/vdsm/storage/blockVolume.py", line 404, in validateImagePath raise se.ImagePathError(imageDir) ImagePathError: Image path does not exist or cannot be accessed/created: ('/rhev/data-center/7fd33b43-a9f4-4eb7-a885-e9583a929ceb/7414f930-bbdb-4ec6-8132-4640cbb3c722/images/ae cc0c48-1174-4400-a0e8-002f6d3784a6',) 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::DEBUG::2013-05-30 12:47:49,263::resourceManager::186::ResourceManager.Request::(cancel) ResName=`7414f930-bbdb-4ec6-8132-4640cbb3c722_imag eNS.aecc0c48-1174-4400-a0e8-002f6d3784a6`ReqID=`2d8de0e9-1b8d-4acc-aa2a-6d1bbafbe18e`::Canceled request 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::WARNING::2013-05-30 12:47:49,263::resourceManager::180::ResourceManager.Request::(cancel) ResName=`7414f930-bbdb-4ec6-8132-4640cbb3c722_im ageNS.aecc0c48-1174-4400-a0e8-002f6d3784a6`ReqID=`2d8de0e9-1b8d-4acc-aa2a-6d1bbafbe18e`::Tried to cancel a processed request 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::ERROR::2013-05-30 12:47:49,263::task::850::TaskManager.Task::(_setError) Task=`101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 857, in _run return fn(*args, **kargs) File "/usr/share/vdsm/storage/task.py", line 318, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper return f(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 1743, in moveImage with nested(rmanager.acquireResource(srcImageResourcesNamespace, imgUUID, srcLock), File "/usr/share/vdsm/storage/resourceManager.py", line 468, in acquireResource raise se.ResourceAcqusitionFailed() ResourceAcqusitionFailed: Could not acquire resource. Probably resource factory threw an exception.: () 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::DEBUG::2013-05-30 12:47:49,264::task::869::TaskManager.Task::(_run) Task=`101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd`::Task._run: 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd () {} failed - stopping task 101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd::DEBUG::2013-05-30 12:47:49,264::task::1194::TaskManager.Task::(stop) Task=`101d31fa-e9e5-4c9c-a5b1-6705e0af6cdd`::stopping in state running (force False)
also, until vdsm restart vm will not run: Thread-745::ERROR::2013-05-30 13:06:45,085::vm::704::vm.Vm::(_startUnderlyingVm) vmId=`50b98a53-88ce-4364-863b-e8e80f1b0674`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 664, in _startUnderlyingVm self._run() File "/usr/share/vdsm/libvirtvm.py", line 1458, in _run self.preparePaths(devices[vm.DISK_DEVICES]) File "/usr/share/vdsm/vm.py", line 725, in preparePaths drive['path'] = self.cif.prepareVolumePath(drive, self.id) File "/usr/share/vdsm/clientIF.py", line 275, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'index': 0, 'iface': 'ide', 'reqsize': '0', 'format': 'cow', 'bootOrder': '1', 'poolID': '7fd33b43-a9f4-4eb7-a885-e9583a929ceb', 'volumeI D': '7ae20c35-27d0-4e9e-848d-ab480b05472f', 'apparentsize': '1073741824', 'imageID': 'aecc0c48-1174-4400-a0e8-002f6d3784a6', 'specParams': {}, 'readonly': 'false', 'domainID': '7414f930-bbdb-4ec6-8132-4640cbb3c722', 'optional': 'false', 'deviceId': 'aecc0c48-1174-4400-a0e8-002f6d3784a6', 'truesize': '1073741824', 'address': {' controller': '0', ' tar get': '0', 'unit': '0', ' bus': '0', ' type': 'drive'}, 'device': 'disk', 'shared': 'false', 'propagateErrors': 'off', 'type': 'disk'}
I could not reproduce this error - after shutting the VM down, cold move succeeds. Moving to ON_QA to verify.
I was able to move the disk of the VM after shutting the VM down. I am moving this defect to Verified. I followed the scenario described above, HOWEVER there was a change in the behavior described as follows: See my remarks in step 6: Steps to Reproduce: 1. create two iscsi storage domains located on different servers 2. create a template and copy it to both domains 3. create a vm from the template on the master domain 4. run the vm and start a LSM on its disk 5. when the task is sent to the vdsm block the master domain in both hosts using iptables 6. once the reconstruct finished, vm is paused >>>>(VM was not paused)<<<< and src domain becomes inactive, restore the connectivity to the master >>>>>(Master was transferred)<<<<< domain from both hosts. 7. once the domain becomes active resume the vm 8. power off the vm and try to move the disk offline