Description of problem: Attaching iscsi disk to running VM failed with error "VolumeError: Bad volume specification". According to logs LUN existed on storage server and server was connected. Thread-265::DEBUG::2013-03-20 11:06:15,273::BindingXMLRPC::913::vds::(wrapper) client [10.35.161.17]::call vmHotplugDisk with ({'vmId': '7d65ff39-0bc9-4085-bf98-e0e5497c27f7', 'drive': {'iface': 'virtio', 'format': 'raw', 'type': 'disk', 'readonly': 'false', 'deviceId': '821dbcc6-9ceb-4e40-abcf-76170241960b', 'propagateErrors': 'off', 'device': 'disk', 'shared': 'false', 'GUID': '1ce110704-604d-40c2', 'optional': 'false'}},) {} Thread-265::ERROR::2013-03-20 11:06:15,274::BindingXMLRPC::932::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/BindingXMLRPC.py", line 251, in vmHotplugDisk return vm.hotplugDisk(params) File "/usr/share/vdsm/API.py", line 423, in hotplugDisk return curVm.hotplugDisk(params) File "/usr/share/vdsm/libvirtvm.py", line 1760, in hotplugDisk diskParams['path'] = self.cif.prepareVolumePath(diskParams) File "/usr/share/vdsm/clientIF.py", line 291, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'iface': 'virtio', 'format': 'raw', 'type': 'disk', 'readonly': 'false', 'deviceId': '821dbcc6-9ceb-4e40-abcf-76170241960b', 'propagateErrors': 'off', 'device': 'disk', 'shared': 'false', 'GUID': '1ce110704-604d-40c2', 'optional': 'false'} Version-Release number of selected component (if applicable): How reproducible: Happened once in automated tests: http://jenkins.qa.lab.tlv.redhat.com:8080/job/Developer_job/728/ Steps to Reproduce: 1. create 2 LUNs and a VM 2. attach one of the LUNs when the VM is down 3. start VM 4. attach another LUN Actual results: attaching second LUN failed with following error VolumeError: Bad volume specification {'iface': 'virtio', 'format': 'raw', 'type': 'disk', 'readonly': 'false', 'deviceId': '821dbcc6-9ceb-4e40-abcf-76170241960b', 'propagateErrors': 'off', 'device': 'disk', 'shared': 'false', 'GUID': '1ce110704-604d-40c2', 'optional': 'false'} Expected results: attaching second LUN should succeed Additional info:
please attach logs. please confirm VM is installed with an operating system.
Created attachment 713255 [details] whole test logs (with vdsm & engine logs etc.)
According to http://jenkins.qa.lab.tlv.redhat.com:8080/job/Developer_job/728/artifact/logs/automation_run.log this VM was in state "up" at 2013-03-20 11:06:14 and 2013-03-20 11:06:17. The installed system was RHEL6.4. At 2013-03-20 11:06:13,802 test invoked an 'ls' command on the VM (you can see it in the above log file) and it passed.
Yeela, please take a look at this one
This looks like another bug resulting from the multipath race. You can see connectSS: Thread-264::INFO::2013-03-20 11:06:15,214::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '1424248b-0a09-49a7-877d-c 5e205767381'}]} in : 11:06:15,214 and then right afterwards in : 11:06:15,273: Thread-265::DEBUG::2013-03-20 11:06:15,273::BindingXMLRPC::913::vds::(wrapper) client [10.35.161.17]::call vmHotplugDisk with ({'vmId': '7d65ff39-0bc9-4085-bf98-e0e5497c27f7', 'drive': {'iface': 'v irtio', 'format': 'raw', 'type': 'disk', 'readonly': 'false', 'deviceId': '821dbcc6-9ceb-4e40-abcf-76170241960b', 'propagateErrors': 'off', 'device': 'disk', 'shared': 'false', 'GUID': '1ce110704-6 04d-40c2', 'optional': 'false'}},) {} Thread-265::ERROR::2013-03-20 11:06:15,274::BindingXMLRPC::932::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/BindingXMLRPC.py", line 251, in vmHotplugDisk return vm.hotplugDisk(params) File "/usr/share/vdsm/API.py", line 423, in hotplugDisk return curVm.hotplugDisk(params) File "/usr/share/vdsm/libvirtvm.py", line 1760, in hotplugDisk diskParams['path'] = self.cif.prepareVolumePath(diskParams) File "/usr/share/vdsm/clientIF.py", line 291, in prepareVolumePath raise vm.VolumeError(drive) VolumeError: Bad volume specification {'iface': 'virtio', 'format': 'raw', 'type': 'disk', 'readonly': 'false', 'deviceId': '821dbcc6-9ceb-4e40-abcf-76170241960b', 'propagateErrors': 'off', 'device': 'disk', 'shared': 'false', 'GUID': '1ce110704-604d-40c2', 'optional': 'false'} This is not enough time for multipath to create the /dev/mapper entry for the lun on the host the vm is running on.
The patch is merged upstream.
I've run the test 5 times on sf17 and it passed all times. I have no idea how to test it better.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0886.html
This bug was never fixed becasue a bug in the patch that should have fixed this.