Created attachment 587745 [details] full log & testcase Description of problem: when appling to suspend VM this error appers (and machine ends in paused state): Thread-1245::ERROR::2012-05-29 16:34:55,565::vm::176::vm.Vm::(_recover) vmId=`7d1605f4-7d0b-496c-af52-74e63934d7d5`::unable to close /rhev/data-center/3c3bb676-a5a7-4f9c-8e2a-3ff2cc505285/3e91a341-f773-44b3-89f2-ce2e97672b6f/images/f8ecb0c2-9218-40fb-a31d-27e8dc93a3f8/f4619812-e573-495f-a1f8-1ef05934ad2e: Input/output error Thread-1245::ERROR::2012-05-29 16:34:55,565::vm::821::vm.Vm::(cont) vmId=`7d1605f4-7d0b-496c-af52-74e63934d7d5`::cannot cont while Saving State Thread-1245::ERROR::2012-05-29 16:34:55,576::vm::240::vm.Vm::(run) vmId=`7d1605f4-7d0b-496c-af52-74e63934d7d5`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 223, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 409, in _startUnderlyingMigration self._vm._dom.save(fname) File "/usr/share/vdsm/libvirtvm.py", line 491, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1215, in save if ret == -1: raise libvirtError ('virDomainSave() failed', dom=self) libvirtError: unable to close /rhev/data-center/3c3bb676-a5a7-4f9c-8e2a-3ff2cc505285/3e91a341-f773-44b3-89f2-ce2e97672b6f/images/f8ecb0c2-9218-40fb-a31d-27e8dc93a3f8/f4619812-e573-495f-a1f8-1ef05934ad2e: Input/output error Thread-1247::DEBUG::2012-05-29 16:34:55,662::BindingXMLRPC::864::vds::(wrapper) client [10.35.16.225]::call vmGetStats with ('7d1605f4-7d0b-496c-af52-74e63934d7d5',) {} Additional info: Attached is full log & testcase
vdsm version? is it nfs or block domain? since you pinpoint a libvirtError - could you attach relevant part of libvirtd.log, and mention libvirt version?
- NFS storage was used software versions: ------------------ vdsm-4.9.6-16.0.el6.x86_64 libvirt-0.9.10-21.el6.x86_64 from vdsm.log: -------------- Thread-1319::ERROR::2012-06-13 14:52:36,483::vm::176::vm.Vm::(_recover) vmId=`73594152-cec5-4115-9357-a1b805945e04`::unable to close /rhev/data-center/61285a30-7edd-467a-8596-ce62bb4f3d44/602a3739-d688-4471-ae17-47007e18154b/images/6a21b667-da7e-4bdc-a8fd-e35dfc6bc9bf/3909323e-ecd3-4bd2-97ee-7314f460d49c: Input/output error Thread-1319::ERROR::2012-06-13 14:52:36,483::vm::821::vm.Vm::(cont) vmId=`73594152-cec5-4115-9357-a1b805945e04`::cannot cont while Saving State Thread-1319::ERROR::2012-06-13 14:52:36,589::vm::240::vm.Vm::(run) vmId=`73594152-cec5-4115-9357-a1b805945e04`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 223, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 409, in _startUnderlyingMigration self._vm._dom.save(fname) File "/usr/share/vdsm/libvirtvm.py", line 491, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1215, in save if ret == -1: raise libvirtError ('virDomainSave() failed', dom=self) libvirtError: unable to close /rhev/data-center/61285a30-7edd-467a-8596-ce62bb4f3d44/602a3739-d688-4471-ae17-47007e18154b/images/6a21b667-da7e-4bdc-a8fd-e35dfc6bc9bf/3909323e-ecd3-4bd2-97ee-7314f460d49c: Input/output error Thread-1322::DEBUG::2012-06-13 14:52:37,530::BindingXMLRPC::864::vds::(wrapper) client [10.35.16.189]::call vmGetStats with ('73594152-cec5-4115-9357-a1b805945e04',) {} Thread-1322::DEBUG::2012-06-13 14:52:37,530::libvirtvm::240::vm.Vm::(_getDiskStats) vmId=`73594152-cec5-4115-9357-a1b805945e04`::Disk hdc stats not available from libvirtd.log: ------------------ 2012-06-13 11:52:36.331+0000: 2215: debug : qemuMonitorUnref:210 : QEMU_MONITOR_UNREF: mon=0x7f821800b800 refs=3 2012-06-13 11:52:36.331+0000: 2225: debug : qemuMonitorUnref:210 : QEMU_MONITOR_UNREF: mon=0x7f821800b800 refs=2 2012-06-13 11:52:36.420+0000: 2225: debug : virStorageFileIsSharedFSType:1031 : Check if path /rhev/data-center/61285a30-7edd-467a-8596-ce62bb4f3d44/602a3739-d688-4471-ae17-47007e18154b/images/6a21b667-da7e-4bdc-a8fd-e35dfc6bc9bf/3909323e-ecd3-4bd2-97ee-7314f460d49c with FS magic 26985 is shared 2012-06-13 11:52:36.421+0000: 2225: error : qemuDomainSaveInternal:2722 : unable to close /rhev/data-center/61285a30-7edd-467a-8596-ce62bb4f3d44/602a3739-d688-4471-ae17-47007e18154b/images/6a21b667-da7e-4bdc-a8fd-e35dfc6bc9bf/3909323e-ecd3-4bd2-97ee-7314f460d49c: Input/output error 2012-06-13 11:52:36.421+0000: 2225: debug : qemuDomainObjEndAsyncJob:887 : Stopping async job: save 2012-06-13 11:52:36.465+0000: 2225: debug : virDomainFree:2313 : dom=0x7f821800b060, (VM: name=restvm_basic_operations, uuid=73594152-cec5-4115-9357-a1b805945e04) 2012-06-13 11:52:36.487+0000: 2224: debug : virDomainGetXMLDesc:4379 : dom=0x7f822400e880, (VM: name=restvm_basic_operations, uuid=73594152-cec5-4115-9357-a1b805945e04), flags=0 new vdsm.log & corresponding libvirtd.log are attached
Created attachment 591466 [details] vdsm & libvirtd logs
why is this a regression? What do we expect the VM to do in case of disk io error/out of space?
(In reply to comment #4) > why is this a regression? It worked in 3.0. > What do we expect the VM to do in case of disk io error/out of space? To pause. But I do wonder why we get an IO error. Petr - any idea? Anything in VDSM log?
actually not, but will arrange the machine with this simulated tomorrow and will contact Michal Skrivanek offline to have a look directly on the machine.
(In reply to comment #6) > actually not, but will arrange the machine with this simulated tomorrow and > will contact Michal Skrivanek offline to have a look directly on the machine. any update?
the behavior seems to be ok, nevertheless no update for a looong time...