Description of problem: AttributeError: 'SourceThread' object has no attribute '_destServer' in vdsm log when failing to migrate VM to destination. Following errors appears in vdsm log if destination not reachable or can't be found. It's a bug in the code and we shouldn't see this error in the vdsm log. See - vim /usr/share/vdsm/virt/migration.py +208 Thread-499::ERROR::2016-01-19 08:19:14,160::migration::208::virt.vm::(_recover) vmId=`0974fb9c-131f-4ee4-a428-1d8172e489a2`::[Errno -2] Name or service not known Thread-499::ERROR::2016-01-19 08:19:14,160::migration::213::virt.vm::(_recover) vmId=`0974fb9c-131f-4ee4-a428-1d8172e489a2`::Failed to destroy remote VM Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 211, in _recover self._destServer.destroy(self._vm.id) AttributeError: 'SourceThread' object has no attribute '_destServer' Thread-499::DEBUG::2016-01-19 08:19:14,160::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4301361260, "0974fb9c-131f-4ee4-a428-1d8172e489a2": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|0974fb9c-131f-4ee4-a428-1d8172e489a2"} Thread-499::ERROR::2016-01-19 08:19:14,160::migration::310::virt.vm::(run) vmId=`0974fb9c-131f-4ee4-a428-1d8172e489a2`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 278, in run self._setupVdsConnection() File "/usr/share/vdsm/virt/migration.py", line 143, in _setupVdsConnection client = self._createClient(port) File "/usr/share/vdsm/virt/migration.py", line 130, in _createClient self.remoteHost, int(port), sslctx) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1267, in create_connected_socket sock.connect((host, port)) File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 181, in connect self.socket.connect(addr) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) gaierror: [Errno -2] Name or service not known Version-Release number of selected component (if applicable): vdsm- 4.17.17
this is useful debug info in migration failure recovery flow (also next time, please attach full log as well)
(In reply to Michal Skrivanek from comment #1) > this is useful debug info in migration failure recovery flow > > (also next time, please attach full log as well) The failure to connect to a destination server should be logged, that is clear. The bug here is that the exception-handling code uses a data member before its initialization. I'm taking the liberty to re-open this bug, and send a patch.
Michael, would you be kind to verify the patch?
Done)
Michal, I think we should fix this for next 3.6, the fix is trivial.
Another case from the user list: http://lists.ovirt.org/pipermail/users/2016-March/038545.html
Michael, would you like to verify the 3.6 backport? http://gerrit.ovirt.org/54949
sure, why not
(In reply to Nir Soffer from comment #7) > Michael, would you like to verify the 3.6 backport? > http://gerrit.ovirt.org/54949 Done)
Verify with: RHEVM:3.6.5.3-0.1.el6 Host: OS Version:RHEL - 7.2 - 9.el7_2.1 Kernel Version:3.10.0 - 327.13.1.el7.x86_64 KVM Version:2.3.0 - 31.el7_2.10 LIBVIRT Version:libvirt-1.2.17-13.el7_2.4 VDSM Version:vdsm-4.17.26-0.el7ev SPICE Version:0.12.4 - 15.el7 steps: Case 1: 1. Migrate VM 2. Reboot destination host Case 2: 1. Migrate VM 2. Disconnect NIC (ifdown nic_name) Results:PASS In both cases the migrate failed, and not reference in the log for: AttributeError: 'SourceThread' object has no attribute '_destServer vdsm log output: ---------------------------------------------------- Case1: With reboot for dest host after migration started: ---------------------------------------------------- Thread-41738::DEBUG::2016-04-14 11:24:25,914::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 5187293370, "d3d27d79-4534-4806-aa59-4684a7d8625b": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|d3d27d79-4534-4806-aa59-4684a7d8625b"} Thread-41738::ERROR::2016-04-14 11:24:25,915::migration::310::virt.vm::(run) vmId=`d3d27d79-4534-4806-aa59-4684a7d8625b`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 298, in run self._startUnderlyingMigration(time.time()) File "/usr/share/vdsm/virt/migration.py", line 364, in _startUnderlyingMigration self._perform_migration(duri, muri) File "/usr/share/vdsm/virt/migration.py", line 403, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: internal error: process exited while connecting to monitor: *********************************** ----------------------------------- Case 2: With disconnect the nic: ---------------------------------- Thread-41821::ERROR::2016-04-14 11:29:43,489::migration::214::virt.vm::(_recover) vmId=`d3d27d79-4534-4806-aa59-4684a7d8625b`::Failed to destroy remote VM Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 212, in _recover self._destServer.destroy(self._vm.id) File "/usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py", line 68, in _callMethod raise JsonRpcNoResponseError(method) JsonRpcNoResponseError: [-32605] No response for JSON-RPC VM.destroy request. Thread-41821::DEBUG::2016-04-14 11:29:43,509::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 5187610970, "d3d27d79-4534-4806-aa59-4684a7d8625b": {"status": "Migration Source"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|d3d27d79-4534-4806-aa59-4684a7d8625b"} Thread-41821::ERROR::2016-04-14 11:29:43,510::migration::310::virt.vm::(run) vmId=`d3d27d79-4534-4806-aa59-4684a7d8625b`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 298, in run self._startUnderlyingMigration(time.time()) File "/usr/share/vdsm/virt/migration.py", line 364, in _startUnderlyingMigration self._perform_migration(duri, muri) File "/usr/share/vdsm/virt/migration.py", line 403, in _perform_migration self._vm._dom.migrateToURI3(duri, params, flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: internal error: received hangup / error event on socket