Bug 1252820 - VM migration fails with error: 'xxx.yyy.zzz:16509': No route to host
VM migration fails with error: 'xxx.yyy.zzz:16509': No route to host
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: nobody nobody
sla
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-12 05:54 EDT by Dudi Maroshi
Modified: 2016-02-10 15:15 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-08-12 11:05:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: SLA
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dudi Maroshi 2015-08-12 05:54:49 EDT
Description of problem: VM migration fails. 


Version-Release number of selected component (if applicable): 3.6


How reproducible:
All time.
Manual and automatic.


Steps to Reproduce:
1. Have to active hosts in cluster, host A and host B.
2. Start a VM (new or existing) on host A.
3. Migrate VM from host A to host B.

Actual results:
4. VM start migration process (as expected in GUI).
5. Migration fails.


Expected results:
VM Migration success.


Additional info:
-----------------engine.log-----------------

2015-08-12 12:37:42,665 INFO  [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (org.ovirt.thread.pool-8-thread-50) [62a594d0] Running command: MigrateVmToServerCommand internal: false. Entities affected :  ID: bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c Type: VMAction group MIGRATE_VM with role type USER,  ID: bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c Type: VMAction group EDIT_VM_PROPERTIES with role type USER,  ID: bb7509e5-7645-4f8f-a3ba-17c842e73103 Type: VdsGroupsAction group CREATE_VM with role type USER
2015-08-12 12:37:42,699 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-8-thread-50) [62a594d0] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='39285980-3913-4a07-aa99-4b18f4b01a35', vmId='bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c', srcHost='vmhost1-dmaroshi.eng.lab.tlv', dstVdsId='6b34865d-f9e8-4826-861a-d91cfdbb91f4', dstHost='nott09.eng.lab.tlv:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='null', migrateCompressed='null', consoleAddress='null'}), log id: 6fe15189
2015-08-12 12:37:42,701 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [62a594d0] START, MigrateBrokerVDSCommand(HostName = vmh1, MigrateVDSCommandParameters:{runAsync='true', hostId='39285980-3913-4a07-aa99-4b18f4b01a35', vmId='bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c', srcHost='vmhost1-dmaroshi.eng.lab.tlv', dstVdsId='6b34865d-f9e8-4826-861a-d91cfdbb91f4', dstHost='nott09.eng.lab.tlv:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='null', migrateCompressed='null', consoleAddress='null'}), log id: 531ddf89
2015-08-12 12:37:42,992 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [62a594d0] FINISH, MigrateBrokerVDSCommand, log id: 531ddf89
2015-08-12 12:37:43,023 INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-8-thread-50) [62a594d0] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 6fe15189
2015-08-12 12:37:43,042 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-50) [62a594d0] Correlation ID: 62a594d0, Job ID: 21e2731b-4617-4d39-94a5-4f569f39fb8d, Call Stack: null, Custom Event ID: -1, Message: Migration started (VM: j-2, Source: vmh1, Destination: nott09, User: admin@internal).
2015-08-12 12:37:46,023 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-40) [] VM 'bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c'(j-2) moved from 'MigratingFrom' --> 'Up'
2015-08-12 12:37:46,023 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-40) [] Adding VM 'bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c' to re-run list
2015-08-12 12:37:46,035 ERROR [org.ovirt.engine.core.vdsbroker.VmsMonitoring] (DefaultQuartzScheduler_Worker-40) [] Rerun VM 'bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c'. Called from VDS 'vmh1'
2015-08-12 12:37:46,067 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [] START, MigrateStatusVDSCommand(HostName = vmh1, MigrateStatusVDSCommandParameters:{runAsync='true', hostId='39285980-3913-4a07-aa99-4b18f4b01a35', vmId='bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c'}), log id: 49bfdc02
2015-08-12 12:37:46,335 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (org.ovirt.thread.pool-8-thread-48) [] FINISH, MigrateStatusVDSCommand, log id: 49bfdc02
2015-08-12 12:37:46,344 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-48) [] Correlation ID: 62a594d0, Job ID: 21e2731b-4617-4d39-94a5-4f569f39fb8d, Call Stack: null, Custom Event ID: -1, Message: Migration failed  (VM: j-2, Source: vmh1, Destination: nott09).


-----------vdsm.log-------------------
Thread-31322::DEBUG::2015-08-12 12:37:43,747::migration::202::virt.vm::(_prepareGuest) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::Migration started
Thread-31322::DEBUG::2015-08-12 12:37:43,758::migration::286::virt.vm::(run) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::migration semaphore acquired after 0 seconds
Thread-31322::DEBUG::2015-08-12 12:37:43,767::stompreactor::377::jsonrpc.AsyncoreClient::(send) Sending response
Thread-31322::INFO::2015-08-12 12:37:43,801::migration::335::virt.vm::(_startUnderlyingMigration) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::Creation of destination VM took: 0 sec
onds
Thread-31322::INFO::2015-08-12 12:37:43,801::migration::354::virt.vm::(_startUnderlyingMigration) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::starting migration to qemu+tcp://nott0
9.eng.lab.tlv/system with miguri tcp://10.35.19.208
Thread-31323::DEBUG::2015-08-12 12:37:43,802::migration::443::virt.vm::(run) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::migration downtime thread started (10 steps)
Thread-31323::DEBUG::2015-08-12 12:37:43,802::migration::466::virt.vm::(_set_downtime) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::setting migration downtime to 51
Thread-31324::DEBUG::2015-08-12 12:37:43,803::migration::500::virt.vm::(monitor_migration) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::starting migration monitor thread
Thread-31322::DEBUG::2015-08-12 12:37:44,144::migration::558::virt.vm::(stop) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::stopping migration monitor thread
Thread-31322::DEBUG::2015-08-12 12:37:44,144::migration::453::virt.vm::(stop) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::stopping migration downtime thread
Thread-31322::ERROR::2015-08-12 12:37:44,145::migration::208::virt.vm::(_recover) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::operation failed: Failed to connect to remote libvirt 
URI qemu+tcp://nott09.eng.lab.tlv/system: unable to connect to server at 'nott09.eng.lab.tlv:16509': No route to host
Thread-31322::DEBUG::2015-08-12 12:37:44,145::stompreactor::377::jsonrpc.AsyncoreClient::(send) Sending response
Thread-31322::DEBUG::2015-08-12 12:37:44,173::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c": {"status": "Migration Sou
rce"}, "notify_time": 4550132440}, "jsonrpc": "2.0", "method": "|virt|VM_status|bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c"}
Thread-31322::DEBUG::2015-08-12 12:37:44,173::stompreactor::304::yajsonrpc.StompServer::(send) Sending response
Thread-31322::ERROR::2015-08-12 12:37:44,173::migration::310::virt.vm::(run) vmId=`bcb2ba8f-b884-4d01-945e-3cd1e1eb5b8c`::Failed to migrate
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/migration.py", line 294, in run
    self._startUnderlyingMigration(time.time())
  File "/usr/share/vdsm/virt/migration.py", line 364, in _startUnderlyingMigration
    self._perform_migration(duri, muri)
  File "/usr/share/vdsm/virt/migration.py", line 403, in _perform_migration
    self._vm._dom.migrateToURI3(duri, params, flags)
  File "/usr/share/vdsm/virt/vm.py", line 206, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1712, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://nott09.eng.lab.tlv/system: unable to connect to server at 'nott09.eng.lab.tlv:16509': No route to host

Unrelated to bug 1248788
Comment 1 Dudi Maroshi 2015-08-12 09:33:51 EDT
Researching into the problem.
1. Running fresh host deploy on both hosts (A and B), including firewall config.
2. Test migration from A to B, Problem persist.
3. Disable iptables service on migrate destination (host B).
4. Migrate vm from host A to host B. Migration successful.

I believe the problem is with firewall config setting in ovirt 3.6.
Comment 2 Dudi Maroshi 2015-08-12 11:05:04 EDT
Developer configured Libvirt open non secured ports.
This is not a bug, but a development configuration.

Note You need to log in before you can comment on or make changes to this bug.