Bug 1690862 - The engine fails to migrate the hosted-engine VM with "Could not fetch data needed for VM migrate operation"
Summary: The engine fails to migrate the hosted-engine VM with "Could not fetch data n...
Keywords:
Status: CLOSED DUPLICATE of bug 1670701
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Michal Skrivanek
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-20 11:11 UTC by Simone Tiraboschi
Modified: 2019-03-20 11:20 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-20 11:19:11 UTC
oVirt Team: Virt
Embargoed:

Attachments (Terms of Use)
he_migrate.png (382.66 KB, image/png)
2019-03-20 11:13 UTC, Simone Tiraboschi 		no flags Details
engine.log (1.14 MB, text/plain)
2019-03-20 11:14 UTC, Simone Tiraboschi 		no flags Details
View All

Description Simone Tiraboschi 2019-03-20 11:11:44 UTC 
Description of problem:
We have an environment with an hosted-engine cluster with 2 hosts.

The two hosts are correctly deployed and configured for hosted-engine, both have an HA score of 3400 points and are up and running for the engine.

At cluster level we have:
Intel SandyBridge IBRS SSBD Family

At guest level:
Guest CPU Type: Intel SandyBridge IBRS SSBD Family

and on both the hosts:
CPU Type:
Intel SandyBridge IBRS SSBD Family

Kernel Features:
PTI: 1, IBRS: 0, RETP: 1

when we try to migrate the hosted-engine VM from the engine we get a warning with  "Could not fetch data needed for VM migrate operation" (see the attached screenshot).

in engine.log we see just:
2019-03-20 10:44:14,789Z INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-51) [d03b77ca-4c67-4303-b27e-e41d828fb174] Candidate host 'host_mixed_1' ('42a138b6-efd6-4a7e-a6b2-d962b279f9ec') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU' (correlation id: null)
2019-03-20 10:44:14,789Z INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-51) [d03b77ca-4c67-4303-b27e-e41d828fb174] Candidate host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' ('c83a8008-cd71-4671-9f10-1325f3364034') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU' (correlation id: null)


it's worth to mention that in the past we got:
2019-03-20 09:45:51,290Z ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com', last response arrived 16876 ms ago.
2019-03-20 09:48:31,204Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-8) [] EVENT_ID: VM_DOWN_ERROR(119), VM HostedEngine is down with error. Exit message: resource busy: Failed to acquire lock: Lease is held by another host.
2019-03-20 09:51:12,416Z ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com', last response arrived 12659 ms ago.
2019-03-20 09:56:39,075Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [] Unable to RefreshCapabilities: NoRouteToHostException: No route to host
2019-03-20 09:56:39,077Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.NoRouteToHostException: No route to host
2019-03-20 09:56:42,086Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:42,087Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:45,096Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:45,097Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:48,115Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:48,115Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:51,126Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-49) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:51,127Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-49) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:54,135Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:54,135Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:56:57,145Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [] Unable to RefreshCapabilities: ConnectException: Connection refused
2019-03-20 09:56:57,145Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused
2019-03-20 09:57:00,290Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM b01-h03-r620.rhev.openstack.engineering.redhat.com command Get Host Capabilities failed: Recovering from crash or Initializing
2019-03-20 09:57:00,290Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Unable to RefreshCapabilities: VDSRecoveringException: Recovering from crash or Initializing


but then:
2019-03-20 10:27:10,106Z INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [4f6604d7] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the capabilities of host b01-h03-r620.rhev.openstack.engineering.redhat.com.
2019-03-20 10:27:20,651Z INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Before acquiring lock in order to prevent monitoring for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed'
2019-03-20 10:27:20,651Z INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Lock acquired, from now a monitoring of host will be skipped for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed'
2019-03-20 10:27:20,654Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] START, HostSetupNetworksVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, HostSetupNetworksVdsCommandParameters:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='false', bonding='false', networkName='internal_network', vdsmName='on73d84c2b703a4', nicName='em2', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='DHCP', ipv4Address='null', ipv4Netmask='null', ipv4Gateway='null', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='false'}), log id: 34f3981c
2019-03-20 10:27:29,704Z INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Host setup networks finished. Lock released. Monitoring can run now for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed'
2019-03-20 10:27:29,974Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (default task-40) [7588b84a-cec8-48fc-ad9e-5180ca08dc4e] START, SetSafeNetworkConfigVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034'}), log id: 3fd7d378
2019-03-20 10:27:30,002Z INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-40) [7588b84a-cec8-48fc-ad9e-5180ca08dc4e] EVENT_ID: NETWORK_COMMINT_NETWORK_CHANGES(1,104), Network changes were saved on host b01-h03-r620.rhev.openstack.engineering.redhat.com



Version-Release number of selected component (if applicable):
4.3.2

How reproducible:
?

Steps to Reproduce:
1. deploy an hosted-engine cluster with two HE hosts
2. try to migrate the engine VM from one HE host to the second one from the engine web ui
3.

Actual results:
"Could not fetch data needed for VM migrate operation"

Expected results:
the engine triggers the migration

Additional info:
Comment 1 Simone Tiraboschi 2019-03-20 11:13:46 UTC 
Created attachment 1546014 [details]
he_migrate.png
Comment 2 Simone Tiraboschi 2019-03-20 11:14:24 UTC 
Created attachment 1546016 [details]
engine.log
Comment 3 Simone Tiraboschi 2019-03-20 11:19:11 UTC 

*** This bug has been marked as a duplicate of bug 1670701 ***
Comment 4 Simone Tiraboschi 2019-03-20 11:20:58 UTC 
Ok, also in this case is just a frontend issue:

DataProvider.js:35 DataProvider failed to fetch data Error: VmMigrateDataProvider: Failed to fetch target hosts
    at VmMigrateDataProvider.js:70
    at r (vendor.f6c3dc5f.js:sourcemap:46)
    at Generator._invoke (vendor.f6c3dc5f.js:sourcemap:46)
    at Generator.e.(/ovirt-engine/webadmin/anonymous function) [as next] (https://scale-hosted-engine-01.rhev.openstack.engineering.redhat.com/ovirt-engine/webadmin/plugin/ui-extensions/js/vendor.f6c3dc5f.js:46:146147)
    at r (plugin.1c5a6711.js:1886)
    at plugin.1c5a6711.js:1886
Note You need to log in before you can comment on or make changes to this bug.