Bug 1690862
| Summary: | The engine fails to migrate the hosted-engine VM with "Could not fetch data needed for VM migrate operation" | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Simone Tiraboschi <stirabos> | ||||||
| Component: | BLL.Virt | Assignee: | Michal Skrivanek <michal.skrivanek> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | meital avital <mavital> | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 4.3.0 | CC: | bugs | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2019-03-20 11:19:11 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1546014 [details]
he_migrate.png
Created attachment 1546016 [details]
engine.log
*** This bug has been marked as a duplicate of bug 1670701 *** Ok, also in this case is just a frontend issue:
DataProvider.js:35 DataProvider failed to fetch data Error: VmMigrateDataProvider: Failed to fetch target hosts
at VmMigrateDataProvider.js:70
at r (vendor.f6c3dc5f.js:sourcemap:46)
at Generator._invoke (vendor.f6c3dc5f.js:sourcemap:46)
at Generator.e.(/ovirt-engine/webadmin/anonymous function) [as next] (https://scale-hosted-engine-01.rhev.openstack.engineering.redhat.com/ovirt-engine/webadmin/plugin/ui-extensions/js/vendor.f6c3dc5f.js:46:146147)
at r (plugin.1c5a6711.js:1886)
at plugin.1c5a6711.js:1886
|
Description of problem: We have an environment with an hosted-engine cluster with 2 hosts. The two hosts are correctly deployed and configured for hosted-engine, both have an HA score of 3400 points and are up and running for the engine. At cluster level we have: Intel SandyBridge IBRS SSBD Family At guest level: Guest CPU Type: Intel SandyBridge IBRS SSBD Family and on both the hosts: CPU Type: Intel SandyBridge IBRS SSBD Family Kernel Features: PTI: 1, IBRS: 0, RETP: 1 when we try to migrate the hosted-engine VM from the engine we get a warning with "Could not fetch data needed for VM migrate operation" (see the attached screenshot). in engine.log we see just: 2019-03-20 10:44:14,789Z INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-51) [d03b77ca-4c67-4303-b27e-e41d828fb174] Candidate host 'host_mixed_1' ('42a138b6-efd6-4a7e-a6b2-d962b279f9ec') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU' (correlation id: null) 2019-03-20 10:44:14,789Z INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-51) [d03b77ca-4c67-4303-b27e-e41d828fb174] Candidate host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' ('c83a8008-cd71-4671-9f10-1325f3364034') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'CPU' (correlation id: null) it's worth to mention that in the past we got: 2019-03-20 09:45:51,290Z ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com', last response arrived 16876 ms ago. 2019-03-20 09:48:31,204Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-8) [] EVENT_ID: VM_DOWN_ERROR(119), VM HostedEngine is down with error. Exit message: resource busy: Failed to acquire lock: Lease is held by another host. 2019-03-20 09:51:12,416Z ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com', last response arrived 12659 ms ago. 2019-03-20 09:56:39,075Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [] Unable to RefreshCapabilities: NoRouteToHostException: No route to host 2019-03-20 09:56:39,077Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.NoRouteToHostException: No route to host 2019-03-20 09:56:42,086Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2019-03-20 09:56:42,087Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused 2019-03-20 09:56:45,096Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2019-03-20 09:56:45,097Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused 2019-03-20 09:56:48,115Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2019-03-20 09:56:48,115Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-45) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused 2019-03-20 09:56:51,126Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-49) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2019-03-20 09:56:51,127Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-49) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused 2019-03-20 09:56:54,135Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2019-03-20 09:56:54,135Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused 2019-03-20 09:56:57,145Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2019-03-20 09:56:57,145Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-32) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdAndVdsVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]'})' execution failed: java.net.ConnectException: Connection refused 2019-03-20 09:57:00,290Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM b01-h03-r620.rhev.openstack.engineering.redhat.com command Get Host Capabilities failed: Recovering from crash or Initializing 2019-03-20 09:57:00,290Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Unable to RefreshCapabilities: VDSRecoveringException: Recovering from crash or Initializing but then: 2019-03-20 10:27:10,106Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [4f6604d7] EVENT_ID: HOST_REFRESHED_CAPABILITIES(606), Successfully refreshed the capabilities of host b01-h03-r620.rhev.openstack.engineering.redhat.com. 2019-03-20 10:27:20,651Z INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Before acquiring lock in order to prevent monitoring for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed' 2019-03-20 10:27:20,651Z INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Lock acquired, from now a monitoring of host will be skipped for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed' 2019-03-20 10:27:20,654Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] START, HostSetupNetworksVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, HostSetupNetworksVdsCommandParameters:{hostId='c83a8008-cd71-4671-9f10-1325f3364034', vds='Host[b01-h03-r620.rhev.openstack.engineering.redhat.com,c83a8008-cd71-4671-9f10-1325f3364034]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='false', bonding='false', networkName='internal_network', vdsmName='on73d84c2b703a4', nicName='em2', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='DHCP', ipv4Address='null', ipv4Netmask='null', ipv4Gateway='null', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='false'}), log id: 34f3981c 2019-03-20 10:27:29,704Z INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-40) [83d66dac-6170-49c7-8502-b3f740ff08df] Host setup networks finished. Lock released. Monitoring can run now for host 'b01-h03-r620.rhev.openstack.engineering.redhat.com' from data-center 'golden_env_mixed' 2019-03-20 10:27:29,974Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (default task-40) [7588b84a-cec8-48fc-ad9e-5180ca08dc4e] START, SetSafeNetworkConfigVDSCommand(HostName = b01-h03-r620.rhev.openstack.engineering.redhat.com, VdsIdVDSCommandParametersBase:{hostId='c83a8008-cd71-4671-9f10-1325f3364034'}), log id: 3fd7d378 2019-03-20 10:27:30,002Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-40) [7588b84a-cec8-48fc-ad9e-5180ca08dc4e] EVENT_ID: NETWORK_COMMINT_NETWORK_CHANGES(1,104), Network changes were saved on host b01-h03-r620.rhev.openstack.engineering.redhat.com Version-Release number of selected component (if applicable): 4.3.2 How reproducible: ? Steps to Reproduce: 1. deploy an hosted-engine cluster with two HE hosts 2. try to migrate the engine VM from one HE host to the second one from the engine web ui 3. Actual results: "Could not fetch data needed for VM migrate operation" Expected results: the engine triggers the migration Additional info: