Bug 1026377
| Summary: | HA agent dies after manual migration of engine vm. | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Leonid Natapov <lnatapov> | ||||||||||
| Component: | ovirt-hosted-engine-ha | Assignee: | Greg Padgett <gpadgett> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Artyom <alukiano> | ||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | unspecified | CC: | acathrow, amureini, dfediuck, gpadgett, iheim, mavital, sbonazzo | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Target Release: | 3.3.1 | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | sla | ||||||||||||
| Fixed In Version: | ovirt-hosted-engine-ha-0.1.0-0.8.rc.el6ev | Doc Type: | Bug Fix | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2014-01-21 16:51:19 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | SLA | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | |||||||||||||
| Bug Blocks: | 1078909, 1142926 | ||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 819182 [details]
logs
Merged Change-Id: I9da40599098446e6915a2bd675096d00fa07516d Created attachment 827293 [details]
agen log
Checked on ovirt-hosted-engine-ha-0.1.0-0.6.beta1.el6ev.noarch
Migration also look ok, but after migration source host score is dropped to 0
And I also have error message in source host agent log
This bug is currently attached to errata RHEA-2013:15591. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag. Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information: * Cause: What actions or circumstances cause this bug to present. * Consequence: What happens when the bug presents. * Fix: What was done to fix the bug. * Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore') Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug. For further details on the Cause, Consequence, Fix, Result format please refer to: https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes Thanks in advance. ovirt-hosted-engine-ha is a new package; does not need errata for bugs during its development. Created attachment 834311 [details]
vdsm and agent log
Checked on ovirt-hosted-engine-ha-0.1.0-0.8.rc.el6ev.noarch, migration still failed, with error in agent and vdsm log, but vm shutdown and start on other log
Created attachment 834357 [details]
libvirt logs
We identified a separate issue causing migration failures, moving back to ON_QA to verify the existing HA agent code submitted. Verified on ovirt-hosted-engine-ha-0.1.0-0.9.rc.el6ev.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0080.html |
Description of problem: I am manually migrating engine VM form one host to another. It seems that everything is ok. At least according to UI. migration starts and migration ends. The engine vm runs on the second host. ha-agent log shows different situation: Here is the destination machine ha agent log: --------------------------------------------- MainThread::INFO::2013-11-04 15:29:17,254::brokerlink::108::BrokerLink::(put_stats_on_storage) Storing blocks on storage at /rhev/data-center/mnt/orion.qa.lab.tlv.redhat.com:_export_hosted__engine2/6177a07d-7789-49ba-96d2-1469700a1509/ha_agent MainThread::ERROR::2013-11-04 15:29:17,363::hosted_engine::720::HostedEngine::(_collect_all_host_stats) Host green-vdsb.qa.lab.tlv.redhat.com (id 1) is no longer updating its metadata MainThread::INFO::2013-11-04 15:29:17,367::hosted_engine::729::HostedEngine::(_collect_all_host_stats) Host green-vdsa.qa.lab.tlv.redhat.com (id 2) metadata updated MainThread::INFO::2013-11-04 15:29:17,367::hosted_engine::734::HostedEngine::(_collect_all_host_stats) Host green-vdsa.qa.lab.tlv.redhat.com (id 2): {'last-update-host-ts': 1383571749, 'last-update-local-ts': 1383571757.36339, 'hostname': 'green-vdsa.qa.lab.tlv.redhat.com', 'alive': True, 'engine-status': 'vm-down', 'score': 0, 'first-update': False} MainThread::INFO::2013-11-04 15:29:17,368::hosted_engine::729::HostedEngine::(_collect_all_host_stats) Host green-vdsc.qa.lab.tlv.redhat.com (id 3) metadata updated MainThread::INFO::2013-11-04 15:29:17,368::hosted_engine::734::HostedEngine::(_collect_all_host_stats) Host green-vdsc.qa.lab.tlv.redhat.com (id 3): {'last-update-host-ts': 1383571757, 'last-update-local-ts': 1383571757.36339, 'hostname': 'green-vdsc.qa.lab.tlv.redhat.com', 'alive': True, 'engine-status': 'vm-up good-health-status', 'score': 2400, 'first-update': False} MainThread::WARNING::2013-11-04 15:29:17,419::hosted_engine::262::HostedEngine::(start_monitoring) Error while monitoring engine: Error 12 from migrateStatus: Fatal error during migration MainThread::WARNING::2013-11-04 15:29:17,419::hosted_engine::265::HostedEngine::(start_monitoring) Unexpected error Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 259, in start_monitoring self._perform_engine_actions() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 788, in _perform_engine_actions = self._vm_state_actions[self._rinfo['current-state']]() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 46, in cleanup_wrapper ret = f(self) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 1118, in _handle_migrate vm_id, File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/vds_client.py", line 85, in run_vds_client_cmd response['status']['message']) DetailedError: Error 12 from migrateStatus: Fatal error during migration MainThread::ERROR::2013-11-04 15:29:17,419::hosted_engine::279::HostedEngine::(start_monitoring) Shutting down the agent because of 3 failures in a row! MainThread::INFO::2013-11-04 15:29:17,419::brokerlink::54::BrokerLink::(disconnect) Closing connection to ha-broker MainThread::INFO::2013-11-04 15:29:17,420::agent::107::Broker::(run) Agent shutting down ---------------------------------------------------------------------------- Agent shutting down. On the source machine ha-agent tries to start a VM because it can't see that the first host updating its metadata. I am attaching ha and vdsm logs from both hosts.