Bug 1092075 - Could not start ha-agent with exception AttributeError: 'dict' object has no attribute 'engine_status'
Summary: Could not start ha-agent with exception AttributeError: 'dict' object has no ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-ha
Version: 3.4.0
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: 3.4.0
Assignee: Jiri Moskovcak
QA Contact: Artyom
URL:
Whiteboard: sla
Depends On:
Blocks: 1108871
TreeView+ depends on / blocked
 
Reported: 2014-04-28 16:47 UTC by Artyom
Modified: 2016-02-10 20:18 UTC (History)
7 users (show)

Fixed In Version: ovirt-hosted-engine-ha-1.1.2-4.el6ev
Doc Type: Bug Fix
Doc Text:
This update fixes a typo in the meta data dictionary that would prevent environments in a hosted engine configuration from upgrading successfully under certain conditions.
Clone Of:
: 1101299 (view as bug list)
Environment:
Last Closed: 2014-06-09 14:26:48 UTC
oVirt Team: SLA


Attachments (Terms of Use)
agent.log (163.31 KB, text/x-log)
2014-04-28 16:47 UTC, Artyom
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:0671 normal SHIPPED_LIVE ovirt-hosted-engine-ha bug fix and enhancement update 2014-06-09 18:25:01 UTC
oVirt gerrit 27177 master MERGED fixed exception when trying to access dict item Never
oVirt gerrit 27588 ovirt-hosted-engine-ha-1.1 MERGED fixed exception when trying to access dict item Never
oVirt gerrit 27832 master MERGED fixed typo when accessing the metadata dict Never
oVirt gerrit 27833 ovirt-hosted-engine-ha-1.1 MERGED fixed typo when accessing the metadata dict Never

Internal Links: 1101299

Description Artyom 2014-04-28 16:47:15 UTC
Created attachment 890533 [details]
agent.log

Description of problem:
Exception appeared in time of upgrade all environment from 3.3 to 3.4
Have hosted engine environment environment:
In step when environment have one host_3.3 with hosted-engine 3.3(on it run vm now)
and another host_3.4 with 3.4, agent on host_3.4 failed to start with exception:
MainThread::CRITICAL::2014-04-28 19:10:25,777::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could not start ha-agent
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 97, in run
    self._run_agent()
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 154, in _run_agent
    hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 303, in start_monitoring
    for old_state, state, delay in self.fsm:
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", line 125, in next
    new_data = self.refresh(self._state.data)
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", line 132, in refresh
    new_data["best_engine_status"] = best_engine.engine_status
AttributeError: 'dict' object has no attribute 'engine_status'


Version-Release number of selected component (if applicable):
Before upgrade:
2 Host - ovirt-hosted-engine-ha.noarch 0:1.0.0-3.el6ev
engine vm - rhevm-3.3.2-0.50.el6ev.noarch
After upgrade:
2 Host - ovirt-hosted-engine-ha.noarch 0:1.0.0-3.el6ev
engine vm - ovirt-hosted-engine-ha-1.1.2-1.el6ev.noarch

How reproducible:
Always

Steps to Reproduce:
Have hosted engine environment environment(hosts and engine vm 3.3)
1. From one of hosts  hosted-engine --set-maintenance --mode=global
2. Put to maintenance host without engine vm
3. Upgrade host to 3.4(now it host_3.4), service vdsmd restart && service ovirt-ha-broker restart && service ovirt-ha-agent restart
4. Upgrade engine vm to 3.4
5. hosted-engine --set-maintenance --mode=none
6. Wait few minute until error appear in agent.log of host_3.4 

Actual results:
Appear error in agent.log and HA agent failed to start

Expected results:
No error agent.log success to start

Additional info:
If after all steps I run hosted-engine --set-maintenance --mode=global
and upgrade also second host(without maintenance and with vm) all back to normal and HA agent succes to run on host_3.4

Comment 2 Artyom 2014-05-16 08:00:43 UTC
on 3.4 host: ovirt-hosted-engine-ha-1.1.2-3.el6ev.noarch

ERROR in agent log:
MainThread::CRITICAL::2014-05-16 10:57:09,147::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could not start ha-agent
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 97, in run
    self._run_agent()
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 154, in _run_agent
    hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring()
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 303, in start_monitoring
    for old_state, state, delay in self.fsm:
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", line 125, in next
    new_data = self.refresh(self._state.data)
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", line 132, in refresh
    new_data["best_engine_status"] = best_engine["engine_status"]
KeyError: 'engine_status'

on 3.3 host: ovirt-hosted-engine-ha-1.0.0-3.el6ev.noarch

ainThread::WARNING::2014-05-16 10:57:14,073::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unexpected error
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 329, in start_monitoring
    self._collect_all_host_stats()
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 885, in _collect_all_host_stats
    in json.loads(md['engine-status']).iteritems()])
  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
    obj, end = self._scanner.iterscan(s, **kw).next()
  File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib64/python2.6/json/decoder.py", line 171, in JSONObject
    raise ValueError(errmsg("Expecting property name", s, end))
ValueError: Expecting property name: line 1 column 1 (char 1)

Comment 3 Jiri Moskovcak 2014-05-20 08:14:01 UTC
(In reply to Artyom from comment #2)

> on 3.3 host: ovirt-hosted-engine-ha-1.0.0-3.el6ev.noarch
> 
> ainThread::WARNING::2014-05-16
> 10:57:14,073::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(start_monitoring) Unexpected error
> Traceback (most recent call last):
>   File
> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.
> py", line 329, in start_monitoring
>     self._collect_all_host_stats()
>   File
> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.
> py", line 885, in _collect_all_host_stats
>     in json.loads(md['engine-status']).iteritems()])
>   File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
>     return _default_decoder.decode(s)
>   File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
>     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>   File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
>     obj, end = self._scanner.iterscan(s, **kw).next()
>   File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
>     rval, next_pos = action(m, context)
>   File "/usr/lib64/python2.6/json/decoder.py", line 171, in JSONObject
>     raise ValueError(errmsg("Expecting property name", s, end))
> ValueError: Expecting property name: line 1 column 1 (char 1)

- this exception is not related to this bug, so let's continue this in a separate ticket

Comment 5 Artyom 2014-05-26 16:48:21 UTC
Verified on ovirt-hosted-engine-ha-1.1.2-4.el6ev.noarch
ovirt-ha-agent up on host 3.4, but go down on 3.3 host https://bugzilla.redhat.com/show_bug.cgi?id=1101299

Comment 6 errata-xmlrpc 2014-06-09 14:26:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0671.html


Note You need to log in before you can comment on or make changes to this bug.