Created attachment 890533 [details] agent.log Description of problem: Exception appeared in time of upgrade all environment from 3.3 to 3.4 Have hosted engine environment environment: In step when environment have one host_3.3 with hosted-engine 3.3(on it run vm now) and another host_3.4 with 3.4, agent on host_3.4 failed to start with exception: MainThread::CRITICAL::2014-04-28 19:10:25,777::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could not start ha-agent Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 97, in run self._run_agent() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 154, in _run_agent hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 303, in start_monitoring for old_state, state, delay in self.fsm: File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", line 125, in next new_data = self.refresh(self._state.data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", line 132, in refresh new_data["best_engine_status"] = best_engine.engine_status AttributeError: 'dict' object has no attribute 'engine_status' Version-Release number of selected component (if applicable): Before upgrade: 2 Host - ovirt-hosted-engine-ha.noarch 0:1.0.0-3.el6ev engine vm - rhevm-3.3.2-0.50.el6ev.noarch After upgrade: 2 Host - ovirt-hosted-engine-ha.noarch 0:1.0.0-3.el6ev engine vm - ovirt-hosted-engine-ha-1.1.2-1.el6ev.noarch How reproducible: Always Steps to Reproduce: Have hosted engine environment environment(hosts and engine vm 3.3) 1. From one of hosts hosted-engine --set-maintenance --mode=global 2. Put to maintenance host without engine vm 3. Upgrade host to 3.4(now it host_3.4), service vdsmd restart && service ovirt-ha-broker restart && service ovirt-ha-agent restart 4. Upgrade engine vm to 3.4 5. hosted-engine --set-maintenance --mode=none 6. Wait few minute until error appear in agent.log of host_3.4 Actual results: Appear error in agent.log and HA agent failed to start Expected results: No error agent.log success to start Additional info: If after all steps I run hosted-engine --set-maintenance --mode=global and upgrade also second host(without maintenance and with vm) all back to normal and HA agent succes to run on host_3.4
on 3.4 host: ovirt-hosted-engine-ha-1.1.2-3.el6ev.noarch ERROR in agent log: MainThread::CRITICAL::2014-05-16 10:57:09,147::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could not start ha-agent Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 97, in run self._run_agent() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 154, in _run_agent hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 303, in start_monitoring for old_state, state, delay in self.fsm: File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", line 125, in next new_data = self.refresh(self._state.data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", line 132, in refresh new_data["best_engine_status"] = best_engine["engine_status"] KeyError: 'engine_status' on 3.3 host: ovirt-hosted-engine-ha-1.0.0-3.el6ev.noarch ainThread::WARNING::2014-05-16 10:57:14,073::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unexpected error Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 329, in start_monitoring self._collect_all_host_stats() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 885, in _collect_all_host_stats in json.loads(md['engine-status']).iteritems()]) File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode obj, end = self._scanner.iterscan(s, **kw).next() File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan rval, next_pos = action(m, context) File "/usr/lib64/python2.6/json/decoder.py", line 171, in JSONObject raise ValueError(errmsg("Expecting property name", s, end)) ValueError: Expecting property name: line 1 column 1 (char 1)
(In reply to Artyom from comment #2) > on 3.3 host: ovirt-hosted-engine-ha-1.0.0-3.el6ev.noarch > > ainThread::WARNING::2014-05-16 > 10:57:14,073::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine. > HostedEngine::(start_monitoring) Unexpected error > Traceback (most recent call last): > File > "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine. > py", line 329, in start_monitoring > self._collect_all_host_stats() > File > "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine. > py", line 885, in _collect_all_host_stats > in json.loads(md['engine-status']).iteritems()]) > File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads > return _default_decoder.decode(s) > File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode > obj, end = self.raw_decode(s, idx=_w(s, 0).end()) > File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode > obj, end = self._scanner.iterscan(s, **kw).next() > File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan > rval, next_pos = action(m, context) > File "/usr/lib64/python2.6/json/decoder.py", line 171, in JSONObject > raise ValueError(errmsg("Expecting property name", s, end)) > ValueError: Expecting property name: line 1 column 1 (char 1) - this exception is not related to this bug, so let's continue this in a separate ticket
Verified on ovirt-hosted-engine-ha-1.1.2-4.el6ev.noarch ovirt-ha-agent up on host 3.4, but go down on 3.3 host https://bugzilla.redhat.com/show_bug.cgi?id=1101299
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0671.html