Bug 1292089

Summary: ha-broker failed to start after upgrade of RHEV-H from 3.5 to 3.6
Product: [oVirt] ovirt-hosted-engine-ha Reporter: Artyom <alukiano>
Component: BrokerAssignee: Martin Sivák <msivak>
Status: CLOSED CURRENTRELEASE QA Contact: Artyom <alukiano>
Severity: urgent Docs Contact:
Priority: high    
Version: 1.3.3.2CC: alukiano, bugs, cshao, dfediuck, fdeutsch, gklein, huiwa, huzhao, mgoldboi, stirabos, ycui
Target Milestone: ovirt-3.6.1Keywords: Triaged
Target Release: 1.3.3.4Flags: rule-engine: ovirt-3.6.z+
rule-engine: blocker+
mgoldboi: planning_ack+
dfediuck: devel_ack+
gklein: testing_ack+
Hardware: x86_64   
OS: Linux   
Whiteboard: sla
Fixed In Version: ovirt-hosted-engine-ha-1.3.3.4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-13 14:40:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1285700    
Attachments:
Description Flags
sosreport from RHEV-H 7.2 none

Description Artyom 2015-12-16 12:53:33 UTC
Created attachment 1106388 [details]
sosreport from RHEV-H 7.2

Description of problem:
ha-broker failed to start after upgrade of RHEV-H from 3.5 to 3.6 with exception
MainThread::CRITICAL::2015-12-16 09:56:14,885::broker::108::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Could not start ha-broker
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 102, in run
    self._run_server()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 155, in _run_server
    self._monitor_instance = self._get_monitor()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 173, in _get_monitor
    return monitor.Monitor()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/monitor.py", line 33, in __init__
    self._discover_submonitors()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/monitor.py", line 41, in _discover_submonitors
    for filename in (f for f in os.listdir(smdir) if
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 147, in _handle_quit
    self._listener.close_connections()
AttributeError: 'NoneType' object has no attribute 'close_connections'

Version-Release number of selected component (if applicable):
3.5 - Red Hat Enterprise Virtualization Hypervisor release 7.1 (20151015.0.el7ev)
=================================
vdsm-4.16.27-1.el7ev.x86_64
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
ovirt-hosted-engine-ha-1.2.7.2-1.el7ev.noarch

3.6 - Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151210.1.el7ev)
=================================
vdsm-4.17.13-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.1.2-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.3.3-1.el7ev.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy 3.5 HE environment on one host
2. Put host to global maintenance and stop engine vm(hosted-engine --set-maintenance --mode=global && hosted-engine --vm-poweroff)
3. Reboot host and upgrade it to 3.6
4. Check hosted-engine services

Actual results:
ovirt-ha-broker service down

Expected results:
ovirt-ha-broker service up

Additional info:

Comment 1 Martin Sivák 2015-12-17 14:26:52 UTC
Can you please retest with the latest version? (1.3.3.4-1) We committed a fix recently that might have fixed this: https://gerrit.ovirt.org/#/c/50662/

Comment 2 Red Hat Bugzilla Rules Engine 2015-12-17 14:34:57 UTC
Fixed bug tickets must have target milestone set prior to fixing them. Please set the correct milestone and move the bugs back to the previous status after this is corrected.

Comment 3 Artyom 2015-12-17 15:54:38 UTC
I checked it with patch above and looks like error not appear anymore and ovirt-ha-broker succeed to start after upgrade. But need to check it on RHEV-H with new package to be sure if all works correct.

Comment 5 Red Hat Bugzilla Rules Engine 2015-12-21 07:17:04 UTC
Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.

Comment 6 Artyom 2015-12-22 15:54:47 UTC
Verified
Upgrade from 3.5 - Red Hat Enterprise Virtualization Hypervisor release 7.1 (20151015.0.el7ev)
==============================================================================================
ovirt-hosted-engine-ha-1.2.7.2-1.el7ev.noarch
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
vdsm-4.16.27-1.el7ev.x86_64

to 3.6 - Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151221.1.el7ev)
==============================================================================================
ovirt-hosted-engine-setup-1.3.1.3-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.3.6-1.el7ev.noarch
vdsm-4.17.13-1.el7ev.noarch

1) Install RHEV-H 3.5
2) Deploy hosted-engine on one host with NFS storage
3) Enable global maintenance(on host hosted-engine --set-maintenance --mode=global)
4) Upgrade engine to 3.6
5) Power off engine vm(hosted-engine --vm-poweroff)
6) Upgrade host to RHEV-H 3.6 via usb-key

Upgrade succeed and after host upgrade ovirt-he-agent and ovirt-ha-broker up.

Comment 7 Sandro Bonazzola 2016-01-13 14:40:06 UTC
oVirt 3.6.1 has been released, closing current release