Bug 1276740

Summary:	race condition refreshing /var/lib/ovirt-hosted-engine-ha/broker.conf
Product:	[oVirt] ovirt-hosted-engine-ha	Reporter:	Simone Tiraboschi <stirabos>
Component:	Broker	Assignee:	Simone Tiraboschi <stirabos>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Artyom <alukiano>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	1.3.0	CC:	alukiano, bmcclain, bugs, dfediuck, gklein, mavital, sbonazzo
Target Milestone:	ovirt-3.6.0-rc3	Keywords:	Triaged
Target Release:	1.3.2.1	Flags:	rule-engine: ovirt-3.6.0+ rule-engine: blocker+ bmcclain: planning_ack+ dfediuck: devel_ack+ gklein: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	integration
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-22 13:29:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1278130
Bug Blocks:	1234906, 1285700

Description Simone Tiraboschi 2015-10-30 17:19:21 UTC

Description of problem:
rpm upgrade of ovirt-ha-agent will move the broker configuration file from
/etc/ovirt-hosted-engine-ha/broker.conf
to
/var/lib/ovirt-hosted-engine-ha/broker.conf

The agent upgrade procedure will save /var/lib/ovirt-hosted-engine-ha/broker.conf 
on the shared storage.
The broker will periodically refresh from there to /var/lib/ovirt-hosted-engine-ha/broker.conf 

Due to race condition /var/lib/ovirt-hosted-engine-ha/broker.conf can be overwritten with none before it got copied to the shared storage.

Version-Release number of selected component (if applicable):


How reproducible:
race condition

Steps to Reproduce:
1. upgrade form 3.5 to 3.6
2.
3.

Actual results:
'None' could get written in /var/lib/ovirt-hosted-engine-ha/broker.conf if the race condition happens and than it got copied to the shared storage like that.

Expected results:
var/lib/ovirt-hosted-engine-ha/broker.conf is got overwritten only if the content from the shared storage is valid.

Additional info:

Comment 1 Red Hat Bugzilla Rules Engine 2015-11-01 16:24:09 UTC

This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.

Comment 2 Simone Tiraboschi 2015-11-02 14:33:24 UTC

*** Bug 1277137 has been marked as a duplicate of this bug. ***

Comment 3 Artyom 2015-12-01 16:23:54 UTC

Verified on ovirt-hosted-engine-ha-1.3.3-1.el7ev.noarch
1) Deploy hosted-engine 3.5 on two hosts and on NFS storage
2) Put first host to maintenance via webadmin
3) Upgrade packages and restart host(restart host W/A because bug https://bugzilla.redhat.com/show_bug.cgi?id=1282187)
4) Wait for correct status via hosted-engine --vm-status(can take around 5-7 minutes)
5) Activate host via webadmin
6) Put second host to maintenance(wait until all vms migrated and he vm migrate on first host)
7) Upgrade packages and restart second host
8) Wait for correct status via hosted-engine --vm-status(can take around 5-7 minutes)
9) Activate second host via webadmin
10) Put environment to global maintenance
11) Update rhevm-setup.noarch package on engine
12) Run engine-setup on vm and finish upgrade process
13) Disable global maintenance via webadmin

Comment 4 Sandro Bonazzola 2015-12-22 13:29:26 UTC

oVirt 3.6.0 has been released and the bz verified, moving to closed current release.