Bug 1146433

Summary:	E-Mail Spamming from Node during HA state changes
Product:	[Retired] oVirt	Reporter:	Zordrak <redhat.bugzilla>
Component:	ovirt-hosted-engine-ha	Assignee:	Martin Sivák <msivak>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Artyom <alukiano>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.5	CC:	bugs, dfediuck, ecohen, gklein, iheim, lsurette, rbalakri, rgroten, sbonazzo, yeylon
Target Milestone:	---	Keywords:	Triaged
Target Release:	3.5.0
Hardware:	x86_64
OS:	Linux
Whiteboard:	sla
Fixed In Version:	ovirt-hosted-engine-ha-1.2.2-1.el6ev.noarch	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-10-17 12:38:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	SLA	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Zordrak 2014-09-25 09:09:49 UTC

Description of problem:

When nodes are placed into maintenance, instead of one e-mail per state change, it seems to generate e-mails regularly during any state other than up and running.

Version-Release number of selected component (if applicable):
3.5-rc3

How reproducible:
Always

Steps to Reproduce:
1. Run HostedEngine HA across multiple nodes
2. Place a node into Maintenance
3. Perform maintenance
4. Activate node

Actual results:

ovirt-hosted-engine\ state\ transition\ EngineDown-EngineStart\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineDown-EngineStart\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ EngineDown-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1141.eml
ovirt-hosted-engine\ state\ transition\ EngineStart-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineStart-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineUp\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineUp\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254.eml
ovirt-hosted-engine\ state\ transition\ EngineUp-LocalMaintenanceMigrateVm\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenanceMigrateVm-EngineMigratingAway\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ ReinitializeFSM-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1136.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1140.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1204.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1317.eml


Expected results:

<10 e-mails

Comment 1 Martin Sivák 2014-09-26 09:14:08 UTC

Hi,

please see the /etc/ovirt-hosted-engine-ha/broker.conf file, there is a notification section:

[notify]
state_transition = maintenance|start|stop|migrate|on

The logic takes the state transition info (eg. StartState-ReinitializeFSM) and tries to match the state_transition regular expression on it.

If it matches, email is sent.

We might change the default rule a bit, but you can fix this by changing the configuration.

Comment 2 Martin Sivák 2014-09-26 09:17:46 UTC

Looking at the list of emails more closely.. we might reconsider if emitting no change transitions is a good idea.

Comment 3 Artyom 2014-10-06 11:04:07 UTC

Checked on ovirt-hosted-engine-ha-1.2.1-1.el6ev.noarch
Hosted-engine still notify not only when state changed, for example I have notification about local maintenance each time I have line about local maintenance in log. Also I not receive notification when host change status to up.
1) Put host to local maintenance(hosted-engine --set-maintenance --mode=local)
2) Deactivate local maintenance(hosted-engine --set-maintenance --mode=none)
Mails:
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:52 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition LocalMaintenance-ReinitializeFSM October 6, 2014 1:54 PM

Comment 4 Martin Sivák 2014-10-06 11:23:31 UTC

Please test with proper version. Fixed in ovirt-hosted-engine-ha-1.2.1-1.el6ev.noarch

Comment 5 Martin Sivák 2014-10-06 11:25:48 UTC

I ment ovirt-hosted-engine-ha-1.2.2-1.el6ev.noarch

Comment 6 Artyom 2014-10-06 12:07:59 UTC

ok vt4 still have just ovirt-hosted-engine-ha-1.2.1-1.el6ev.noarch version, when I will have newer version will check

Comment 7 Artyom 2014-10-07 10:45:05 UTC

Verified on ovirt-hosted-engine-ha-1.2.2-2.el6ev.noarch

Comment 8 Sandro Bonazzola 2014-10-17 12:38:58 UTC

oVirt 3.5 has been released and should include the fix for this issue.