Bug 1146433

Summary: E-Mail Spamming from Node during HA state changes
Product: [Retired] oVirt Reporter: Zordrak <redhat.bugzilla>
Component: ovirt-hosted-engine-haAssignee: Martin Sivák <msivak>
Status: CLOSED CURRENTRELEASE QA Contact: Artyom <alukiano>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.5CC: bugs, dfediuck, ecohen, gklein, iheim, lsurette, rbalakri, rgroten, sbonazzo, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: sla
Fixed In Version: ovirt-hosted-engine-ha-1.2.2-1.el6ev.noarch Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-17 12:38:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Zordrak 2014-09-25 09:09:49 UTC
Description of problem:

When nodes are placed into maintenance, instead of one e-mail per state change, it seems to generate e-mails regularly during any state other than up and running.

Version-Release number of selected component (if applicable):
3.5-rc3

How reproducible:
Always

Steps to Reproduce:
1. Run HostedEngine HA across multiple nodes
2. Place a node into Maintenance
3. Perform maintenance
4. Activate node

Actual results:

ovirt-hosted-engine\ state\ transition\ EngineDown-EngineStart\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineDown-EngineStart\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ EngineDown-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1141.eml
ovirt-hosted-engine\ state\ transition\ EngineStart-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineStart-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineStarting\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineUp\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1200.eml
ovirt-hosted-engine\ state\ transition\ EngineStarting-EngineUp\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254.eml
ovirt-hosted-engine\ state\ transition\ EngineUp-LocalMaintenanceMigrateVm\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1142.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1143.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1144.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1145.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1252.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1253.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1254.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1255.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3-4-5-6.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1256.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2-3-4-5.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2-3-4.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2-3.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257-2.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenance-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1257.eml
ovirt-hosted-engine\ state\ transition\ LocalMaintenanceMigrateVm-EngineMigratingAway\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ ReinitializeFSM-LocalMaintenance\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1251.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1136.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1140.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1204.eml
ovirt-hosted-engine\ state\ transition\ StartState-ReinitializeFSM\ -\ root\@ovirt.example.org\ -\ 2014-09-24\ 1317.eml


Expected results:

<10 e-mails

Comment 1 Martin Sivák 2014-09-26 09:14:08 UTC
Hi,

please see the /etc/ovirt-hosted-engine-ha/broker.conf file, there is a notification section:

[notify]
state_transition = maintenance|start|stop|migrate|on

The logic takes the state transition info (eg. StartState-ReinitializeFSM) and tries to match the state_transition regular expression on it.

If it matches, email is sent.

We might change the default rule a bit, but you can fix this by changing the configuration.

Comment 2 Martin Sivák 2014-09-26 09:17:46 UTC
Looking at the list of emails more closely.. we might reconsider if emitting no change transitions is a good idea.

Comment 3 Artyom 2014-10-06 11:04:07 UTC
Checked on ovirt-hosted-engine-ha-1.2.1-1.el6ev.noarch
Hosted-engine still notify not only when state changed, for example I have notification about local maintenance each time I have line about local maintenance in log. Also I not receive notification when host change status to up.
1) Put host to local maintenance(hosted-engine --set-maintenance --mode=local)
2) Deactivate local maintenance(hosted-engine --set-maintenance --mode=none)
Mails:
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:52 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:53 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition EngineDown-LocalMaintenance October 6, 2014 1:54 PM
ovirt-hosted-engine state transition LocalMaintenance-ReinitializeFSM October 6, 2014 1:54 PM

Comment 4 Martin Sivák 2014-10-06 11:23:31 UTC
Please test with proper version. Fixed in ovirt-hosted-engine-ha-1.2.1-1.el6ev.noarch

Comment 5 Martin Sivák 2014-10-06 11:25:48 UTC
I ment ovirt-hosted-engine-ha-1.2.2-1.el6ev.noarch

Comment 6 Artyom 2014-10-06 12:07:59 UTC
ok vt4 still have just ovirt-hosted-engine-ha-1.2.1-1.el6ev.noarch version, when I will have newer version will check

Comment 7 Artyom 2014-10-07 10:45:05 UTC
Verified on ovirt-hosted-engine-ha-1.2.2-2.el6ev.noarch

Comment 8 Sandro Bonazzola 2014-10-17 12:38:58 UTC
oVirt 3.5 has been released and should include the fix for this issue.