Bug 1059309 - [events] 'Available memory of host $host (...) under defined threshold...' is logged only once
Summary: [events] 'Available memory of host $host (...) under defined threshold...' is...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-core
Version: 3.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.4.4
Assignee: Martin Sivák
QA Contact: bugs@ovirt.org
URL:
Whiteboard: sla
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-29 15:55 UTC by Jiri Belka
Modified: 2014-09-24 08:41 UTC (History)
8 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2014-09-23 15:08:52 UTC
oVirt Team: ---
Embargoed:


Attachments (Terms of Use)
logs (796.00 KB, application/x-gzip)
2014-01-29 15:55 UTC, Jiri Belka
no flags Details

Description Jiri Belka 2014-01-29 15:55:17 UTC
Created attachment 857111 [details]
logs

Description of problem:

'Available memory of host $host (...) under defined threshold...' is logged only once.

# tail -f /var/log/ovirt-engine/engine.log | grep -i 'Event.*memory'                                                             
2014-01-29 15:31:41,091 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Available memory of host dell-r210ii-03 [2825 MB] is under defined threshold [4098 MB].

Then my host was with memory usage in 'green'.

Then again create memory usage. No new logged event.

I also did not receive any notification even I'm subscribed to this event.

engine=# select * from event_subscriber ;
-[ RECORD 1 ]--+-------------------------------------
subscriber_id  | fdfc627c-d875-11e0-90f0-83df133b58cc
event_up_name  | USER_VDS_MAINTENANCE
method_id      | 0
method_address | jbelka
tag_name       | 
-[ RECORD 2 ]--+-------------------------------------
subscriber_id  | fdfc627c-d875-11e0-90f0-83df133b58cc
event_up_name  | VDS_HIGH_MEM_USE
method_id      | 0
method_address | jbelka
tag_name       |

engine=# select * from event_notification_hist where sent_at > '2014-01-29';
-[ RECORD 1 ]-+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
subscriber_id | fdfc627c-d875-11e0-90f0-83df133b58cc
event_name    | USER_VDS_MAINTENANCE
audit_log_id  | 254
method_type   | EMAIL     
sent_at       | 2014-01-29 13:37:18.887+01
status        | t
reason        |                                                                                                                                                                                                                                                                
-[ RECORD 2 ]-+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
subscriber_id | fdfc627c-d875-11e0-90f0-83df133b58cc
event_name    | VDS_ACTIVATE
audit_log_id  | 261
method_type   | EMAIL     
sent_at       | 2014-01-29 14:00:18.009+01
status        | t
reason        |                                       
                                                                                               
(2 rows)


Version-Release number of selected component (if applicable):
ovirt-engine-backend-3.4.0-0.2.master.20140112020439.git9ad8529.el6.noarch
ovirt-engine-tools-3.4.0-0.2.master.20140112020439.git9ad8529.el6.noarch

How reproducible:
100%

Steps to Reproduce:
1. assign 'Host memory usage exceeded defined threshold' to an user
2. (you can tune engine-config memory related options to easy get high mem usage)
3. create mem usage (I was using: `perl -e '$m = "x" x 2_000_000_000; sleep 600'` twice on the host and I got 98% mem usage)
4. check is event is logged
5. kill those processes or wait when the host is in green with its mem usage
6. repeat step 3
7. check if event is logged

Actual results:
logged only once

Expected results:
logged for every occurrence (of course change between OK - FALSE status)

Additional info:

Comment 1 Jiri Belka 2014-01-29 15:57:03 UTC
If this is two separate issues, one for logged event and other one for notifier, please clone it to notifier component. Thank you.

Comment 2 Itamar Heim 2014-02-02 08:16:51 UTC
Setting target release to current version for consideration and review. please
do not push non-RFE bugs to an undefined target release to make sure bugs are
reviewed for relevancy, fix, closure, etc.

Comment 3 Martin Sivák 2014-02-20 12:58:41 UTC
The VDS_LOW_MEM message repetition is limited to once per 30 minutes. Can you check again but with additional 30m wait time between 5th and 6th step?

Comment 4 Jiri Belka 2014-02-20 13:23:06 UTC
OK, I'll check but I don't understand why such more "advanced" correlation is part of engine. I would expect from the engine only UP|DOWN events types, to stay simple. More advanced correlation, reset counter etc... should be part only of an specialized external correlation engine.

Comment 5 Sandro Bonazzola 2014-03-04 09:24:18 UTC
This is an automated message.
Re-targeting all non-blocker bugs still open on 3.4.0 to 3.4.1.

Comment 6 Sandro Bonazzola 2014-05-08 13:52:09 UTC
This is an automated message.

oVirt 3.4.1 has been released.
This issue has been retargeted to 3.4.2 as it has severity high, please retarget if needed.
If this is a blocker please add it to the tracker Bug #1095370

Comment 7 Gilad Chaplik 2014-06-01 07:58:03 UTC
Martin, did you solve it already?

Comment 8 Sandro Bonazzola 2014-06-11 07:04:27 UTC
This is an automated message:
oVirt 3.4.2 has been released.
This bug has been re-targeted from 3.4.2 to 3.4.3 since priority or severity were high or urgent.

Comment 9 Sandro Bonazzola 2014-06-11 07:05:05 UTC
This is an automated message:
oVirt 3.4.2 has been released.
This bug has been re-targeted from 3.4.2 to 3.4.3 since priority or severity were high or urgent.

Comment 10 Sandro Bonazzola 2014-07-18 09:42:58 UTC
This is an automated message.
oVirt 3.4.3 has been released while this bug was still open.
The bug has been re-targeted to 3.4.4 accordingly.

Comment 11 Martin Sivák 2014-09-18 10:46:18 UTC
What were the results of the test Jiri?

Comment 12 Jiri Belka 2014-09-23 15:08:52 UTC
this BZ is chaotic:

1. the event is VDS_LOW_MEM but subscriber is VDS_HIGH_MEM
2. #3 clearly states there's 30m ignore window and RHEVM doesn't do any start/up end/down event correlation.

i'm closing as notabug. (i have problem to generate VDS_HIGH_MEM events anyway which i'm investigating, i'll file new BZ if it is really an issue.)

Comment 13 Jiri Belka 2014-09-24 08:41:40 UTC
I made new BZ1145977 about VDS_HIGH_MEM ;)


Note You need to log in before you can comment on or make changes to this bug.