Bug 1529465

Summary: [HA] after 1 day the agent.log increased to 11GB
Product: [oVirt] ovirt-hosted-engine-ha Reporter: Kobi Hakimi <khakimi>
Component: AgentAssignee: bugs <bugs>
Status: CLOSED DUPLICATE QA Contact: meital avital <mavital>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.2.1CC: bugs, dfediuck, khakimi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-01 08:42:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: SLA RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
agent.log file size and provision 1 day ago
none
zip file of agent.log after few minutes its become to 215MB none

Description Kobi Hakimi 2017-12-28 08:56:56 UTC 
Created attachment 1373108 [details]
agent.log file size and provision 1 day ago

Description of problem:
[HA] after 1 day the agent.log increased to 11GB

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-ha-2.2.2-1.el7ev.noarch
Software Version:4.2.0-0.6.el7

How reproducible:
100%

Steps to Reproduce:
1. In case of some problem like bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1529458 

Actual results:
The agent log increased all the time.
in my case, its increased to 11GB.

Expected results:
To limit the log file size.
when its reach the limited size archive it and create a new one.

Additional info:
See attached snapshot of agent.log file size
Comment 1 Yaniv Kaul 2017-12-29 14:51:14 UTC 
- Severity is missing.
- A snippet of the log would have been useful - to see if there's a repeating log entry there.
- Does it have a log rotate policy?
Comment 2 Kobi Hakimi 2017-12-31 09:55:04 UTC 
Sorry but this machine no longer exists.
if I'll see this again I'll add a snippet of the log file.

about the log rotate question, I think No.
otherwise, we didn't get this file with enormous size. 
but the developer should answer it better than me.
Comment 3 Yaniv Kaul 2017-12-31 10:05:09 UTC 
(In reply to Kobi Hakimi from comment #2)
> Sorry but this machine no longer exists.
> if I'll see this again I'll add a snippet of the log file.

No worries, re-setting the NEEDINFO to get it.
> 
> about the log rotate question, I think No.
> otherwise, we didn't get this file with enormous size. 
> but the developer should answer it better than me.

You should see the config as part of the package.
Comment 4 Doron Fediuck 2017-12-31 11:39:13 UTC 
The HA daemon rotates once a day, keeping the last 7 days for history.
What we need is to find the reason for the flood, which means we need to see a snippet of the logs you had.
Comment 5 Kobi Hakimi 2017-12-31 12:21:28 UTC 
from what I reproduced when killing the vdsmd we can see in agent.log the following line repeat 2 times in one millisecond:
Client localhost:54321::WARNING::2017-12-31 14:12:17,772::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event

as you can see in the following log snippet:
============================================
Client localhost:54321::WARNING::2017-12-31 14:12:17,772::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,773::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,774::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,775::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,775::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,776::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,776::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,778::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,778::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,779::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,779::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,780::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,780::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,782::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,782::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,783::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,783::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,784::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,785::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,786::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,786::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,787::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,787::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,789::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,789::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,790::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Client localhost:54321::WARNING::2017-12-31 14:12:17,790::betterAsyncore::177::vds.dispatcher::(log_info) unhandled close event
Client localhost:54321::WARNING::2017-12-31 14:12:17,791::betterAsyncore::177::vds.dispatcher::(log_info) unhandled write event
Comment 6 Kobi Hakimi 2017-12-31 12:30:04 UTC 
Created attachment 1374896 [details]
zip file of agent.log after few minutes its become to 215MB
Comment 7 Doron Fediuck 2018-01-01 08:42:54 UTC 

*** This bug has been marked as a duplicate of bug 1525453 ***