Bug 854598 - [ovirt-engine-backend] - Flooding Logs - server.log & engine.log are flooded when postgresql is stopped in Remote DB Setup
[ovirt-engine-backend] - Flooding Logs - server.log & engine.log are flooded ...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.1.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Yair Zaslavsky
David Botzer
infra
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-05 08:08 EDT by David Botzer
Modified: 2016-02-10 14:38 EST (History)
11 users (show)

See Also:
Fixed In Version: si19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-04 15:01:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
remote-sql-stop (69.27 KB, application/x-gzip)
2012-09-05 08:08 EDT, David Botzer
no flags Details

  None (edit)
Description David Botzer 2012-09-05 08:08:08 EDT
Created attachment 610012 [details]
remote-sql-stop

Description of problem:
High rate of logs in server.log & engine.log when postgresql is stopped in Remote DB Setup

Version-Release number of selected component (if applicable):
3.1/si17

How reproducible:
always

Steps to Reproduce:
1.Install rhevm_reports+dwh in Remote DB setup
2.create 1 entity of a kind (DC,CL, Host,VM, Template)
3.Stop the postgresql service on the remote DB server
  
Actual results:
always

Expected results:
When the same issue is reported it should send messages in low rate
at a longer period of time between messages

Additional info:
Comment 3 David Botzer 2012-09-06 10:51:22 EDT
If postgresql is down, an administrator never suspect the rhevm logs are getting flooded.
So the admin can leave the postgres issue to be dealt in lower priority not realizing the impact it has on other components.

Also realizing this issue, could bring him to resolution of setting up the system differently.

Happens only on remote Setup
Comment 4 Barak 2012-09-08 11:27:56 EDT
Moved severity to medium - per comment #1

David - When you say flood, what do you mean? how many logs a minute are written?
Comment 5 David Botzer 2012-09-09 05:14:29 EDT
engine.log resolution of a second
every second, 130 messages like this ->
2012-09-05 14:48:09,184 ERROR [org.ovirt.engine.core.bll.Backend] (ajp-/127.0.0.1:8009-6) Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving database metadata; nested exception is org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource
------------------------------------------
server.log resolution of a second
every second, 700 messages like this ->
2012-09-05 15:01:46,993 WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject] (ajp-/127.0.0.1:8009-4) IJ000604: Throwable while attempting to get a new connection: null: javax.resource.ResourceException: Could not create connection
Comment 6 Yair Zaslavsky 2012-09-11 03:19:27 EDT
From checking the logs -
Error in getting DB connection. The database is inaccessible. Original exception is: DataAccessResourceFailureException: Error retreiving 

Is logged when backend starts, but the db is inaccessible.

What happens when backend is up, and then the server goes down?
Comment 7 David Botzer 2012-09-11 03:48:24 EDT
Sorry, I didnt understood the message,
---
db is unaccessible because I did service postgresql stop
while ovirt-engine is started

when starting postgresql its ok,
Comment 8 Yair Zaslavsky 2012-09-11 04:02:58 EDT
David, let me clarify my question/request -
Can you please do the same after ovirt-engine is up for a while, and see how this behaves?
Comment 9 David Botzer 2012-09-11 04:45:34 EDT
Hi,

Let me fix the bug steps:

1. I stopped postgresql on the remote DB server
2. I Restart the ovirt-engine service and only than the flooding started,
Comment 10 David Botzer 2012-09-11 05:10:20 EDT
I stand corrected:
Same issue on local setup.
1. I stopped postgresql on the remote DB server
2. I Restart the ovirt-engine service and only than the flooding started,
Comment 11 Yair Zaslavsky 2012-09-11 07:44:49 EDT
David explained me the issue is with service restart after db is down.

The following is an upstream patch to relax the flooding:

http://gerrit.ovirt.org/#/c/7928/
Comment 16 David Botzer 2012-09-30 04:07:40 EDT
Fixed, 3.1/si19.1
I repeated the same sequence and no flooding in the logs,
Nor after I started sql server,
Webadmin and reports are accessibly - ok, 
Fixed, 3.1/si19.1

Note You need to log in before you can comment on or make changes to this bug.