Bug 881071

Summary: ovirt-engine-notification-service : "Failed to run the service" errors due to connection limit exceeded
Product: Red Hat Enterprise Virtualization Manager Reporter: Oded Ramraz <oramraz>
Component: ovirt-engine-notification-serviceAssignee: Yair Zaslavsky <yzaslavs>
Status: CLOSED DUPLICATE QA Contact: Pavel Stehlik <pstehlik>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.1.0CC: dyasny, iheim, istein, oourfali, Rhev-m-bugs, ykaul
Target Milestone: ---   
Target Release: 3.1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-03 15:27:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
notifier log none

Description Oded Ramraz 2012-11-28 15:01:44 UTC
Description of problem:

I have notification service running on my RHEVM for 2 days . The service is running but its log is flooded with "Failed to run the service error" ( see Additional Information ) .

When running lsof command on my host I noticed that there are more than 200 open connections for this process ( see Additional Information ) 

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
### number of open connections:  

[root@aqua-rhel notifier]# ps -ef | grep -i notifier
ovirt    10951     1  0 Nov27 ?        00:02:04 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -cp /etc/ovirt-engine/notifier:/usr/share/java/antlr.jar:/usr/share/java/commons-codec.jar:/usr/share/java/commons-collections.jar:/usr/share/java/commons-configuration.jar:/usr/share/java/commons-jxpath.jar:/usr/share/java/commons-lang.jar:/usr/share/java/commons-logging.jar:/usr/share/java/dom4j.jar:/usr/share/java/hibernate-jpa-2.0-api.jar:/usr/share/java/javamail.jar:/usr/share/java/log4j.jar:/usr/share/java/ovirt-engine/common.jar:/usr/share/java/ovirt-engine/compat.jar:/usr/share/java/ovirt-engine/engineencryptutils.jar:/usr/share/java/ovirt-engine/engine-notifier-service.jar:/usr/share/java/ovirt-engine/engine-tools-common.jar:/usr/share/java/ovirt-engine/utils.jar:/usr/share/java/postgresql-jdbc.jar:/usr/share/java/slf4j/api.jar:/usr/share/java/jaxb-impl.jar:/usr/share/ovirt-engine/engine.ear/lib/hibernate-annotations-3.4.0.GA.jar:/usr/share/ovirt-engine/engine.ear/lib/hibernate-commons-annotations-3.1.0.GA.jar:/usr/share/ovirt-engine/engine.ear/lib/hibernate-core-3.3.0.SP1.jar:/usr/share/ovirt-engine/engine.ear/lib/hibernate-validator-4.0.2.GA.jar:/usr/share/ovirt-engine/engine.ear/lib/validation-api-1.0.0.GA.jar org.ovirt.engine.core.notifier.Notifier /etc/ovirt-engine/notifier/notifier.conf
root     19128 17633  0 16:55 pts/1    00:00:00 grep -i notifier
[root@aqua-rhel notifier]# lsof | grep 10951 | grep ovirt | wc -l
242


## exception in the log: 

2012-11-28 16:50:05,690 DEBUG [org.ovirt.engine.core.notifier.EngineMonitorService] checkServerStatus return: true
2012-11-28 16:50:08,694 DEBUG [org.ovirt.engine.core.notifier.EngineMonitorService] checkServerStatus return: true
2012-11-28 16:50:09,695 DEBUG [org.ovirt.engine.core.notifier.NotificationService] Start event notification service iteration
2012-11-28 16:50:09,699 ERROR [org.ovirt.engine.core.notifier.NotificationService] Failed to run the service: [Failed to initialize the connection helper]
org.ovirt.engine.core.notifier.NotificationServiceException: Failed to initialize the connection helper
        at org.ovirt.engine.core.notifier.NotificationService.startup(NotificationService.java:117)
        at org.ovirt.engine.core.notifier.NotificationService.run(NotificationService.java:94)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.postgresql.util.PSQLException: FATAL: connection limit exceeded for non-superusers
        at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:464)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
        at org.postgresql.Driver.makeConnection(Driver.java:393)
        at org.postgresql.Driver.connect(Driver.java:267)
        at java.sql.DriverManager.getConnection(DriverManager.java:579)
        at java.sql.DriverManager.getConnection(DriverManager.java:221)
        at org.ovirt.engine.core.tools.common.db.StandaloneDataSource.openConnection(StandaloneDataSource.java:118)
        at org.ovirt.engine.core.tools.common.db.StandaloneDataSource.<init>(StandaloneDataSource.java:104)
        at org.ovirt.engine.core.notifier.NotificationService.startup(NotificationService.java:114)
        ... 9 more
2012-11-28 16:50:11,698 DEBUG [org.ovirt.engine.core.notifier.EngineMonitorService] checkServerStatus return: true

Comment 1 Oded Ramraz 2012-11-28 15:06:00 UTC
Created attachment 653580 [details]
notifier log

Comment 3 Ilanit Stein 2012-12-03 15:27:38 UTC
closed duplicate of 882847

*** This bug has been marked as a duplicate of bug 882847 ***