Bug 1283936

Summary:	Database connections are not properly closed
Product:	[oVirt] ovirt-engine-extension-aaa-jdbc	Reporter:	Martin Perina <mperina>
Component:	Core	Assignee:	Martin Perina <mperina>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Petr Matyáš <pmatyas>
Severity:	high	Docs Contact:
Priority:	high
Version:	1.0.1	CC:	bugs, mgoldboi, mperina, oourfali, pstehlik
Target Milestone:	ovirt-3.6.1	Flags:	rule-engine: ovirt-3.6.z+ mgoldboi: planning_ack+ oourfali: devel_ack+ pstehlik: testing_ack+
Target Release:	1.0.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	infra
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-16 12:23:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Martin Perina 2015-11-20 10:35:44 UTC

Description of problem:

When accessing engine using REST API periodically using user from domain that is provided by aaa-jdbc extension, database connection are not properly closed until all available Postgresql connections are allocated and following error appears in the log (it's needed to turn on DEBUG log for aaa-jdbc extension):

2015-11-19 15:51:05,453 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (default task-285) [] Unexpected Exception invoking: AAA_AUTHN_AUTHENTICATE_CREDENTIALS[d9605c75-6b43-4b00-b32c-06bdfa80244c]
2015-11-19 15:51:05,453 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (default task-285) [] Exception:: org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
	at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:471) [postgresql.jar:]
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112) [postgresql.jar:]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) [postgresql.jar:]
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) [postgresql.jar:]
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) [postgresql.jar:]
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) [postgresql.jar:]
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32) [postgresql.jar:]
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:]
	at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:]
	at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:]
	at java.sql.DriverManager.getConnection(DriverManager.java:571) [rt.jar:1.7.0_91]
	at java.sql.DriverManager.getConnection(DriverManager.java:215) [rt.jar:1.7.0_91]
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:]
	at com.sun.proxy.$Proxy162.getConnection(Unknown Source)
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:40) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.Sql$Query.asResults(Sql.java:298) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.Sql$Query.asResults(Sql.java:285) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.Schema.get(Schema.java:743) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.Tasks.execute(Tasks.java:49) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:50) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49)
	at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73)
	at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109)
	at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.handleCredentials(BasicAuthenticationFilter.java:134)
	at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:84)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:77)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111)
	at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.api.common.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70)
	at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:248)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:77)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:167)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:766)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_91]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_91]

Version-Release number of selected component (if applicable):

ovirt-engine-extension-aaa-jdbc-1.0.1-1.el7

How reproducible:

100%

Steps to Reproduce:

Number of opened Postgresql connections can be shown using command:

  su - postgres -c "psql -c \"select count(*) from pg_stat_activity;\""

Getting list of existing data centers can be shown using command (replace PASSWORD and ENGINE_FQDN with valid values):

  curl -v -u "admin@internal:PASSWORD" -H "Content-type: application/xml" -X GET https://ENGINE_FQDN/api/datacenters


1. Start ovirt-engine service
2. Check number of opened connections
3. Get list of existing data centers
4. Check number of opened connections
5. Repeat steps 3. - 4. -> number of opened connections increases with each get of existing data centers

Actual results:

Database connections are not properly closed in aaa-jdbc extension

Expected results:

Database connections should be closed properly

Additional info:

Comment 1 Yaniv Kaul 2015-11-22 08:17:18 UTC

Is it a regression? if so, when was it introduced?

Comment 2 Oved Ourfali 2015-11-22 08:19:17 UTC

(In reply to Yaniv Kaul from comment #1)
> Is it a regression? if so, when was it introduced?

The bug is referring to the new aaa-jdbc extension, so this component is new in 3.6.

Comment 3 Martin Perina 2015-11-23 15:51:10 UTC

Fix is contained in ovirt-engine-extension-aaa-jdbc-1.0.3

Comment 4 Petr Matyáš 2015-11-26 10:39:18 UTC

Verified on 3.6.0-22

Comment 5 Sandro Bonazzola 2015-12-16 12:23:26 UTC

According to verification status and target milestone this issue should be fixed in oVirt 3.6.1. Closing current release.