Bug 1283936 - Database connections are not properly closed
Database connections are not properly closed
Status: CLOSED CURRENTRELEASE
Product: ovirt-engine-extension-aaa-jdbc
Classification: oVirt
Component: Core (Show other bugs)
1.0.1
Unspecified Unspecified
high Severity high (vote)
: ovirt-3.6.1
: 1.0.3
Assigned To: Martin Perina
Petr Matyáš
infra
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-20 05:35 EST by Martin Perina
Modified: 2016-02-10 14:10 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-16 07:23:26 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑3.6.z+
mgoldboi: planning_ack+
oourfali: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 48863 master MERGED core: Fix improper releasing of SQL objects Never
oVirt gerrit 48985 ovirt-engine-extension-aaa-jdbc-1.0 MERGED core: Fix improper releasing of SQL objects Never

  None (edit)
Description Martin Perina 2015-11-20 05:35:44 EST
Description of problem:

When accessing engine using REST API periodically using user from domain that is provided by aaa-jdbc extension, database connection are not properly closed until all available Postgresql connections are allocated and following error appears in the log (it's needed to turn on DEBUG log for aaa-jdbc extension):

2015-11-19 15:51:05,453 ERROR [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (default task-285) [] Unexpected Exception invoking: AAA_AUTHN_AUTHENTICATE_CREDENTIALS[d9605c75-6b43-4b00-b32c-06bdfa80244c]
2015-11-19 15:51:05,453 DEBUG [org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension] (default task-285) [] Exception:: org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
	at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:471) [postgresql.jar:]
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112) [postgresql.jar:]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) [postgresql.jar:]
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) [postgresql.jar:]
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) [postgresql.jar:]
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) [postgresql.jar:]
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32) [postgresql.jar:]
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) [postgresql.jar:]
	at org.postgresql.Driver.makeConnection(Driver.java:393) [postgresql.jar:]
	at org.postgresql.Driver.connect(Driver.java:267) [postgresql.jar:]
	at java.sql.DriverManager.getConnection(DriverManager.java:571) [rt.jar:1.7.0_91]
	at java.sql.DriverManager.getConnection(DriverManager.java:215) [rt.jar:1.7.0_91]
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.DataSourceProvider$1.invoke(DataSourceProvider.java:86) [ovirt-engine-extension-aaa-jdbc.jar:]
	at com.sun.proxy.$Proxy162.getConnection(Unknown Source)
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.SchemaAwareDataSource.getConnection(SchemaAwareDataSource.java:40) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.Sql$Query.asResults(Sql.java:298) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.datasource.Sql$Query.asResults(Sql.java:285) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.Schema.get(Schema.java:743) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.core.Tasks.execute(Tasks.java:49) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.extension.aaa.jdbc.binding.api.AuthnExtension.invoke(AuthnExtension.java:50) [ovirt-engine-extension-aaa-jdbc.jar:]
	at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:49)
	at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:73)
	at org.ovirt.engine.core.extensions.mgr.ExtensionProxy.invoke(ExtensionProxy.java:109)
	at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.handleCredentials(BasicAuthenticationFilter.java:134)
	at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:84)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:77)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.core.aaa.filters.EngineSessionTokenAuthenticationFilter.doFilter(EngineSessionTokenAuthenticationFilter.java:31)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111)
	at org.ovirt.engine.api.common.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at org.ovirt.engine.api.common.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183)
	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70)
	at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:248)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:77)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:167)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:766)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_91]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_91]

Version-Release number of selected component (if applicable):

ovirt-engine-extension-aaa-jdbc-1.0.1-1.el7

How reproducible:

100%

Steps to Reproduce:

Number of opened Postgresql connections can be shown using command:

  su - postgres -c "psql -c \"select count(*) from pg_stat_activity;\""

Getting list of existing data centers can be shown using command (replace PASSWORD and ENGINE_FQDN with valid values):

  curl -v -u "admin@internal:PASSWORD" -H "Content-type: application/xml" -X GET https://ENGINE_FQDN/api/datacenters


1. Start ovirt-engine service
2. Check number of opened connections
3. Get list of existing data centers
4. Check number of opened connections
5. Repeat steps 3. - 4. -> number of opened connections increases with each get of existing data centers

Actual results:

Database connections are not properly closed in aaa-jdbc extension

Expected results:

Database connections should be closed properly

Additional info:
Comment 1 Yaniv Kaul 2015-11-22 03:17:18 EST
Is it a regression? if so, when was it introduced?
Comment 2 Oved Ourfali 2015-11-22 03:19:17 EST
(In reply to Yaniv Kaul from comment #1)
> Is it a regression? if so, when was it introduced?

The bug is referring to the new aaa-jdbc extension, so this component is new in 3.6.
Comment 3 Martin Perina 2015-11-23 10:51:10 EST
Fix is contained in ovirt-engine-extension-aaa-jdbc-1.0.3
Comment 4 Petr Matyáš 2015-11-26 05:39:18 EST
Verified on 3.6.0-22
Comment 5 Sandro Bonazzola 2015-12-16 07:23:26 EST
According to verification status and target milestone this issue should be fixed in oVirt 3.6.1. Closing current release.

Note You need to log in before you can comment on or make changes to this bug.