Bug 1435807

Summary: OOM kill Candlepin by updating a custom product on a very large pool
Product: [Community] Candlepin Reporter: Barnaby Court <bcourt>
Component: candlepinAssignee: Kevin Howell <khowell>
Status: CLOSED CURRENTRELEASE QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 0.9.54CC: cduryee, khowell, redakkan, skallesh
Target Milestone: ---Keywords: Triaged
Target Release: 0.9.54   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: candlepin-0.9.54.21-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1435809 1436258 (view as bug list) Environment:
Last Closed: 2017-04-03 15:25:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1435809, 1436258, 1446715    

Description Barnaby Court 2017-03-24 20:55:55 UTC
Description of problem:

Out of memory when updating a content set that is associated to a pool with very large numbers of entitlements:

This was found using candlepin-0.9.54.17-1.el7.noarch

2017-03-24 18:30:06,614 [thread=http-bio-8443-exec-3] [req=7a4c428e-9f46-45fe-b7ad-b796c9da3cb8, org=] WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 53200
2017-03-24 18:30:06,614 [thread=http-bio-8443-exec-3] [req=7a4c428e-9f46-45fe-b7ad-b796c9da3cb8, org=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Ran out of memory retrieving query results.
2017-03-24 18:30:06,632 [thread=http-bio-8443-exec-3] [req=7a4c428e-9f46-45fe-b7ad-b796c9da3cb8, org=] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error Ran out of memory ret
rieving query results.
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1817) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273) ~[postgresql-9.0-801.jdbc4.jar:na]
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) ~[c3p0-0.9.1.2.jar:0.9.1.2]
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:56) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.getResultSet(Loader.java:2040) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1837) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1816) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.doQuery(Loader.java:900) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:312) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.loadCollection(Loader.java:2238) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:65) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:674) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:85) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:1849) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:549) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:234) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:545) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:124) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.PersistentSet.iterator(PersistentSet.java:180) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.candlepin.controller.CandlepinPoolManager.regenerateCertificatesOf(CandlepinPoolManager.java:1025) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
        at org.candlepin.controller.CandlepinPoolManager.regenerateCertificatesOf(CandlepinPoolManager.java:1139) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na]
        at org.candlepin.resource.ContentResource.updateContent(ContentResource.java:193) ~[ContentResource.class:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_131]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]

Comment 1 Chris Duryee 2017-03-24 21:16:22 UTC
this might be caused by updating the URLs on products in a pool with lots of clients