Bug 1435809 - OOM kill Candlepin by updating a custom product on a very large pool
Summary: OOM kill Candlepin by updating a custom product on a very large pool
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.2.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
: 1441949 (view as bug list)
Depends On: 1435807
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-24 21:00 UTC by Barnaby Court
Modified: 2020-07-16 09:20 UTC (History)
12 users (show)

Fixed In Version: candlepin-0.9.54.21-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1435807
: 1446715 (view as bug list)
Environment:
Last Closed: 2017-06-20 17:23:36 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1553 0 normal SHIPPED_LIVE Satellite 6.2.10 Async Bug Release 2017-06-20 21:19:07 UTC

Description Barnaby Court 2017-03-24 21:00:20 UTC
+++ This bug was initially created as a clone of Bug #1435807 +++

Description of problem:

Out of memory when updating a content set that is associated to a pool with very large numbers of entitlements:

This was found using candlepin-0.9.54.17-1.el7.noarch

2017-03-24 18:30:06,614 [thread=http-bio-8443-exec-3] [req=7a4c428e-9f46-45fe-b7ad-b796c9da3cb8, org=] WARN  org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 53200
2017-03-24 18:30:06,614 [thread=http-bio-8443-exec-3] [req=7a4c428e-9f46-45fe-b7ad-b796c9da3cb8, org=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Ran out of memory retrieving query results.
2017-03-24 18:30:06,632 [thread=http-bio-8443-exec-3] [req=7a4c428e-9f46-45fe-b7ad-b796c9da3cb8, org=] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error Ran out of memory ret
rieving query results.
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1817) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) ~[postgresql-9.0-801.jdbc4.jar:na]
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273) ~[postgresql-9.0-801.jdbc4.jar:na]
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76) ~[c3p0-0.9.1.2.jar:0.9.1.2]
        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:56) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.getResultSet(Loader.java:2040) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1837) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1816) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.doQuery(Loader.java:900) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:312) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.Loader.loadCollection(Loader.java:2238) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:65) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:674) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:85) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:1849) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:549) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:234) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:545) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:124) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.hibernate.collection.internal.PersistentSet.iterator(PersistentSet.java:180) ~[hibernate-core-4.2.5.Final.jar:4.2.5.Final]
        at org.candlepin.controller.CandlepinPoolManager.regenerateCertificatesOf(CandlepinPoolManager.java:1025) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:58) ~[guice-persist-3.0.jar:na]
        at org.candlepin.controller.CandlepinPoolManager.regenerateCertificatesOf(CandlepinPoolManager.java:1139) ~[CandlepinPoolManager.class:na]
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:66) ~[guice-persist-3.0.jar:na]
        at org.candlepin.resource.ContentResource.updateContent(ContentResource.java:193) ~[ContentResource.class:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_131]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]

Comment 3 Chris Duryee 2017-03-30 14:24:20 UTC
time to alter URL on product without patch: 171 sec

with patch: 6 sec

Comment 4 Barnaby Court 2017-04-03 15:25:45 UTC
Fixed in Candlepin 0.9.54.21-1

Comment 5 Chris Duryee 2017-04-13 12:42:05 UTC
*** Bug 1441949 has been marked as a duplicate of this bug. ***

Comment 6 jcallaha 2017-06-01 14:04:44 UTC
Verified in Satellite 6.2.10 Snap 2.

I tested this on 5 total products, all of which were aligned to 25k content hosts. The time it took for changes to individual repositories varied from 2 to 8 seconds. The average completion time was right around 6 seconds.

Comment 8 errata-xmlrpc 2017-06-20 17:23:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1553


Note You need to log in before you can comment on or make changes to this bug.