* Create custom base channel. * Push F9 i386 packages into channel. Should end up with about 2500 packages. * Delete channel from GUI which should 'orphan' all 2500 F9 packages. * Go to Manage Software Packages * Try to delete all 2500 F9 packages and you get: Caused by: com.redhat.rhn.common.hibernate.HibernateRuntimeException: HibernateException executing CachedStatement at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:463) at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:425) at com.redhat.rhn.common.db.datasource.CachedStatement.executeUpdate(CachedStatement.java:269) at com.redhat.rhn.common.db.datasource.WriteMode.executeUpdate(WriteMode.java:32) at com.redhat.rhn.manager.rhnpackage.PackageManager.clearNeededPackageCache(PackageManager.java:1004) at com.redhat.rhn.manager.rhnpackage.PackageManager.deletePackages(PackageManager.java:1020) at com.redhat.rhn.frontend.action.rhnpackage.CustomPackagesDeleteConfirmAction.deletePackages(CustomPackagesDeleteConfirmAction.java:103) at com.redhat.rhn.frontend.action.rhnpackage.CustomPackagesDeleteConfirmAction.execute(CustomPackagesDeleteConfirmAction.java:80) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) ... 40 more Caused by: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:61) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:46) at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:24) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2520) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2697) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:74) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:146) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000) at com.redhat.rhn.common.db.datasource.CachedStatement.stealConnection(CachedStatement.java:847) at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:435) ... 48 more 2009-02-05 17:28:57,719 [TP-Processor2] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/rhn].[action] - Servlet.service() for servlet action threw exception org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 at org.hibernate.jdbc.Expectations$BasicExpectation.checkBatched(Expectations.java:61) at org.hibernate.jdbc.Expectations$BasicExpectation.verifyOutcome(Expectations.java:46) at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:24) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2520) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:2697) at org.hibernate.action.EntityDeleteAction.execute(EntityDeleteAction.java:74) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:146) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000) at com.redhat.rhn.common.db.datasource.CachedStatement.stealConnection(CachedStatement.java:847) at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:435) at com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:425) at com.redhat.rhn.common.db.datasource.CachedStatement.executeUpdate(CachedStatement.java:269) at com.redhat.rhn.common.db.datasource.WriteMode.executeUpdate(WriteMode.java:32) at com.redhat.rhn.manager.rhnpackage.PackageManager.clearNeededPackageCache(PackageManager.java:1004) at com.redhat.rhn.manager.rhnpackage.PackageManager.deletePackages(PackageManager.java:1020) at com.redhat.rhn.frontend.action.rhnpackage.CustomPackagesDeleteConfirmAction.deletePackages(CustomPackagesDeleteConfirmAction.java:103) at com.redhat.rhn.frontend.action.rhnpackage.CustomPackagesDeleteConfirmAction.execute(CustomPackagesDeleteConfirmAction.java:80) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:237) at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:82) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:73) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142) at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685) at java.lang.Thread.run(Thread.java:636) [root@rlx-0-04 ~]#
Commit: efdafa2977d28ed0729f7d61b43d45c31ca92de3 484305 - The ChangeLogEntry id should have simply been the id value and not a composite key. The composite key can cause issues with cascading deletes (in this case from Package->ChangeLogEntry) manifesting themselves in StaleStateExceptions. The API indicates this may be an issue with unsaved-value, however since we have access to an ID value in the DB it's much easier to correct the relationship in the hibernate mapping. http://www.hibernate.org/hib_docs/v3/api/org/hibernate/StaleStateException.html
Going to reopen this and address some performance issues in this BZ. Now that the hibernate issues are addressed and this can run, now at times, I run into "java.lang.OutOfMemoryError: GC overhead limit exceeded".
Commit: 45bdda0b44113957abb494b3e92cc5675e316675 Refactored to use sets to keep the heavy lifting for package deletion taking place in the database. Using this approach instead of hibernate fixes OutOfMemoryErrors when deleting large numbers of packages in addition to the sheer performance increase.
*** Bug 487412 has been marked as a duplicate of this bug. ***
*** Bug 465492 has been marked as a duplicate of this bug. ***
Verified. Satellite-5.3.0-RHEL4-re20090414.0 x86_64 No ISE, 2500+ packages deleted in 5 mins.
What's happening here is that the connection between Tomcat and Apache is timing out. The performance fixes minimize this to a certain extent, but in very large environments it is still a possibility. To be honest, I'm a bit surprised 4000 have this error. The idea was to queue up the deletions to be addressed asynchronously. However, that queuing does still take time. It sounds like this needs a second look. Personally, I'd close this bug out and file a new one as an issue deleting large amounts of packages as this bug is centered around issues in a much wider range of environments.
I deleted 2500 packages in a while without any problem. tried Satellite-5.3.0-RHEL5-re20090820.1 but I could see timeout issue with more amount of packages created : Bug 524625 - Package deletion from GUI causes 500 error moving this bug with amount=2500 to release_pending