Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1847515

Summary:	cleanup ActiveMQ journal via foreman-installer when candlepin refuses to start when having too many messages on filesystem
Product:	Red Hat Satellite	Reporter:	Jonathon Turel <jturel>
Component:	Installation	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Devendra Singh <desingh>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.7.0	CC:	desingh, dsulliva, ehelms, mmccune, nmoumoul, pcreech, wpinheir, zhunting
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-04-22 18:46:14 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jonathon Turel 2020-06-16 14:24:48 UTC

This bug was initially created as a copy of Bug #1667952

I am copying this bug because: 



+++ This bug was initially created as a clone of Bug #1665893 +++

Description of problem:
When candlepin is unable to send further events to qpidd (e.g. due to katello_event_queue full), it stores further events in /var/lib/candlepin/activemq-artemis/journal . Once this journal becomes "too full", an attempt to start tomcat/candlepin fails - regardless that candlepin newly *can* send messages to the qpidd broker.

This completely prevents starting candlepin, until one purges away the journal content.


Version-Release number of selected component (if applicable):
Sat 6.4
candlepin-2.4.8-1.el7.noarch


How reproducible:
100%


Steps to Reproduce:
1. Either have Satellite6.4 installed and mimic LOCE task problems such that katello_event_queue grows over time - and patiently generate more and more events in candlepin, even when it will be already in suspended mode.

2. Or use prepared journal content (tarball will follow) for a fast reproducer.
3. ensure candlepin can properly send messages to qpidd (again)
4. restart tomcat and wait approx. 3 minutes.


Actual results:
4. candlepin fails to start, logging a huge thread dump to error.log (details in additional info)


Expected results:
4. candlepin can start any time.


Additional info:
error.log having thread dump:

==> /var/log/candlepin/error.log <==
2019-01-14 12:28:14,415 [thread=Thread-0 (-scheduled-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222199: Thread dump: *******************************************************************************
Complete Thread dump 
"Thread-4 (ActiveMQ-scheduled-threads)" Id=63 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-3 (ActiveMQ-scheduled-threads)" Id=62 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-2 (ActiveMQ-scheduled-threads)" Id=61 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-1 (ActiveMQ-scheduled-threads)" Id=60 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-0 (ActiveMQ-scheduled-threads)" Id=57 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"activemq-buffer-timeout" Id=56 WAITING on java.util.concurrent.Semaphore$NonfairSync@3902672a
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.Semaphore$NonfairSync@3902672a
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:427)
	at java.lang.Thread.run(Thread.java:748)


"Thread-0 (-scheduled-threads)" Id=55 RUNNABLE
	at sun.management.ThreadImpl.dumpThreads0(Native Method)
	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
	at org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:929)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$2(ActiveMQServerImpl.java:642)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$13/1979890430.run(Unknown Source)
	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireAction(CriticalAnalyzerImpl.java:145)
	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:130)
	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53)
	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284)
	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$3.run(ActiveMQScheduledComponent.java:294)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@4a914d3d


"pool-2-thread-1" Id=52 RUNNABLE
	at java.io.FileInputStream.readBytes(Native Method)
	at java.io.FileInputStream.read(FileInputStream.java:255)
	at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424)
	at sun.security.provider.NativePRNG$RandomIO.ensureBufferValid(NativePRNG.java:525)
	at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:544)
	-  locked java.lang.Object@ba508d9
	at sun.security.provider.NativePRNG$RandomIO.access$400(NativePRNG.java:331)
	at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:220)
	at java.security.SecureRandom.nextBytes(SecureRandom.java:468)
	at java.security.SecureRandom.next(SecureRandom.java:491)
	at java.util.Random.nextInt(Random.java:329)
	at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:101)
	at javax.net.ssl.SSLContext.init(SSLContext.java:282)
	at org.apache.qpid.ssl.SSLContextFactory.buildContext(SSLContextFactory.java:131)
	at org.apache.qpid.ssl.SSLContextFactory.buildClientContext(SSLContextFactory.java:62)
	at org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.<init>(SecurityLayerFactory.java:79)
	at org.apache.qpid.transport.network.security.SecurityLayerFactory.newInstance(SecurityLayerFactory.java:52)
	at org.apache.qpid.transport.Connection.connect(Connection.java:233)
	-  locked java.lang.Object@8f1bb59
	at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:222)
	at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:664)
	at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:444)
	at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
	at org.candlepin.audit.QpidConnection.newConnection(QpidConnection.java:170)
	at org.candlepin.audit.QpidQmf.runQuery(QpidQmf.java:101)
	at org.candlepin.audit.QpidQmf.getExchangeBoundQueueNames(QpidQmf.java:220)
	at org.candlepin.audit.QpidQmf.getStatus(QpidQmf.java:178)
	at org.candlepin.controller.QpidStatusMonitor.monitor(QpidStatusMonitor.java:124)
	-  locked org.candlepin.controller.QpidStatusMonitor@73bc8a0f
	at org.candlepin.controller.QpidStatusMonitor.run(QpidStatusMonitor.java:86)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@3db5f5a6


"IoReceiver - localhost/127.0.0.1:5671" Id=50 RUNNABLE (in native)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:156)
	at java.lang.Thread.run(Thread.java:748)


"IoSender - localhost/127.0.0.1:5671" Id=51 WAITING on java.lang.Object@592d9865
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@592d9865
	at java.lang.Object.wait(Object.java:502)
	at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:279)
	at java.lang.Thread.run(Thread.java:748)


"ack-flusher" Id=45 TIMED_WAITING on java.util.TaskQueue@7aedb059
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@7aedb059
	at java.util.TimerThread.mainLoop(Timer.java:552)
	at java.util.TimerThread.run(Timer.java:505)


"QuartzScheduler_QuartzSchedulerThread" Id=42 TIMED_WAITING on java.lang.Object@7f43e72e
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@7f43e72e
	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:253)


"QuartzScheduler_Worker-15" Id=41 TIMED_WAITING on java.lang.Object@68b8e63b
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@68b8e63b
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-14" Id=40 TIMED_WAITING on java.lang.Object@5d63a5f9
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@5d63a5f9
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-13" Id=39 TIMED_WAITING on java.lang.Object@4a1ba974
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@4a1ba974
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-12" Id=38 TIMED_WAITING on java.lang.Object@2a815d8e
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@2a815d8e
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-11" Id=37 TIMED_WAITING on java.lang.Object@527efdb
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@527efdb
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-10" Id=36 TIMED_WAITING on java.lang.Object@45abceeb
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@45abceeb
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-9" Id=35 TIMED_WAITING on java.lang.Object@52ec8f23
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@52ec8f23
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-8" Id=34 TIMED_WAITING on java.lang.Object@72a1d5dd
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@72a1d5dd
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-7" Id=33 TIMED_WAITING on java.lang.Object@48c2a565
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@48c2a565
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-6" Id=32 TIMED_WAITING on java.lang.Object@5385f600
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@5385f600
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-5" Id=31 TIMED_WAITING on java.lang.Object@65a6fec4
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@65a6fec4
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-4" Id=30 TIMED_WAITING on java.lang.Object@1955d090
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@1955d090
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-3" Id=29 TIMED_WAITING on java.lang.Object@4461b752
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@4461b752
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-2" Id=28 TIMED_WAITING on java.lang.Object@38286c28
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@38286c28
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-1" Id=27 TIMED_WAITING on java.lang.Object@6aaaf537
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@6aaaf537
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"Statistics Thread-__DEFAULT__-1" Id=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5dcc79a2
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5dcc79a2
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"__DEFAULT__" Id=24 WAITING on java.util.TaskQueue@14de9e6e
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@14de9e6e
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)


"C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-HelperThread-#2" Id=23 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
	at java.lang.Object.wait(Native Method)
	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


"C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-HelperThread-#1" Id=22 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
	at java.lang.Object.wait(Native Method)
	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


"C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-HelperThread-#0" Id=21 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
	at java.lang.Object.wait(Native Method)
	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


"C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-AdminTaskTimer" Id=20 TIMED_WAITING on java.util.TaskQueue@5252aa66
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@5252aa66
	at java.util.TimerThread.mainLoop(Timer.java:552)
	at java.util.TimerThread.run(Timer.java:505)


"Statistics Thread-file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
  context: /candlepin
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
java.net.URLClassLoader@7a81197d
::org.ehcache.jcache.JCacheCachingProvider@2bdd47be-1" Id=17 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cedd8e7
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cedd8e7
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
  context: /candlepin
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
java.net.URLClassLoader@7a81197d
::org.ehcache.jcache.JCacheCachingProvider@2bdd47be" Id=16 WAITING on java.util.TaskQueue@c2016fa
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@c2016fa
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)


"com.google.inject.internal.util.$Finalizer" Id=15 WAITING on java.lang.ref.ReferenceQueue$Lock@40bd5fe
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.ref.ReferenceQueue$Lock@40bd5fe
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)


"localhost-startStop-1" Id=14 RUNNABLE
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl$11.checkDeleteSize(JournalImpl.java:1406)
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl$11.deleteRecord(JournalImpl.java:1435)
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl$13.onReadCommitRecord(JournalImpl.java:1932)
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:729)
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1753)
	-  locked org.apache.activemq.artemis.core.journal.impl.JournalImpl@434d4e83
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1390)
	-  locked org.apache.activemq.artemis.core.journal.impl.JournalImpl@434d4e83
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1374)
	at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:846)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2573)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2336)
	-  locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@41658ced
	at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:71)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:543)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:482)
	-  locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@41658ced
	at org.apache.activemq.artemis.core.server.embedded.EmbeddedActiveMQ.start(EmbeddedActiveMQ.java:115)
	at org.candlepin.audit.ActiveMQContextListener.contextInitialized(ActiveMQContextListener.java:159)
	at org.candlepin.guice.CandlepinContextListener.withInjector(CandlepinContextListener.java:182)
	at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:57)
	at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:144)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
	-  locked org.apache.catalina.core.StandardContext@6939ae95
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardContext@6939ae95
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@5541d560


"Catalina-startStop-1" Id=13 WAITING on java.util.concurrent.FutureTask@4d7ed5fd
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.FutureTask@4d7ed5fd
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1163)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:497)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1713)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:337)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:388)
	-  locked org.apache.catalina.core.StandardHost@26024709
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:333)
	-  locked org.apache.catalina.core.StandardHost@26024709
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1136)
	-  locked org.apache.catalina.core.StandardHost@26024709
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:819)
	-  locked org.apache.catalina.core.StandardHost@26024709
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardHost@26024709
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@42dafa95


"GC Daemon" Id=12 TIMED_WAITING on sun.misc.GC$LatencyLock@17804d89
	at java.lang.Object.wait(Native Method)
	-  waiting on sun.misc.GC$LatencyLock@17804d89
	at sun.misc.GC$Daemon.run(GC.java:117)


"Signal Dispatcher" Id=4 RUNNABLE


"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3e9030ac
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.ref.ReferenceQueue$Lock@3e9030ac
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)


"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@53ffc177
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.ref.Reference$Lock@53ffc177
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)


"main" Id=1 WAITING on java.util.concurrent.FutureTask@6f9f0c51
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.FutureTask@6f9f0c51
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1119)
	-  locked org.apache.catalina.core.StandardEngine@681537c8
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:300)
	-  locked org.apache.catalina.core.StandardEngine@681537c8
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardEngine@681537c8
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:444)
	-  locked org.apache.catalina.core.StandardEngine@681537c8
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardService@7d6cc346
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:758)
	-  locked java.lang.Object@42468917
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardServer@3e6fa38a
	at org.apache.catalina.startup.Catalina.start(Catalina.java:693)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:428)


===============================================================================
End Thread dump 
*******************************************************************************

--- Additional comment from Red Hat Bugzilla Rules Engine on 2019-01-14 11:42:16 UTC ---

Since this bug report was entered in Red Hat Bugzilla, the 'sat-backlog' flag has been set to ? to ensure that it is properly evaluated for release.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2019-01-14 11:42:16 UTC ---

Since this issue was entered in Red Hat Bugzilla, the pm_ack has been set to + automatically for the next planned release.

--- Additional comment from Pavel Moravec on 2019-01-14 11:49:44 UTC ---

The journal dir content is being copied to https://provisioning.usersys.redhat.com/pub/activemq-data-huge.tar.gz

To use it:

service tomcat stop
rm -rf /var/lib/candlepin/activemq-artemis/journal
wget https://provisioning.usersys.redhat.com/pub/activemq-data-huge.tar.gz
tar xzf activemq-data-huge.tar.gz
cp journal-backup.tgz/var/lib/candlepin/activemq-artemis/journal-backup/* /var/lib/candlepin/activemq-artemis/journal/
chown tomcat:tomcat /var/lib/candlepin/activemq-artemis/journal/*
service tomcat start

--- Additional comment from Kevin Howell on 2019-01-14 15:06:17 UTC ---

Have you seen this occur in real-world scenarios? If so, can you describe the conditions where this occurs?

--- Additional comment from Pavel Moravec on 2019-01-14 15:09:52 UTC ---

(In reply to Kevin Howell from comment #4)
> Have you seen this occur in real-world scenarios? If so, can you describe
> the conditions where this occurs?

Yes. The scenario from point 1. (katello_event_queue gets full, candlepin to suspended mode as it cant send more msgs to qpidd but somehow generating more events/messages and storing inside the internal broker) was observed at customer. And resulted in the huge journal directory I provided.

Backing up the journal dir (to the copy I shared) was resolution for them.

Comment 1 Jonathon Turel 2020-06-16 14:26:30 UTC

Connecting redmine issue https://projects.theforeman.org/issues/30135 from this bug

Comment 3 Bryan Kearney 2020-06-16 16:02:04 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/30135 has been resolved.

Comment 4 Brad Buckingham 2020-06-18 14:24:57 UTC

The change provided by this bugzilla is to the Installer to ensure that the event queue is cleaned up.  There is a corresponding change in candlepin that is tracked by bug 1665893.

Comment 15 Devendra Singh 2020-10-07 15:01:22 UTC

Failed QA 

Tested on 6.8.0 Snap17

# less candlepin.log|grep "AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings"
2020-10-02 01:25:37,765 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings
2020-10-02 01:29:19,049 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings

# less candlepin.log|grep "AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings"
2020-10-02 01:25:37,765 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings
2020-10-02 01:29:19,048 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings

Comment 33 Eric Helms 2022-04-22 18:46:14 UTC

Given the age of the bug, and the associated issue having been closed and released already in a Satellite release. I am opting to close current release.