Description of problem:
Upgrading to foreman-2.1.3 we experienced hanging tasks on some of our test-systems.
Looking into the logs Candlepin did no longer process tasks due to too much disk usage (more than 90%):
WARN org.apache.activemq.artemis.core.server - AMQ222210: Free storage space is at 40.7GB of 644.2GB total. Usage rate is 93.7% which is beyond the configured <max-disk-usage>. System will start blocking producers.
WARN org.apache.activemq.artemis.core.client - AMQ212054: Destination address=event.default is blocked. If the system is configured to block make sure you consume messages on this configuration.
Looks like this is a default setting of the Apache Artemis server.
However, I doubt the necessity of doing this by disk-usage percentage, because this would mean that in the above example 60GB are not enough for Candlepin to function.
Solution would be to explicitly set this to sane value in MB or GB rather than a percentage, unless there is a good reason for that.
See also https://projects.theforeman.org/issues/31210
Comment 1Nikos Moumoulidis
2020-11-20 14:01:30 UTC
(In reply to Markus Bucher from comment #0)
> WARN org.apache.activemq.artemis.core.server - AMQ222210: Free storage space
> is at 40.7GB of 644.2GB total. Usage rate is 93.7% which is beyond the
> configured <max-disk-usage>. System will start blocking producers.
Hi, can you please provide the broker.xml that is used? That is located at either:
1) Whatever candlepin.audit.hornetq.config_path= is pointing to in /etc/candlepin/candlepin.conf
or, in case that property is not set
2) Whatever is shipped in candlepin war, at /var/lib/tomcat/webapps/candlepin/WEB-INF/classes/broker.xml
The exact version of candlepin used and the candlepin.log would also be useful.