Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
After an upgrade to 6.8, users might see some Katello tasks (like a publish) hang when talking to Candlepin (SetContent for publishes) and Candlepin logs:
Usage rate is 91.2% which is beyond the configured <max-disk-usage>. System will start blocking producers.
This effectively makes Satellite unusable, especially with very big disks for /var where this check triggers way to early.
Not entirely sure this needs a fix in Candlepin or in the Installer (which deploys the Candlepin configuration for Satellite), so filing in Candlepin for now.
+++ This bug was initially created as a clone of Bug #1898605 +++
Description of problem:
Upgrading to foreman-2.1.3 we experienced hanging tasks on some of our test-systems.
Looking into the logs Candlepin did no longer process tasks due to too much disk usage (more than 90%):
WARN org.apache.activemq.artemis.core.server - AMQ222210: Free storage space is at 40.7GB of 644.2GB total. Usage rate is 93.7% which is beyond the configured <max-disk-usage>. System will start blocking producers.
WARN org.apache.activemq.artemis.core.client - AMQ212054: Destination address=event.default is blocked. If the system is configured to block make sure you consume messages on this configuration.
Looks like this is a default setting of the Apache Artemis server.
However, I doubt the necessity of doing this by disk-usage percentage, because this would mean that in the above example 60GB are not enough for Candlepin to function.
Solution would be to explicitly set this to sane value in MB or GB rather than a percentage, unless there is a good reason for that.
See also https://projects.theforeman.org/issues/31210
--- Additional comment from Nikos Moumoulidis on 2020-11-20 14:01:30 UTC ---
(In reply to Markus Bucher from comment #0)
> WARN org.apache.activemq.artemis.core.server - AMQ222210: Free storage space
> is at 40.7GB of 644.2GB total. Usage rate is 93.7% which is beyond the
> configured <max-disk-usage>. System will start blocking producers.
Hi, can you please provide the broker.xml that is used? That is located at either:
1) Whatever candlepin.audit.hornetq.config_path= is pointing to in /etc/candlepin/candlepin.conf
or, in case that property is not set
2) Whatever is shipped in candlepin war, at /var/lib/tomcat/webapps/candlepin/WEB-INF/classes/broker.xml
The exact version of candlepin used and the candlepin.log would also be useful.
--- Additional comment from Markus Bucher on 2020-11-20 14:25:44 UTC ---
--- Additional comment from Markus Bucher on 2020-11-20 14:26:23 UTC ---
--- Additional comment from Markus Bucher on 2020-11-20 14:28:30 UTC ---
# rpm -q candlepin
candlepin-3.1.16-1.el7.noarch