Bug 784607

Created attachment 557464 [details]
/var/log/katello/production.log (tail -n4000)

Attaching a small sample of the production.log (only the last 4000 lines)

1) We already do have logrotate configured on the weekly basis:

/var/log/katello/production.log /var/log/katello/production_sql.log /var/log/katello/elasticsearch.log /var/log/katello/thin-log.*.log {
  missingok
  notifempty
  create 0644 katello katello
  sharedscripts
  rotate 5
  compress
  weekly
  postrotate
    [ -e /etc/init.d/katello ] && /etc/init.d/katello restart >/dev/null 2>&1 || true
  endscript
}

2) We do not have DEBUG enabled in the default installation. It must be filling up with ERROR entries.

But yes, the plan was to reconfigure Katello to use system logging facilities which should (I believe) do automatic rotation. Plus Katello does not need to be restarted. The question is if we should do this automatically, or let this on user. Not sure if we should touch syslogd configuration.

The simplest solution is to reconfigure our logrotate script - not weekly, but by size. I am setting this to 100 MB with the history of 7 files as a default value, that is 700 MB max. We will add possibility to configure against syslog later.

Thanks for the report James!

500e70f 784607 - katello production.log can rapidly increase in size

(In reply to comment #3)
> The simplest solution is to reconfigure our logrotate script - not weekly, but
> by size. I am setting this to 100 MB with the history of 7 files as a default
> value, that is 700 MB max. We will add possibility to configure against syslog
> later.

Hi Lukas!  I'm not sure that changing the logrotate configuration to use 'size' rather a time-based interval would fix this issue.  My logfile would still have grown to 2.9G before cron runs logrotate and sees the logfile has increased beyond 'size'.

NOTE: ON_QA is *only* used for bugs that have fixes available in packages, and the package is available to QA for testing.  Is 500e70f commited, packaged, and available for testing?

# cat /etc/logrotate.d/katello*
/var/log/katello/production.log /var/log/katello/production_sql.log /var/log/katello/elasticsearch.log /var/log/katello/thin-log.*.log {
  size=100M
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}
/var/log/katello/*delayed_job*.log {
  size=100M
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}

This matches the commit code:

https://fedorahosted.org/katello/changeset/500e70fbbdbcf0a5d4871cf346a33932095ef513/src

Verified on:
* candlepin-0.5.18-1.el6.noarch
* candlepin-tomcat6-0.5.18-1.el6.noarch
* katello-0.1.229-2.el6.noarch
* katello-all-0.1.229-2.el6.noarch
* katello-certs-tools-1.0.2-2.el6.noarch
* katello-cli-0.1.44-2.el6.noarch
* katello-cli-common-0.1.44-2.el6.noarch
* katello-common-0.1.229-2.el6.noarch
* katello-configure-0.1.61-2.el6.noarch
* katello-glue-candlepin-0.1.229-2.el6.noarch
* katello-glue-foreman-0.1.229-2.el6.noarch
* katello-glue-pulp-0.1.229-2.el6.noarch
* katello-httpd-ssl-key-pair-1.0-1.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-repos-0.1.5-1.el6.noarch
* katello-selinux-0.1.3-1.el6.noarch
* katello-trusted-ssl-cert-1.0-1.noarch
* pulp-0.0.265-1.el6.noarch
* pulp-common-0.0.265-1.el6.noarch
* pulp-selinux-server-0.0.265-1.el6.noarch