Hide Forgot
Description of problem: Due to a failure I encountered (see bug#784601), the katello production.log is *huge*. Further investigation shows that logrotation of katello/production.log is configured for "weekly", not a "size" value. Additionally, logrotate only runs once a day, so a value of "size" would not prevent a rapid spike in the log size (as was the case for me). Perhaps something like python's logging.handlers.RotatingFileHandler (with a value for time and maxsize) would be appropriate? Version-Release number of selected component (if applicable): * candlepin-0.5.10-1.el6.src.rpm * katello-0.1.193-2.el6.src.rpm * katello-certs-tools-1.0.2-2.el6.src.rpm * katello-cli-0.1.40-2.el6.src.rpm * katello-configure-0.1.54-2.el6.src.rpm * katello-httpd-ssl-key-pair-1.0-1.src.rpm * katello-qpid-broker-key-pair-1.0-1.src.rpm * katello-selinux-0.1.3-1.el6.src.rpm * katello-trusted-ssl-cert-1.0-1.src.rpm * pulp-0.0.256-1.el6.src.rpm How reproducible: Steps to Reproduce: 1. Fill up yor katello/production.log (see bug#784601) Actual results: # ls -lshd /var/log/katello/* 8.0K -rw-r--r--. 1 katello katello 4.5K Jan 24 10:10 /var/log/katello/delayed_job.log 4.0K -rw-r--r--. 1 katello katello 45 Jan 24 09:22 /var/log/katello/jobs-startup.log 0 lrwxrwxrwx. 1 katello katello 33 Jan 24 09:19 /var/log/katello/katello-configure -> katello-configure-20120124-091942 4.0K drwxr-xr-x. 2 katello katello 4.0K Jan 24 09:21 /var/log/katello/katello-configure-20120124-091942 0 -rw-r--r--. 1 katello root 0 Jan 24 09:21 /var/log/katello/production_delayed_jobs.log 2.9G -rw-r--r--. 1 katello root 2.9G Jan 25 04:47 /var/log/katello/production.log ^^^^----- OUCH 4.0K -rw-r--r--. 1 root root 824 Jan 24 09:48 /var/log/katello/thin-log.5000.log 4.0K -rw-r--r--. 1 root root 223 Jan 24 09:22 /var/log/katello/thin-log.5001.log 4.0K -rw-r--r--. 1 root root 223 Jan 24 09:22 /var/log/katello/thin-log.5002.log 4.0K -rw-r--r--. 1 root root 223 Jan 24 09:22 /var/log/katello/thin-log.5003.log 4.0K -rw-r--r--. 1 root root 223 Jan 24 09:22 /var/log/katello/thin-log.5004.log Expected results: Not a 2.9G single log file. Perhaps... 1) splitting the log into smaller pieces 2) move some msgs to DEBUG and disable DEBUG in production 3) Stop the sync upon the first error Additional info:
Created attachment 557464 [details] /var/log/katello/production.log (tail -n4000) Attaching a small sample of the production.log (only the last 4000 lines)
1) We already do have logrotate configured on the weekly basis: /var/log/katello/production.log /var/log/katello/production_sql.log /var/log/katello/elasticsearch.log /var/log/katello/thin-log.*.log { missingok notifempty create 0644 katello katello sharedscripts rotate 5 compress weekly postrotate [ -e /etc/init.d/katello ] && /etc/init.d/katello restart >/dev/null 2>&1 || true endscript } 2) We do not have DEBUG enabled in the default installation. It must be filling up with ERROR entries. But yes, the plan was to reconfigure Katello to use system logging facilities which should (I believe) do automatic rotation. Plus Katello does not need to be restarted. The question is if we should do this automatically, or let this on user. Not sure if we should touch syslogd configuration.
The simplest solution is to reconfigure our logrotate script - not weekly, but by size. I am setting this to 100 MB with the history of 7 files as a default value, that is 700 MB max. We will add possibility to configure against syslog later. Thanks for the report James! 500e70f 784607 - katello production.log can rapidly increase in size
(In reply to comment #3) > The simplest solution is to reconfigure our logrotate script - not weekly, but > by size. I am setting this to 100 MB with the history of 7 files as a default > value, that is 700 MB max. We will add possibility to configure against syslog > later. Hi Lukas! I'm not sure that changing the logrotate configuration to use 'size' rather a time-based interval would fix this issue. My logfile would still have grown to 2.9G before cron runs logrotate and sees the logfile has increased beyond 'size'. NOTE: ON_QA is *only* used for bugs that have fixes available in packages, and the package is available to QA for testing. Is 500e70f commited, packaged, and available for testing?
# cat /etc/logrotate.d/katello* /var/log/katello/production.log /var/log/katello/production_sql.log /var/log/katello/elasticsearch.log /var/log/katello/thin-log.*.log { size=100M missingok rotate 7 compress delaycompress notifempty copytruncate } /var/log/katello/*delayed_job*.log { size=100M missingok rotate 7 compress delaycompress notifempty copytruncate } This matches the commit code: https://fedorahosted.org/katello/changeset/500e70fbbdbcf0a5d4871cf346a33932095ef513/src
Verified on: * candlepin-0.5.18-1.el6.noarch * candlepin-tomcat6-0.5.18-1.el6.noarch * katello-0.1.229-2.el6.noarch * katello-all-0.1.229-2.el6.noarch * katello-certs-tools-1.0.2-2.el6.noarch * katello-cli-0.1.44-2.el6.noarch * katello-cli-common-0.1.44-2.el6.noarch * katello-common-0.1.229-2.el6.noarch * katello-configure-0.1.61-2.el6.noarch * katello-glue-candlepin-0.1.229-2.el6.noarch * katello-glue-foreman-0.1.229-2.el6.noarch * katello-glue-pulp-0.1.229-2.el6.noarch * katello-httpd-ssl-key-pair-1.0-1.noarch * katello-qpid-broker-key-pair-1.0-1.noarch * katello-repos-0.1.5-1.el6.noarch * katello-selinux-0.1.3-1.el6.noarch * katello-trusted-ssl-cert-1.0-1.noarch * pulp-0.0.265-1.el6.noarch * pulp-common-0.0.265-1.el6.noarch * pulp-selinux-server-0.0.265-1.el6.noarch