Description of problem: ============================ Running a appliance for 4 days, no log rotate... [root@host-192-168-55-149 log]# ll total 277048 drwxr-xr-x. 2 root root 90 Sep 17 05:27 apache -rw-r--r--. 1 root root 66 Sep 16 10:34 api.log -rw-r--r--. 1 root root 590 Sep 17 05:26 appliance_console.log -rw-r--r--. 1 root root 15499 Sep 22 21:14 audit.log -rw-r--r--. 1 root root 188511 Sep 22 21:27 automation.log -rw-r--r--. 1 root root 2062 Sep 22 20:19 aws.log -rw-r--r--. 1 root root 119570860 Sep 22 21:31 evm.log -rw-r--r--. 1 root root 66 Sep 16 10:34 fog.log -rw-r--r--. 1 root root 3961 Sep 18 11:52 gem_list.txt -rw-r--r--. 1 root root 66 Sep 16 10:34 kubernetes.log -rw-r--r--. 1 root root 60584 Sep 18 11:52 last_startup.txt -rw-r--r--. 1 root root 471550 Sep 22 21:30 miq_ntpdate.log -rw-r--r--. 1 root root 18006 Sep 18 11:52 package_list_rpm.txt -rw-r--r--. 1 root root 288745 Sep 22 21:28 policy.log -rw-r--r--. 1 root root 1082210 Sep 22 21:29 production.log -rw-r--r--. 1 root root 1867793 Sep 22 21:31 rhevm.log -rw-r--r--. 1 root root 66 Sep 16 10:34 scvmm.log -rw-r--r--. 1 root root 121190230 Sep 22 21:30 top_output.log -rw-r--r--. 1 root root 66 Sep 16 10:34 vim.log -rw-r--r--. 1 root root 527737 Sep 22 21:30 vmstat_output.log [root@host-192-168-55-149 log]# uptime 21:31:32 up 4 days, 9:48, 1 user, load average: 0.42, 0.67, 0.95 Version-Release number of selected component (if applicable): ==================================== 5.5.0.1 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
https://github.com/ManageIQ/manageiq-appliance/pull/25
Additional Details: Log path: /var/www/miq/vmdb/log Check mask: {*.log}-{yyyymmdd}.gz
[root@s-appl-downstream-55z-151021-oyvllkg1 log]# ll total 1955504 drwxr-xr-x. 2 root root 90 Oct 22 08:23 apache -rw-r--r--. 1 root root 66 Oct 21 15:49 api.log -rw-r--r--. 1 root root 594 Oct 22 08:20 appliance_console.log -rw-r--r--. 1 root root 5055 Nov 2 09:18 audit.log -rw-r--r--. 1 root root 226521044 Nov 2 09:54 automation.log -rw-r--r--. 1 root root 66 Oct 21 15:49 aws.log -rw-r--r--. 1 root root 1236876887 Nov 2 09:55 evm.log -rw-r--r--. 1 root root 66 Oct 21 15:49 fog.log -rw-r--r--. 1 root root 8754 Oct 26 09:07 gem_list.txt -rw-r--r--. 1 root root 66 Oct 21 15:49 kubernetes.log -rw-r--r--. 1 root root 60920 Oct 26 09:06 last_startup.txt -rw-r--r--. 1 root root 755129 Nov 2 09:45 miq_ntpdate.log -rw-r--r--. 1 root root 90 Oct 26 09:07 package_list_rpm.txt -rw-r--r--. 1 root root 47033 Nov 2 09:08 policy.log -rw-r--r--. 1 root root 568876 Nov 2 09:18 production.log -rw-r--r--. 1 root root 66 Oct 21 15:49 rhevm.log -rw-r--r--. 1 root root 66 Oct 21 15:49 scvmm.log -rw-r--r--. 1 root root 238695700 Nov 2 09:55 top_output.log -rw-r--r--. 1 root root 66 Oct 21 15:49 vim.log -rw-r--r--. 1 root root 857278 Nov 2 09:55 vmstat_output.log [root@s-appl-downstream-55z-151021-oyvllkg1 log]# uptime 09:56:04 up 7 days, 1:52, 1 user, load average: 0.13, 0.19, 0.23
Can you check /var/www/miq/vmdb/appliance_console.log for any messages relating to log rotation? Such as: "Rotating file..." or "Skipping file, Not enough free space on disk." If we had an issue with checking the free disk space it would be there.
The test appliance from above is no longer available. I have checked on a different appliance. No file exist at /var/www/miq/vmdb/appliance_console.log [root@host-192-168-55-119 log]# pwd /var/www/miq/vmdb/log [root@host-192-168-55-119 log]# ll total 416800 drwxr-xr-x. 2 root root 90 Oct 29 05:38 apache -rw-r--r--. 1 root root 66 Oct 27 18:14 api.log -rw-r--r--. 1 root root 590 Oct 29 05:35 appliance_console.log -rw-r--r--. 1 root root 32788 Nov 1 23:35 audit.log -rw-r--r--. 1 root root 35822268 Nov 2 10:40 automation.log -rw-r--r--. 1 root root 22432 Nov 2 08:52 aws.log -rw-r--r--. 1 root root 231470492 Nov 2 10:43 evm.log -rw-r--r--. 1 root root 66 Oct 27 18:14 fog.log -rw-r--r--. 1 root root 8878 Oct 31 09:36 gem_list.txt -rw-r--r--. 1 root root 66 Oct 27 18:14 kubernetes.log -rw-r--r--. 1 root root 61075 Oct 31 09:36 last_startup.txt -rw-r--r--. 1 root root 224603 Nov 2 10:30 miq_ntpdate.log -rw-r--r--. 1 root root 90 Oct 31 09:36 package_list_rpm.txt -rw-r--r--. 1 root root 1941787 Nov 1 16:47 policy.log -rw-r--r--. 1 root root 13422098 Nov 2 10:42 production.log -rw-r--r--. 1 root root 30510228 Nov 2 10:43 rhevm.log -rw-r--r--. 1 root root 66 Oct 27 18:14 scvmm.log -rw-r--r--. 1 root root 68869312 Nov 2 10:43 top_output.log -rw-r--r--. 1 root root 66 Oct 27 18:14 vim.log -rw-r--r--. 1 root root 253912 Nov 2 10:42 vmstat_output.log [root@host-192-168-55-119 log]# cat appliance_console.log # Logfile created on 2015-10-29 05:31:57 -0400 by logger.rb/47272 I, [2015-10-29T05:31:57.527478 #6800] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#initialize_postgresql) : starting I, [2015-10-29T05:32:38.743973 #6800] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#initialize_postgresql) : complete I, [2015-10-29T05:32:39.108997 #6800] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#create_region) : starting I, [2015-10-29T05:35:36.378738 #6800] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#create_region) : complete [root@host-192-168-55-119 log]# uptime 10:43:28 up 2 days, 2:08, 1 user, load average: 0.34, 0.37, 0.34 [root@host-192-168-55-119 log]#
https://github.com/ManageIQ/manageiq-appliance/pull/43
New commit detected on ManageIQ/manageiq-appliance/master: https://github.com/ManageIQ/manageiq-appliance/commit/e8a7495f3617ad553df9eb1436edd84d7f41ed7f commit e8a7495f3617ad553df9eb1436edd84d7f41ed7f Author: Nick Carboni <ncarboni> AuthorDate: Tue Nov 3 11:14:17 2015 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Tue Nov 3 11:14:17 2015 -0500 Run pre-rotate script even if source command fails There are some things that the various files sourced by /etc/default/evm do that SELinux will prevent based on the context logrotate runs under. Because of this we need to source the file in order to set our environment variables and run the pre-rotate script regardless of the exit status of the source command. https://bugzilla.redhat.com/show_bug.cgi?id=1265463 COPY/etc/logrotate.d/miq_logs.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
MR: http://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme-appliance/merge_requests/2
Version: 5.5.0.9-beta2.20151102161742_5530c9a [root@10-16-7-64 log]# pwd /var/www/miq/vmdb/log [root@10-16-7-64 log]# uptime 09:23:37 up 3 days, 23:41, 1 user, load average: 0.23, 0.22, 0.18 [root@10-16-7-64 log]# ll total 214412 drwxr-xr-x. 2 root root 90 Nov 3 20:20 apache -rw-r--r--. 1 root root 66 Nov 3 11:21 api.log -rw-r--r--. 1 root root 594 Nov 3 20:15 appliance_console.log -rw-r--r--. 1 root root 11483 Nov 3 20:16 audit.log -rw-r--r--. 1 root root 32263 Nov 5 09:46 automation.log -rw-r--r--. 1 root root 66 Nov 3 11:21 aws.log -rw-r--r--. 1 root root 100920509 Nov 9 09:23 evm.log -rw-r--r--. 1 root root 66 Nov 3 11:21 fog.log -rw-r--r--. 1 root root 9024 Nov 5 09:44 gem_list.txt -rw-r--r--. 1 root root 66 Nov 3 11:21 kubernetes.log -rw-r--r--. 1 root root 60127 Nov 5 09:44 last_startup.txt -rw-r--r--. 1 root root 1039 Nov 5 09:46 miq_ntpdate.log -rw-r--r--. 1 root root 90 Nov 5 09:44 package_list_rpm.txt -rw-r--r--. 1 root root 1607 Nov 5 09:46 policy.log -rw-r--r--. 1 root root 35038 Nov 5 09:46 production.log -rw-r--r--. 1 root root 66 Nov 3 11:21 rhevm.log -rw-r--r--. 1 root root 66 Nov 3 11:21 scvmm.log -rw-r--r--. 1 root root 115864294 Nov 9 09:23 top_output.log -rw-r--r--. 1 root root 66 Nov 3 11:21 vim.log -rw-r--r--. 1 root root 483551 Nov 9 09:23 vmstat_output.log [root@10-16-7-64 log]# cat appliance_console.log # Logfile created on 2015-11-03 20:14:11 -0500 by logger.rb/47272 I, [2015-11-03T20:14:11.669854 #10379] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#initialize_postgresql) : starting I, [2015-11-03T20:14:26.919472 #10379] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#initialize_postgresql) : complete I, [2015-11-03T20:14:27.442070 #10379] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#create_region) : starting I, [2015-11-03T20:15:31.481287 #10379] INFO -- : MIQ(ApplianceConsole::InternalDatabaseConfiguration#create_region) : complete
The build used for the test was from before the fix was included. When the bug gets included in a build the "Fixed In Version" field will contain the build version to use for testing this issue.
Tested: 5.5.0.9-beta2.20151102161742_5530c9a
Tested Version: 5.5.0.10-beta2.1.20151110134042_d6f5459 Passed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:2551