Description of problem:

Vdsm logs are by default not properly rotated which causes the filesystem of hosts to fill up pretty easily. 

This is an issue especially with hosts that are configured just with disk space large enough only for hosts installation + few GB of logs (i.e. installation of host on SD card)

Version-Release number of selected component (if applicable):
4.10.*

How reproducible:
100 %

Steps to Reproduce:
1. Have a host with heavy load on it and few hundred VMs and just 12 GB of disk space.
  
Actual results:
Host filesystem is getting filled up by vdsm logs pretty easily, vdsm log size seems to be unlimited

Expected results:
logrotation should work correctly and host filesystem should not fill up

Additional info:

[root@i-mpapp1 ~]# lsof -a +L1 /var
COMMAND  PID USER   FD   TYPE  DEVICE   SIZE/OFF NLINK   NODE NAME
vdsm    9581 vdsm   12w   REG 253,137 3725208820     0 131486 /var/log/vdsm/libvirt.log.1 (deleted)
python  9601 root    4r   REG 253,137   17369349     0 131698 /var/log/vdsm/vdsm.log.1 (deleted)


Blamed git change:


New vdsm 4.10.* packages have removed "copytruncate" from /etc/logrotate.d/vdsm:

@@ -1,7 +1,6 @@
 /var/log/vdsm/*.log {
     rotate 100
     missingok
-    copytruncate
     size 15M
     compress
     compresscmd /usr/bin/xz


Apparently it's not needed any more: from ChangeLog:

2012-05-18  Mark Wu  <wudxw@linux.vnet.ibm.com>

        Change to use WatchedFileHandler instead of FileHandler in logger.conf
        We find that no new log message appears in log file after manually changing
        the vdsm log when vdsm is running. We have to restart vdsm to get logging
        work again. It could happen when people try to add some markers to pinpoint
        the related messages during toubleshooting. It's caused by the log file gets
        associated with a new inode after the manual change. To make logging survive
        the change, use WatchedFileHandler instead of FileHandler.

        In addition, WatchedFileHandler was provided specially for the external
        log rotate tools. It can notice that the log file change performed by
        logrotate, so we don't need to have the option 'copytruncate' in
        vdsm-logrotate.conf any more.

        Reported-by: Changming Bai <baichm@linux.vnet.ibm.com>


In etc/vdsm/logger.conf :
@@ -37,19 +37,20 @@
 propagate=0
 
 [handler_syslog]
+level=WARNING
 class=handlers.SysLogHandler
 formatter=sysform
-args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER)
+args=('/dev/log', handlers.SysLogHandler.LOG_USER)
 
 [handler_logfile]
-class=FileHandler
+class=logging.handlers.WatchedFileHandler
 args=('/var/log/vdsm/vdsm.log',)
 filters=storage.misc.TracebackRepeatFilter
 level=DEBUG
 formatter=long
 
 [handler_metadata]
-class=FileHandler
+class=logging.handlers.WatchedFileHandler
 args=('/var/log/vdsm/metadata.log',)
 level=WARNING
 formatter=long
@@ -69,7 +70,7 @@
 format: %(threadName)s::%(levelname)s::%(asctime)s::%(module)s::%(lineno)d::%(name)s::(%(funcName)s) %(message)s