Bug 928217
| Summary: | Vdsm logs are filling filesystem up - logrotation of vdsm logs doesn't work correctly | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Tomas Dosek <tdosek> | |
| Component: | vdsm | Assignee: | Douglas Schilling Landgraf <dougsland> | |
| Status: | CLOSED ERRATA | QA Contact: | Artyom <alukiano> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 3.1.3 | CC: | abaron, bazulay, djuran, dornelas, dossow, hateya, iheim, italkohe, jentrena, knesenko, lpeer, lyarwood, mkalinin, obasan, pzhukov, sgrinber, sputhenp, tvvcox, ukar, van.zantvoort, ykaul, zdover | |
| Target Milestone: | --- | Keywords: | ZStream | |
| Target Release: | 3.2.0 | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | infra | |||
| Fixed In Version: | vdsm-4.10.2.19.0 | Doc Type: | Bug Fix | |
| Doc Text: |
Previously, VDSM used FileHandler to manage the logs. FileHandler was replaced with WatchedFileHandler but /var/log/vdsm/libvirt.log was not included in WatchedFileHandler.
This caused /var/log/vdsm/libvirt.log to expand endlessly, which led to cases in which the RHEV-H file system could fill with the libvirt log.
libvirt.log has been removed, eliminating any chance that the RHEV-H filesystem could be filled with it.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 951612 (view as bug list) | Environment: | ||
| Last Closed: | 2013-06-10 20:46:51 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | Infra | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 951612 | |||
Hi all, Thanks for your help I have sent a patch to vdsm upstream to add the entry copytruncate to logrotate vdsm file avoiding to keep increasing the log forever because vdsm keep it opened. Until next rhev-h version, I could suggest the following steps: # vi /etc/logrotate.d/vdsm - Add the entry: copytruncate # persist /etc/logrotate.d/vdsm Restart vdsm daemon. Cheers Douglas Hello, I have sent another patch upstream to change the rotation of logs. =============== Logrotate compress VDSM logs (/var/log/vdsm/*) when they achieve >15M. However, we rotates the logs only when it reaches 100 count. This patch purpose the rotation to 10 counts because very old logs cannot help to get out of trouble anyway. Also, reduce the number of files/space in the partition. Rotating 10 files compressed seems reasonable/safe. http://gerrit.ovirt.org/#/c/13500/ ========================= Should we use more then 10 files rotating? Thanks Douglas (In reply to comment #8) > Should we use more then 10 files rotating? Imvho we should keep a bit more than 10 files by default. Current high vdsm verbosity means I sometimes end up looking at vdsm.log.30.xz to find the info I need. (In reply to comment #9) > (In reply to comment #8) > > > Should we use more then 10 files rotating? > > Imvho we should keep a bit more than 10 files by default. > Current high vdsm verbosity means I sometimes end up looking at > vdsm.log.30.xz to find the info I need. I agree. libvirtd logs saved under /var/log directory and not under /var/log/vdsm libvirt-0.10.2-18.el6_4.4 vdsm-4.10.2-17.0.el6ev sf15 libvirtd logs saved under /var/log directory and not under /var/log/vdsm libvirt-0.10.2-18.el6_4.4 vdsm-4.10.2-18.0.el6ev sf16 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0886.html |
Description of problem: Vdsm logs are by default not properly rotated which causes the filesystem of hosts to fill up pretty easily. This is an issue especially with hosts that are configured just with disk space large enough only for hosts installation + few GB of logs (i.e. installation of host on SD card) Version-Release number of selected component (if applicable): 4.10.* How reproducible: 100 % Steps to Reproduce: 1. Have a host with heavy load on it and few hundred VMs and just 12 GB of disk space. Actual results: Host filesystem is getting filled up by vdsm logs pretty easily, vdsm log size seems to be unlimited Expected results: logrotation should work correctly and host filesystem should not fill up Additional info: [root@i-mpapp1 ~]# lsof -a +L1 /var COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME vdsm 9581 vdsm 12w REG 253,137 3725208820 0 131486 /var/log/vdsm/libvirt.log.1 (deleted) python 9601 root 4r REG 253,137 17369349 0 131698 /var/log/vdsm/vdsm.log.1 (deleted) Blamed git change: New vdsm 4.10.* packages have removed "copytruncate" from /etc/logrotate.d/vdsm: @@ -1,7 +1,6 @@ /var/log/vdsm/*.log { rotate 100 missingok - copytruncate size 15M compress compresscmd /usr/bin/xz Apparently it's not needed any more: from ChangeLog: 2012-05-18 Mark Wu <wudxw.ibm.com> Change to use WatchedFileHandler instead of FileHandler in logger.conf We find that no new log message appears in log file after manually changing the vdsm log when vdsm is running. We have to restart vdsm to get logging work again. It could happen when people try to add some markers to pinpoint the related messages during toubleshooting. It's caused by the log file gets associated with a new inode after the manual change. To make logging survive the change, use WatchedFileHandler instead of FileHandler. In addition, WatchedFileHandler was provided specially for the external log rotate tools. It can notice that the log file change performed by logrotate, so we don't need to have the option 'copytruncate' in vdsm-logrotate.conf any more. Reported-by: Changming Bai <baichm.ibm.com> In etc/vdsm/logger.conf : @@ -37,19 +37,20 @@ propagate=0 [handler_syslog] +level=WARNING class=handlers.SysLogHandler formatter=sysform -args=(('localhost', handlers.SYSLOG_UDP_PORT), handlers.SysLogHandler.LOG_USER) +args=('/dev/log', handlers.SysLogHandler.LOG_USER) [handler_logfile] -class=FileHandler +class=logging.handlers.WatchedFileHandler args=('/var/log/vdsm/vdsm.log',) filters=storage.misc.TracebackRepeatFilter level=DEBUG formatter=long [handler_metadata] -class=FileHandler +class=logging.handlers.WatchedFileHandler args=('/var/log/vdsm/metadata.log',) level=WARNING formatter=long @@ -69,7 +70,7 @@ format: %(threadName)s::%(levelname)s::%(asctime)s::%(module)s::%(lineno)d::%(name)s::(%(funcName)s) %(message)s