Description of problem: RHEV-H is failing to boot the HostVG/Logging volume, and the following is repeated in audit_log every 15 minutes: Critical, Low disk space. Host <hostname> has less than 500 MB of free space left on: /var/log. Version-Release number of selected component (if applicable): 20121212.0.el6_3 How reproducible: Unknown, but has been encountered by several customers Steps to Reproduce: 1. Install RHEV-H 2. Run RHEV-H for an unknown period Actual results: HostVG/Logging fails to mount on boot Expected results: HostVG/Logging mounted on boot normally Additional info: It's especially difficult to troubleshoot this problem because the logs are wiped out on the next hypervisor reboot
This issue was a bug in how vdsm was doing logrotate. It's keeping a deleted log open in the background. If you look at df -h /var/log, you'll see that the available size is 0. vdsmd restart should fix this, as would a reboot of the machine. Upgrading to a more recent version will also fix this. On an example host: #lsof /var/log | grep deleted vdsm 13039 vdsm 12w REG 253,11 1594187776 293 /var/log/vdsm/libvirt.log.1 (deleted) I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm in the /var/log/vdsm/*.log stanza. A vdsmd restart would probably be needed to free up disk space immediately, though
(In reply to Mike Burns from comment #5) > This issue was a bug in how vdsm was doing logrotate. It's keeping a > deleted log open in the background. > > If you look at df -h /var/log, you'll see that the available size is 0. > > vdsmd restart should fix this, as would a reboot of the machine. > > Upgrading to a more recent version will also fix this. > > On an example host: > > #lsof /var/log | grep deleted > vdsm 13039 vdsm 12w REG 253,11 1594187776 293 > /var/log/vdsm/libvirt.log.1 (deleted) > > I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm > in the /var/log/vdsm/*.log stanza. A vdsmd restart would probably be > needed to free up disk space immediately, though yes, after restart vdsmd, it returns to normal: [root@dell-per810-01 admin]# lsof /var/log | grep delete [root@dell-per810-01 admin]# df -h /var/log Filesystem Size Used Avail Use% Mounted on - 2.0G 409M 1.5G 22% /var/log and did not see low disk error. Assign the bug to vdsm to add the fix.
Douglas - can you please try reproduce ?
Hello Ouyang, Based on your comment#6 (size of libvirt.log.1 which tracked all vdsm calls to libvirt) I am going to close this bugzilla as duplicates the following: Vdsm logs are filling filesystem up - logrotation of vdsm logs doesn't work correctly: https://bugzilla.redhat.com/show_bug.cgi?id=928217 https://bugzilla.redhat.com/show_bug.cgi?id=951612 (cloned version for z-stream) Fell free to re-open this bugzilla if with updated vdsm (at least vdsm-4.10.2.19.0) the report can be reproduced. Thanks Douglas *** This bug has been marked as a duplicate of bug 951612 ***