Red Hat Bugzilla – Bug 971552
HostVG/Logging volume is not mounted on RHEV-H, and hypervisor reports host has less than 500MB free space remaining
Last modified: 2016-02-10 14:40:01 EST
Description of problem:
RHEV-H is failing to boot the HostVG/Logging volume, and the following is repeated in audit_log every 15 minutes:
Critical, Low disk space. Host <hostname> has less than 500 MB of free space left on: /var/log.
Version-Release number of selected component (if applicable):
Unknown, but has been encountered by several customers
Steps to Reproduce:
1. Install RHEV-H
2. Run RHEV-H for an unknown period
HostVG/Logging fails to mount on boot
HostVG/Logging mounted on boot normally
It's especially difficult to troubleshoot this problem because the logs are wiped out on the next hypervisor reboot
This issue was a bug in how vdsm was doing logrotate. It's keeping a deleted log open in the background.
If you look at df -h /var/log, you'll see that the available size is 0.
vdsmd restart should fix this, as would a reboot of the machine.
Upgrading to a more recent version will also fix this.
On an example host:
#lsof /var/log | grep deleted
vdsm 13039 vdsm 12w REG 253,11 1594187776 293 /var/log/vdsm/libvirt.log.1 (deleted)
I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm in the /var/log/vdsm/*.log stanza. A vdsmd restart would probably be needed to free up disk space immediately, though
(In reply to Mike Burns from comment #5)
> This issue was a bug in how vdsm was doing logrotate. It's keeping a
> deleted log open in the background.
> If you look at df -h /var/log, you'll see that the available size is 0.
> vdsmd restart should fix this, as would a reboot of the machine.
> Upgrading to a more recent version will also fix this.
> On an example host:
> #lsof /var/log | grep deleted
> vdsm 13039 vdsm 12w REG 253,11 1594187776 293
> /var/log/vdsm/libvirt.log.1 (deleted)
> I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm
> in the /var/log/vdsm/*.log stanza. A vdsmd restart would probably be
> needed to free up disk space immediately, though
yes, after restart vdsmd, it returns to normal:
[root@dell-per810-01 admin]# lsof /var/log | grep delete
[root@dell-per810-01 admin]# df -h /var/log
Filesystem Size Used Avail Use% Mounted on
- 2.0G 409M 1.5G 22% /var/log
and did not see low disk error.
Assign the bug to vdsm to add the fix.
Douglas - can you please try reproduce ?
Based on your comment#6 (size of libvirt.log.1 which tracked all vdsm calls to libvirt) I am going to close this bugzilla as duplicates the following:
Vdsm logs are filling filesystem up - logrotation of vdsm logs doesn't work correctly: https://bugzilla.redhat.com/show_bug.cgi?id=928217
https://bugzilla.redhat.com/show_bug.cgi?id=951612 (cloned version for z-stream)
Fell free to re-open this bugzilla if with updated vdsm (at least vdsm-22.214.171.124.0) the report can be reproduced.
*** This bug has been marked as a duplicate of bug 951612 ***