Bug 971552 - HostVG/Logging volume is not mounted on RHEV-H, and hypervisor reports host has less than 500MB free space remaining
HostVG/Logging volume is not mounted on RHEV-H, and hypervisor reports host h...
Status: CLOSED DUPLICATE of bug 951612
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.2.0
All Linux
high Severity high
: ---
: 3.3.0
Assigned To: Douglas Schilling Landgraf
infra
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-06 15:32 EDT by Allan Voss
Modified: 2016-02-10 14:40 EST (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-13 21:26:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Allan Voss 2013-06-06 15:32:31 EDT
Description of problem:
RHEV-H is failing to boot the HostVG/Logging volume, and the following is repeated in audit_log every 15 minutes:
Critical, Low disk space. Host <hostname> has less than 500 MB of free space left on: /var/log.

Version-Release number of selected component (if applicable):
20121212.0.el6_3

How reproducible:
Unknown, but has been encountered by several customers

Steps to Reproduce:
1. Install RHEV-H
2. Run RHEV-H for an unknown period

Actual results:
HostVG/Logging fails to mount on boot

Expected results:
HostVG/Logging mounted on boot normally


Additional info:
It's especially difficult to troubleshoot this problem because the logs are wiped out on the next hypervisor reboot
Comment 5 Mike Burns 2013-06-26 08:47:50 EDT
This issue was a bug in how vdsm was doing logrotate.  It's keeping a deleted log open in the background.  

If you look at df -h /var/log, you'll see that the available size is 0.

vdsmd restart should fix this, as would a reboot of the machine.  

Upgrading to a more recent version will also fix this.

On an example host:

#lsof /var/log | grep deleted
vdsm     13039 vdsm   12w   REG 253,11 1594187776  293 /var/log/vdsm/libvirt.log.1 (deleted)

I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm in the  /var/log/vdsm/*.log stanza.  A vdsmd restart would probably be needed to free up disk space immediately, though
Comment 6 Guohua Ouyang 2013-06-27 04:40:30 EDT
(In reply to Mike Burns from comment #5)
> This issue was a bug in how vdsm was doing logrotate.  It's keeping a
> deleted log open in the background.  
> 
> If you look at df -h /var/log, you'll see that the available size is 0.
> 
> vdsmd restart should fix this, as would a reboot of the machine.  
> 
> Upgrading to a more recent version will also fix this.
> 
> On an example host:
> 
> #lsof /var/log | grep deleted
> vdsm     13039 vdsm   12w   REG 253,11 1594187776  293
> /var/log/vdsm/libvirt.log.1 (deleted)
> 
> I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm
> in the  /var/log/vdsm/*.log stanza.  A vdsmd restart would probably be
> needed to free up disk space immediately, though

yes, after restart vdsmd, it returns to normal:
[root@dell-per810-01 admin]# lsof /var/log | grep delete
[root@dell-per810-01 admin]# df -h /var/log
Filesystem            Size  Used Avail Use% Mounted on
-                     2.0G  409M  1.5G  22% /var/log

and did not see low disk error.

Assign the bug to vdsm to add the fix.
Comment 11 Barak 2013-08-13 08:44:07 EDT
Douglas - can you please try reproduce ?
Comment 12 Douglas Schilling Landgraf 2013-08-13 21:26:30 EDT
Hello Ouyang,

Based on your comment#6 (size of libvirt.log.1 which tracked all vdsm calls to libvirt) I am going to close this bugzilla as duplicates the following:

Vdsm logs are filling filesystem up - logrotation of vdsm logs doesn't work correctly: https://bugzilla.redhat.com/show_bug.cgi?id=928217

https://bugzilla.redhat.com/show_bug.cgi?id=951612 (cloned version for z-stream)

Fell free to re-open this bugzilla if with updated vdsm (at least vdsm-4.10.2.19.0) the report can be reproduced.

Thanks
Douglas

*** This bug has been marked as a duplicate of bug 951612 ***

Note You need to log in before you can comment on or make changes to this bug.