Bug 971552

Summary:	HostVG/Logging volume is not mounted on RHEV-H, and hypervisor reports host has less than 500MB free space remaining
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Allie DeVolder <adevolder>
Component:	vdsm	Assignee:	Douglas Schilling Landgraf <dougsland>
Status:	CLOSED DUPLICATE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	3.2.0	CC:	acathrow, adevolder, bazulay, cshao, gouyang, hadong, hateya, huiwa, iheim, jboggs, jkt, leiwang, lpeer, mburns, ovirt-maint, ycui
Target Milestone:	---	Keywords:	Triaged
Target Release:	3.3.0
Hardware:	All
OS:	Linux
Whiteboard:	infra
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-08-14 01:26:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Allie DeVolder 2013-06-06 19:32:31 UTC

Description of problem:
RHEV-H is failing to boot the HostVG/Logging volume, and the following is repeated in audit_log every 15 minutes:
Critical, Low disk space. Host <hostname> has less than 500 MB of free space left on: /var/log.

Version-Release number of selected component (if applicable):
20121212.0.el6_3

How reproducible:
Unknown, but has been encountered by several customers

Steps to Reproduce:
1. Install RHEV-H
2. Run RHEV-H for an unknown period

Actual results:
HostVG/Logging fails to mount on boot

Expected results:
HostVG/Logging mounted on boot normally


Additional info:
It's especially difficult to troubleshoot this problem because the logs are wiped out on the next hypervisor reboot

Comment 5 Mike Burns 2013-06-26 12:47:50 UTC

This issue was a bug in how vdsm was doing logrotate.  It's keeping a deleted log open in the background.  

If you look at df -h /var/log, you'll see that the available size is 0.

vdsmd restart should fix this, as would a reboot of the machine.  

Upgrading to a more recent version will also fix this.

On an example host:

#lsof /var/log | grep deleted
vdsm     13039 vdsm   12w   REG 253,11 1594187776  293 /var/log/vdsm/libvirt.log.1 (deleted)

I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm in the  /var/log/vdsm/*.log stanza.  A vdsmd restart would probably be needed to free up disk space immediately, though

Comment 6 Guohua Ouyang 2013-06-27 08:40:30 UTC

(In reply to Mike Burns from comment #5)
> This issue was a bug in how vdsm was doing logrotate.  It's keeping a
> deleted log open in the background.  
> 
> If you look at df -h /var/log, you'll see that the available size is 0.
> 
> vdsmd restart should fix this, as would a reboot of the machine.  
> 
> Upgrading to a more recent version will also fix this.
> 
> On an example host:
> 
> #lsof /var/log | grep deleted
> vdsm     13039 vdsm   12w   REG 253,11 1594187776  293
> /var/log/vdsm/libvirt.log.1 (deleted)
> 
> I believe this can be fixed by adding copytruncate to /etc/logrotate.d/vdsm
> in the  /var/log/vdsm/*.log stanza.  A vdsmd restart would probably be
> needed to free up disk space immediately, though

yes, after restart vdsmd, it returns to normal:
[root@dell-per810-01 admin]# lsof /var/log | grep delete
[root@dell-per810-01 admin]# df -h /var/log
Filesystem            Size  Used Avail Use% Mounted on
-                     2.0G  409M  1.5G  22% /var/log

and did not see low disk error.

Assign the bug to vdsm to add the fix.

Comment 11 Barak 2013-08-13 12:44:07 UTC

Douglas - can you please try reproduce ?

Comment 12 Douglas Schilling Landgraf 2013-08-14 01:26:30 UTC

Hello Ouyang,

Based on your comment#6 (size of libvirt.log.1 which tracked all vdsm calls to libvirt) I am going to close this bugzilla as duplicates the following:

Vdsm logs are filling filesystem up - logrotation of vdsm logs doesn't work correctly: https://bugzilla.redhat.com/show_bug.cgi?id=928217

https://bugzilla.redhat.com/show_bug.cgi?id=951612 (cloned version for z-stream)

Fell free to re-open this bugzilla if with updated vdsm (at least vdsm-4.10.2.19.0) the report can be reproduced.

Thanks
Douglas

*** This bug has been marked as a duplicate of bug 951612 ***