Bug 1281918

Summary:	Ovirt-live: Engine and vdsm logging devours live environment
Product:	[Retired] ovirt-live	Reporter:	Adam Litke <alitke>
Component:	General	Assignee:	Lev Veyde <lveyde>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jiri Belka <jbelka>
Severity:	high	Docs Contact:
Priority:	medium
Version:	3.6.0	CC:	alitke, bugs, dfediuck, didi, lveyde, rmartins, sbonazzo, stirabos, ylavi
Target Milestone:	ovirt-3.6.2	Flags:	rule-engine: ovirt-3.6.z+ ylavi: planning_ack+ sbonazzo: devel_ack+ pstehlik: testing_ack+
Target Release:	3.6.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The snapshot size was limited by default to 512MB which was too small in our case. Consequence: The snapshot was running out of space, and thus becoming invalid. Fix: The default size of the snapshot was increased to 1GB, and some directories were moved to tmpfs backed mount points. Result: The issue has been resolved, and now the size of the snapshot can be configured during the boot time.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-02-18 11:13:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Adam Litke 2015-11-13 19:47:53 UTC

Description of problem:

After configuring ovirt-live, the environment only lasts for around 45 minutes (RAM dependent) before the log files consume all available RAM in the live environment


Version-Release number of selected component (if applicable): 3.6.0


How reproducible:
Always


Steps to Reproduce:
1. Boot Live image
2. Complete automatic setup
3. Wait

Actual results:
File system fills and is remounted read only, ovirt services become unresponsive.


Expected results:
System continues to run indefinitely


Additional info:

This situation could be dramatically improved by bumping the log level up to ERROR and by not archiving rotated logs when running in the Live/Demo environment.

Comment 1 Doron Fediuck 2015-11-15 07:16:35 UTC

Adam,
any idea if it's a specific spammer? ie- is it vdsm or engine log files?

Comment 2 Red Hat Bugzilla Rules Engine 2015-11-15 07:16:41 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 3 Red Hat Bugzilla Rules Engine 2015-11-15 07:16:41 UTC

This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.

Comment 4 Adam Litke 2015-11-23 18:44:26 UTC

Doron, it's both unfortunately.  I suspect vdsm is the more urgent offender since we log A LOT of debugging information by default.

Comment 5 Lev Veyde 2015-12-24 10:08:54 UTC

Following debugging the issue, it seems that the real cause is not due to the debugging, but due to space consumed during the OVF backup process.

Comment 6 Red Hat Bugzilla Rules Engine 2016-01-20 13:55:32 UTC

Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.

Comment 7 Jiri Belka 2016-01-21 13:48:52 UTC

ok, ovirt-live-el7-3.6.2_rc3.iso

# journalctl -x | grep 'Command line:'
Jan 21 15:19:02 localhost kernel: Command line: initrd=initrd0.img root=live:CDLABEL=ovirt-live-el7-3.6.2_rc3 rootfstype=auto ro rd.live.image quiet  rhgb rd.luks=0 rd.md=0 rd.dm=0  rd.live.overlay.size=1024 BOOT_IMAGE=vmlinuz0 

[root@livecd ~]# grep -H TEMPO /etc/sysconfig/readonly-root ; grep pgsql -H /etc/rwtab
/etc/sysconfig/readonly-root:TEMPORARY_STATE=yes
/etc/rwtab:dirs   /var/lib/pgsql

Comment 8 Jiri Belka 2016-01-21 13:49:49 UTC

# mount -vv | grep sql
none on /var/lib/pgsql type tmpfs (rw,relatime,seclabel)