Bug 1281918 - Ovirt-live: Engine and vdsm logging devours live environment
Summary: Ovirt-live: Engine and vdsm logging devours live environment
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-live
Classification: Retired
Component: General
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ovirt-3.6.2
: 3.6.2
Assignee: Lev Veyde
QA Contact: Jiri Belka
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-11-13 19:47 UTC by Adam Litke
Modified: 2016-02-18 11:13 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The snapshot size was limited by default to 512MB which was too small in our case. Consequence: The snapshot was running out of space, and thus becoming invalid. Fix: The default size of the snapshot was increased to 1GB, and some directories were moved to tmpfs backed mount points. Result: The issue has been resolved, and now the size of the snapshot can be configured during the boot time.
Clone Of:
Environment:
Last Closed: 2016-02-18 11:13:32 UTC
oVirt Team: Integration
Embargoed:
rule-engine: ovirt-3.6.z+
ylavi: planning_ack+
sbonazzo: devel_ack+
pstehlik: testing_ack+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 51044 0 master MERGED packaging: Enlarge live overlay 2015-12-24 10:52:45 UTC
oVirt gerrit 51045 0 master MERGED packaging: Mount DB directory as tmpfs 2015-12-24 10:56:19 UTC
oVirt gerrit 51047 0 ovirt-live-3.6 MERGED packaging: Enlarge live overlay 2015-12-24 12:35:47 UTC
oVirt gerrit 51048 0 ovirt-live-3.6 MERGED packaging: Mount DB directory as tmpfs 2015-12-24 12:35:52 UTC

Description Adam Litke 2015-11-13 19:47:53 UTC 
Description of problem:

After configuring ovirt-live, the environment only lasts for around 45 minutes (RAM dependent) before the log files consume all available RAM in the live environment


Version-Release number of selected component (if applicable): 3.6.0


How reproducible:
Always


Steps to Reproduce:
1. Boot Live image
2. Complete automatic setup
3. Wait

Actual results:
File system fills and is remounted read only, ovirt services become unresponsive.


Expected results:
System continues to run indefinitely


Additional info:

This situation could be dramatically improved by bumping the log level up to ERROR and by not archiving rotated logs when running in the Live/Demo environment.
Comment 1 Doron Fediuck 2015-11-15 07:16:35 UTC 
Adam,
any idea if it's a specific spammer? ie- is it vdsm or engine log files?
Comment 2 Red Hat Bugzilla Rules Engine 2015-11-15 07:16:41 UTC 
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 3 Red Hat Bugzilla Rules Engine 2015-11-15 07:16:41 UTC 
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.
Comment 4 Adam Litke 2015-11-23 18:44:26 UTC 
Doron, it's both unfortunately.  I suspect vdsm is the more urgent offender since we log A LOT of debugging information by default.
Comment 5 Lev Veyde 2015-12-24 10:08:54 UTC 
Following debugging the issue, it seems that the real cause is not due to the debugging, but due to space consumed during the OVF backup process.
Comment 6 Red Hat Bugzilla Rules Engine 2016-01-20 13:55:32 UTC 
Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.
Comment 7 Jiri Belka 2016-01-21 13:48:52 UTC 
ok, ovirt-live-el7-3.6.2_rc3.iso

# journalctl -x | grep 'Command line:'
Jan 21 15:19:02 localhost kernel: Command line: initrd=initrd0.img root=live:CDLABEL=ovirt-live-el7-3.6.2_rc3 rootfstype=auto ro rd.live.image quiet  rhgb rd.luks=0 rd.md=0 rd.dm=0  rd.live.overlay.size=1024 BOOT_IMAGE=vmlinuz0 

[root@livecd ~]# grep -H TEMPO /etc/sysconfig/readonly-root ; grep pgsql -H /etc/rwtab
/etc/sysconfig/readonly-root:TEMPORARY_STATE=yes
/etc/rwtab:dirs   /var/lib/pgsql
Comment 8 Jiri Belka 2016-01-21 13:49:49 UTC 
# mount -vv | grep sql
none on /var/lib/pgsql type tmpfs (rw,relatime,seclabel)
Note You need to log in before you can comment on or make changes to this bug.