Bug 1281918 - Ovirt-live: Engine and vdsm logging devours live environment
Ovirt-live: Engine and vdsm logging devours live environment
Status: CLOSED CURRENTRELEASE
Product: ovirt-live
Classification: oVirt
Component: General (Show other bugs)
3.6.0
Unspecified Unspecified
medium Severity high (vote)
: ovirt-3.6.2
: 3.6.2
Assigned To: Lev Veyde
Jiri Belka
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-13 14:47 EST by Adam Litke
Modified: 2016-02-18 06:13 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The snapshot size was limited by default to 512MB which was too small in our case. Consequence: The snapshot was running out of space, and thus becoming invalid. Fix: The default size of the snapshot was increased to 1GB, and some directories were moved to tmpfs backed mount points. Result: The issue has been resolved, and now the size of the snapshot can be configured during the boot time.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-18 06:13:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑3.6.z+
ylavi: planning_ack+
sbonazzo: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 51044 master MERGED packaging: Enlarge live overlay 2015-12-24 05:52 EST
oVirt gerrit 51045 master MERGED packaging: Mount DB directory as tmpfs 2015-12-24 05:56 EST
oVirt gerrit 51047 ovirt-live-3.6 MERGED packaging: Enlarge live overlay 2015-12-24 07:35 EST
oVirt gerrit 51048 ovirt-live-3.6 MERGED packaging: Mount DB directory as tmpfs 2015-12-24 07:35 EST

  None (edit)
Description Adam Litke 2015-11-13 14:47:53 EST
Description of problem:

After configuring ovirt-live, the environment only lasts for around 45 minutes (RAM dependent) before the log files consume all available RAM in the live environment


Version-Release number of selected component (if applicable): 3.6.0


How reproducible:
Always


Steps to Reproduce:
1. Boot Live image
2. Complete automatic setup
3. Wait

Actual results:
File system fills and is remounted read only, ovirt services become unresponsive.


Expected results:
System continues to run indefinitely


Additional info:

This situation could be dramatically improved by bumping the log level up to ERROR and by not archiving rotated logs when running in the Live/Demo environment.
Comment 1 Doron Fediuck 2015-11-15 02:16:35 EST
Adam,
any idea if it's a specific spammer? ie- is it vdsm or engine log files?
Comment 2 Red Hat Bugzilla Rules Engine 2015-11-15 02:16:41 EST
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 3 Red Hat Bugzilla Rules Engine 2015-11-15 02:16:41 EST
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.
Comment 4 Adam Litke 2015-11-23 13:44:26 EST
Doron, it's both unfortunately.  I suspect vdsm is the more urgent offender since we log A LOT of debugging information by default.
Comment 5 Lev Veyde 2015-12-24 05:08:54 EST
Following debugging the issue, it seems that the real cause is not due to the debugging, but due to space consumed during the OVF backup process.
Comment 6 Red Hat Bugzilla Rules Engine 2016-01-20 08:55:32 EST
Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.
Comment 7 Jiri Belka 2016-01-21 08:48:52 EST
ok, ovirt-live-el7-3.6.2_rc3.iso

# journalctl -x | grep 'Command line:'
Jan 21 15:19:02 localhost kernel: Command line: initrd=initrd0.img root=live:CDLABEL=ovirt-live-el7-3.6.2_rc3 rootfstype=auto ro rd.live.image quiet  rhgb rd.luks=0 rd.md=0 rd.dm=0  rd.live.overlay.size=1024 BOOT_IMAGE=vmlinuz0 

[root@livecd ~]# grep -H TEMPO /etc/sysconfig/readonly-root ; grep pgsql -H /etc/rwtab
/etc/sysconfig/readonly-root:TEMPORARY_STATE=yes
/etc/rwtab:dirs   /var/lib/pgsql
Comment 8 Jiri Belka 2016-01-21 08:49:49 EST
# mount -vv | grep sql
none on /var/lib/pgsql type tmpfs (rw,relatime,seclabel)

Note You need to log in before you can comment on or make changes to this bug.