Bug 1115357

Created attachment 914070 [details]
Snapshot

This might be related to log levels and excessive libvirt logs.
at least on my production and dev machines these logs are by far the largest, even if you rotate them.

(In reply to Sven Kieske from comment #2)
indeed

libvirtd   8726    root    4w   REG  253,8 1638998016   34 /var/log/libvirtd.log (deleted)

we should revisit the setting for libvirt. Other seemed fine...

requesting backport as I saw that couple of times at various setups and we lost invaluable logs because of that, which prevented a proper analysis of what was actually going on

but this does not just affect node?
also: isn't this about logging less information and not more?
so you will lose even more information?

or can the libvirt log get tuned in a way to contain less "spam"?

fyi: this is at least present since ovirt 3.3

(In reply to Sven Kieske from comment #5)
well, i saw issues exclusively on node due to it's limited /var/log partition.
You had issues on a "full" host?
the one i mentioned, 1.6GB…it's a lot, but should not really cause issues…

to me it's most urgent to solve this for node (either extend /var/log/ or limit the verboseness or rotate more often)

well I have not an issue that my /var/log is full, as it's a centos based host.
but storage does cost something so I really like to use as little as necessary :)
and I like to use it for different things than logs.

and multiple gigabytes of logs really should just be stored if you really need
them (unfortunately this seems to be the case for ovirt).

So I'm looking forward to newer versions of ovirt where the default loglevels
will be "info" or "warning" instead of "debug" for production.

PS: supporting central syslog servers would ease the pain of this, too.

Yaniv, do you know if vdsm is setting the libvirt log interval?

Yuri, is this more a problem about not enough logging space or the space filling up to quickly?

I'd also suggest to take a real look at the logs, and to decide
if they contain useful information, or maybe something spams the same
messages over and over again, and maybe this can be reduced, thus the logs
do not grow so fast and large.

Fabian, 
Once I populated 185 VMs disk space filling up very quickly (1/2 hour).
For performance and scalability tests we don't need debug information. 
And I/O could be the bottleneck.
Regarding production environment it must be ERROR level by default of course.
I found solution for VDSM - change log level at file /etc/vdsm/logger.conf 
from DEBUG to ERROR.
Still open question regarding libvirt ???

I do not recommend messing with vdsm's loglevel

And I'm not entirely sure what to do with libvirt's….I'd be a bit more in favor of limiting their verboseness

Yes Fabien, vdsm sets libvirtd.logrotate configuration file (its under lib/vdsm/tool/libvirtd.logrotate)

Yuri,
could you please paste the contents of /etc/logrotate.d/libvirtd ?

Maybe a dupe of bug 1115997

As you requested /etc/logrotate.d/libvirtd

## beginning of configuration section by vdsm
/var/log/libvirt/libvirtd.log {
    rotate 100
    missingok
    copytruncate
    size 15M
    compress
    compresscmd /usr/bin/xz
    uncompresscmd /usr/bin/unxz
    compressext .xz
}
## end of configuration section by vdsm
## beginning of configuration section by vdsm
/var/log/libvirt/libvirtd.log {
    rotate 100
    missingok
    copytruncate
    size 15M
    compress
    compresscmd /usr/bin/xz
    uncompresscmd /usr/bin/unxz
    compressext .xz
}
## end of configuration section by vdsm
/var/log/libvirt/libvirtd.log {
        weekly
        missingok
        rotate 4
        compress
        delaycompress
        copytruncate
        minsize 100k
}

Hey Yuri, thanks for your quick reply. it seems to be a dupe.

*** This bug has been marked as a duplicate of bug 1115997 ***

@Yuri, could you please attach a sample from the log file?

We need to figure out if the logs are too verbose by default (vdsm/libvirt).

If that's the case, I suggest we re-open this BZ, and take care of the default log verbosity and not just the log rotation.

I reopen the bug due the new information I've got.
Size of libvirt log file increased from 879 Kb up to 1.5 Gb during less 2 hours
I performed VM creations using automatic tool - JMeter
and created 130 VMs
I've got "Low disk space" error again
See below information.

879811 2014-07-30 06:51 libvirtd.log
1490116749 2014-07-30 08:41 libvirtd.log

-rw-r--r--. 1 root root  4220437 2014-07-30 06:51 supervdsm.log
-rw-r--r--. 1 vdsm kvm  16450127 2014-07-30 06:51 vdsm.log

-rw-r--r--. 1 vdsm kvm    853068 2014-07-30 07:00 vdsm.log-20140730.xz
-rw-r--r--. 1 vdsm kvm   1067696 2014-07-30 08:01 vdsm.log.1.xz
-rw-r--r--. 1 root root  4523174 2014-07-30 08:19 supervdsm.log
-rw-r--r--. 1 vdsm kvm  10819610 2014-07-30 08:41 vdsm.log

Log files have very detailed debug information which we don't need by default
(Production environment). 
Default log level configuration for both libvirt and vdsm must be "ERROR" and not "DEBUG".

Attached 2 examples of libvirt.log and vdsm.log

Created attachment 922483 [details]
libvirt log

Created attachment 922484 [details]
vdsm log

(In reply to Yuri Obshansky from comment #18)
> I reopen the bug due the new information I've got.
> Size of libvirt log file increased from 879 Kb up to 1.5 Gb during less 2
> hours
> I performed VM creations using automatic tool - JMeter
> and created 130 VMs
> I've got "Low disk space" error again
> See below information.

This error should be resolved by the patch attached in the bz#1115997.
Next VDSM version should cover that.

> 
> 879811 2014-07-30 06:51 libvirtd.log
> 1490116749 2014-07-30 08:41 libvirtd.log
> 
> -rw-r--r--. 1 root root  4220437 2014-07-30 06:51 supervdsm.log
> -rw-r--r--. 1 vdsm kvm  16450127 2014-07-30 06:51 vdsm.log
> 
> -rw-r--r--. 1 vdsm kvm    853068 2014-07-30 07:00 vdsm.log-20140730.xz
> -rw-r--r--. 1 vdsm kvm   1067696 2014-07-30 08:01 vdsm.log.1.xz
> -rw-r--r--. 1 root root  4523174 2014-07-30 08:19 supervdsm.log
> -rw-r--r--. 1 vdsm kvm  10819610 2014-07-30 08:41 vdsm.log
> 
> Log files have very detailed debug information which we don't need by default
> (Production environment). 
> Default log level configuration for both libvirt and vdsm must be "ERROR"
> and not "DEBUG".
> 
> Attached 2 examples of libvirt.log and vdsm.log

These suggestion should be threat in a different bugzilla, in the VDSM component. Fell free to open, I am closing this bugzilla as the main

*** This bug has been marked as a duplicate of bug 1115997 ***