Description of problem:
Default logging level configuration of VDSM and libvirt is DEBUG
which leads to increase of log files size and impact on performance.
It is not good idea for Production environment
(scale and performance tests performed on similar to production environments).
My last scale test showed the growth of libvirt.log file from 879 Kb up to 1.5 Gb during less 2 hours.
As result I got "Low disk space" error.
At the same time I verified what's an impact of DEBUG logging on vdsm performance.
The scenario is: create VMs on HOST (64 CPUs and 400 G RAM) till RHEVM starts to move VMs status to "Not responding" or vdsm raises exceptions "SSLError: bad write retry" (see details in bug 1102147).
DEBUG level - host can manage 182 VMs
ERROR level - host can manage 205 VMs
i.e. ~ 10% performance improvement only by change logging level.
Logging level changed at:
libvirt - /etc/libvirt/libvirtd.conf
vdsm - /etc/vdsm/logger.conf
Version-Release number of selected component (if applicable):
OS Version: RHEV Hypervisor - 6.5 - 20140624.0.el6ev
Kernel Version: 2.6.32 - 431.20.3.el6.x86_64
KVM Version: 0.12.1.2 - 2.415.el6_5.10
LIBVIRT Version: libvirt-0.10.2-29.el6_5.9
VDSM Version: vdsm-4.14.7-3.el6ev
Steps to Reproduce:
Created attachment 922879 [details]
Created attachment 922880 [details]
Created attachment 922881 [details]
Yaniv - thoughts on that?
note patch 28869 about vdsm log format discussion
other than that there is a general disagreement on that, solution needs to be discussed on devel list
We won't change vdsm log level nor the format.
about libvirt log level,  is supposed to use libvirt defaults () and should fix the too verbose issue
just need to verify that against libvirt version.
Danken any thoughts whether this should be handled at all ?
As Yaniv said - I find DEBUG in vdsm still useful for hard bugs. libvirt.log is less helpful for me recently, so I support dropping our tweaks to it.
This report is interesting, but I'd like to see hard numbers now that the silly bug 1102147 is resolved.
Yuri, could you repeat your measurements with Vdsm-4.16.2, preferably with http://gerrit.ovirt.org/#/c/31135/ applied, too?
> Yuri, could you repeat your measurements with Vdsm-4.16.2, preferably with http://gerrit.ovirt.org/#/c/31135/ applied, too?
master now contains 31135
I cannot do it. It is a little bit problematic. We test only released version of RHEVM since we have a very large environment.
Might need to communicate to customers:
This might cause changing of libvirt's log file from
/var/log/libvirt to journal in accordance with libvirt's
defaults. This happens when /run/systemd/journal/socket
exists on the machine see:
(In reply to Mooli Tayer from comment #13)
> Might need to communicate to customers:
> This might cause changing of libvirt's log file from
> /var/log/libvirt to journal in accordance with libvirt's
> defaults. This happens when /run/systemd/journal/socket
> exists on the machine see:
Please select the correct Doc Type and provide the doc text ASAP for this bug to make it in the 3.5 Beta Manager Release Notes. If this bug is not required for release notes, please set the require_release_note flag to -.
I tried verify bug on
RHEL - 6Server - 220.127.116.11.el6
I didn't find any log configuration in /etc/libvirt/libvirtd.conf
## beginning of configuration section by vdsm-4.13.0
## end of configuration section by vdsm-4.13.0
In spite that log level is ERROR as I can determine from
where only error printouts
The question: where can I verify libvirt log configuration?
The fix for this issue was to remove custom libvirt log configuration.
I'm not sure what you mean by verify libvirt log configuration.
The default log behavior is mentioned in the release note for this bug
(see "Doc Text"). you can also grep for "log_outputs|log_filters" at libvirtd.conf.
I read release note, and it is not clear.
This statement - "Now, the default logging level of libvirt is used and verbosity is decreased."
What level of default logging level? INFO? ERROR?
When you said "remove custom libvirt log configuration"?
Does it mean that we are not able to change logging level?
What do I need?
I should verify that in current version libvirt default logging level is not DEBUG.
[root@ucs1-b420-1 ~]# cat /etc/libvirt/libvirtd.conf |grep log_outputs
[root@ucs1-b420-1 ~]# cat /etc/libvirt/libvirtd.conf |grep log_filters
And all are commented
How can I verify this bug?
(In reply to Yuri Obshansky from comment #17)
> I read release note, and it is not clear.
> This statement - "Now, the default logging level of libvirt is used and
> verbosity is decreased."
> What level of default logging level? INFO? ERROR?
What ever default libvirt is using. The point is we no longer touch it.
3 or "warn": log warnings and errors, that's the default value
from libvirt docs, linked in the release note:
> When you said "remove custom libvirt log configuration"?
> Does it mean that we are not able to change logging level?
It does not mean that we are not able. we do not want to change the default anymore - that is due to this bug and also due to the reason mentioned by Dan in the commit: http://gerrit.ovirt.org/#/c/31135/
> What do I need?
> I should verify that in current version libvirt default logging level is not
You are not checking libvirt or it's defaults...
vdsm previously used to override default configuration in the "## beginning of configuration section by vdsm ..." section. It does not do so anymore.
what you wrote below is a verification of that.
> I grep
> [root@ucs1-b420-1 ~]# cat /etc/libvirt/libvirtd.conf |grep log_outputs
> [root@ucs1-b420-1 ~]# cat /etc/libvirt/libvirtd.conf |grep log_filters
> #log_filters="3:remote 4:event"
> And all are commented
> How can I verify this bug?
To verify I think you need to check the problem and not the solution:
Ignore the last sentence.
I cannot ignore the last sentence. It was impolite.
I know what I need to do.
Release note is not clear.
Bug didn't reproduce on:
- RHEV-M 3.5.0-0.22.el6ev
- RHEL - 6Server - 18.104.22.168.el6
Moved to verified
I did not mean to offend you.
I do not think that you do not know what to do.
What I meant is this:
Since the original bug was about starting vm's the solution should be tested by starting vms. At second look I saw that the logging issue was mentioned in the header and is not only part of the solution so I dropped my idea.
If you have a suggestion on improving the release note please provide it.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.