The log level and filters are set by vdsm. I suggest adding "3:virobject" to the log_filters setting to remove all OBJECT_* messages.

Added:
"1:libvirt 3:event 3:json 1:util 1:qemu 3:virobject"

restarted libvirtd and vdsmd

Log:
2013-04-03 07:51:30.297+0000: 2285: debug : virObjectRef:168 : OBJECT_REF: obj=0x840240
2013-04-03 07:51:30.297+0000: 2285: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x840240
2013-04-03 07:51:35.303+0000: 2285: debug : virObjectRef:168 : OBJECT_REF: obj=0x840240
2013-04-03 07:51:35.303+0000: 2285: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x840240
2013-04-03 07:51:40.308+0000: 2285: debug : virObjectRef:168 : OBJECT_REF: obj=0x840240
2013-04-03 07:51:40.308+0000: 2285: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x840240
2013-04-03 07:51:45.313+0000: 2285: debug : virObjectRef:168 : OBJECT_REF: obj=0x840240
2013-04-03 07:51:45.313+0000: 2285: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x840240


P.S. I have not found *any* documentation about virobject log filter. Is it documented?

(In reply to comment #4)
Change your filters to:

"3:virobject 3:event 3:json 1:libvirt 1:util 1:qemu"

as those take precedence in the order you use them in.

Log filters are not "documented" in this kind of way.  Filter is a string which specifies the source file(s) you want the logs from.  So for example, specifying your first variant:

"1:libvirt 3:event 3:json 1:util 1:qemu 3:virobject"

means filtering the logs from file 'util/virobject.c' with the 4th rule ("1:util"), as it is in the folder "util".  But in the example I suggested above, the logs from file 'util/virobject.c' will be filtered according to rule no. 1 ("3:virobject") as it takes precedence because it is the first rule that maches the file or folder name.

Thank you Martin. 
Changed to 
log_filters="3:virobject 3:virfile 3:virnetlink 3:cgroup 3:libvirt 3:event 3:json 1:util 1:qemu "
and logs are  more readable now.

virfile, cgroups and virobject generates too many useless messages even if host is in maintenance mode.

Hi all,

Thanks for sharing/suggesting the log_filters and I am pretty sure it will reduce the size and help in debugging users cases. 

However, Pavel just to confirm, do you want change from 1:libvirt to 3:libvirt? What about to 2:libvirt (info)? Is that help or still bad?

Just to track useful links: 
http://libvirt.org/guide/html/Application_Development_Guide-Connections-Debug.html
http://libvirt.org/logging.html#log_syntax
https://www.berrange.com/tags/logging/


Thanks
Douglas

Douglas,

In current implementation libvirt.c doen't contain VIR_INFO messages, don't it? So we can enable info level for the libvirt.c without growing of logs, right?

I'm not completely sure that we need debug level of the qemu_monitor.c but the messages "qemuMonitorIOProcess" is like "heartbeat" of the guest IO. Am I right?

Hi,

> In current implementation libvirt.c doen't contain VIR_INFO messages, don't it?
> So we can enable info level for the libvirt.c without growing of logs, right?

Right. I would go with info in most of cases, if the source includes it, we will get the benefit, hopefully without growing too much the log, like virnetlink.

> I'm not completely sure that we need debug level of the qemu_monitor.c but the
> messages "qemuMonitorIOProcess" is like "heartbeat" of the guest IO. Am I right?

I think it's, to monitor the events (although libvirt guys could be the best resource here). I would continue with debug, looks like we have some bugzillas where it was useful. Well, we could change it to other level and if required users could enable it again manually editing /etc/libvirt/libvirtd.conf.

Based on that, I think we might achieve a good filter. Please let me know your thoughts.

Thanks
Douglas

As discussed: 
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 2:libvirt 3:event 3:json 1:util 1:qemu "

virnetlink - virnetlink,c contains only three VIR_INFO messages sources (start, stop and startall) shouldn't fill logs
libvirt - see comment #9 Info level is for the future.

(In reply to comment #7)
> However, Pavel just to confirm, do you want change from 1:libvirt to
> 3:libvirt? What about to 2:libvirt (info)? Is that help or still bad?

Don't do this, please. libvirt.c logs all public API entry points and removing them from logs would make the logs mostly useless for any kind of troubleshooting.

Jiri, 

Currently there are not info messages in libvirt.c file but a lot of mess in debug level for example: 
$ cat libvirt.c | grep VIR_ERROR | wc -l 
1 
$ cat libvirt.c | grep VIR_WARN | wc -l 
1 
$ cat libvirt.c | grep VIR_INFO | wc -l 
0 <= ????? Sorry, But I don't understand this. Why we cannot use this level for important issue and debug level for developing purpose (like memory leaks, deadlocks and other domainFree methods) only.
$ cat libvirt.c | grep VIR_DEBUG | wc -l 
286

I've started live migration of the test VM (There is only one (!) very small VM on the host) and libvirtd produced  tons of logs (973 strings for the migration process in my case),  libvirtd.log file is beeing truncated every 22 minutes . How can I debug issue with 100+ VMS per hosts that happened yesterday for example? Every Every 2 sec libvirtd writes 9 lines of logs (qemuMonitorIOProcess) Why we cann't use one info message (return "stats") and 8 debug messages (startJob endJob qemuMonitorGetBlockExtent and so on) for this ?

Changing log level of API entry points in libvirt.c from debug to info would not really solve anything as about 90% of those 286 VIR_DEBUG calls you found are related to API entry points (along with other ~130 debug messages hidden inside VIR_DOMAIN_DEBUG). The rest of debug messages are either as important as API entry points or they are just one-time messages logged when libvirtd starts.

We know live migration unfortunately produces a lots of debug logs as libvirt polls qemu every 50ms for migration status. Since qemu does not provide any kind of "migration completed" event, we can't just wait for an event instead of this frequent polling.

Anyway, reducing log messages to "return stats" instead of logging all important actions taken in the process of returning the stats would help reduce the size of logs but at the expense of making hardly reproducible bugs impossible to analyze and fix. Such bugs are often races in libvirt code and the ability to fix them is IMHO far more important than the size of logs generated by libvirt.

From my discussion with Douglas yesterday, the following should work for us:

    3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event 3:util/json 1:libvirt 1:util 1:qemu

This would enable debug logs for qemu driver, generic util functions (except for virobject, virfile, virnetlink, cgroup, event, and json parser), and all public API entry points.

Hi,

I had a new talk with Jiri (thanks), the best solution would be:
log_filters="3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event 3:json 1:libvirt 1:util 1:qemu"

Log now is 200k from start and we are getting rid of tons of virObject logs.

Sending now a new patch to upstream.

Cheers
Douglas

The previously suggested 3:util/json would enable debug logs from qemu_monitor_json.c (which is disabled by 3:json), but while having such logs may be useful, we did not have them in the past and we were still able to get everything we needed from the logs. And since enabling qemu_monitor_json.c debugging dramatically increases the size of libvirt's logs, I agree it's better to keep this disabled and use "3:json".

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0886.html