Bug 1141763

Engine's version:
rhevm-3.5.0-0.12.beta.el6ev.noarch

Could not reproduce using master vdsm on el6.5.

I'm not sure if this is because it was a local vdsm
installation and not through the engine or because it
was master.

Could you please attach /etc/libvirt/libvirtd.conf
from one of your hosts OR attach the section after:
## beginning of configuration section by vdsm-4.13.0
?

Also I'm don't know about /var/log/libvirtd.log
but only /var/log/libvirt/libvirtd.log

(In reply to Mooli Tayer from comment #2)
> Could not reproduce using master vdsm on el6.5.
> 
> I'm not sure if this is because it was a local vdsm
> installation and not through the engine or because it
> was master.
> 
> Could you please attach /etc/libvirt/libvirtd.conf
> from one of your hosts OR attach the section after:
> ## beginning of configuration section by vdsm-4.13.0
> ?
> 
> Also I'm don't know about /var/log/libvirtd.log
> but only /var/log/libvirt/libvirtd.log

1.My installation was via WEBUI of the engine.
2.Libvirt.conf attached from both hosts.
3.Those not available at all currently because of the bug opened on this.

Created attachment 938028 [details]
libvirt.conf provided

Seems related to [1]

Specifically adding in /etc/libvirt/libvirtd.conf:
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
Solves the problem.

Otherwise log file is journal (syslog):
$ grep  -r 'libvirt version' /var/log/messages
Sep 16 17:22:58 alma03 journal: libvirt version: 1.1.1, package: 29.el7_0.1 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-07-03-07:38:27, x86-017.build.eng.bos.redhat.com)

it seems that according to libvirt docs [2] if /run/systemd/journal/socket
exists logging is done to journal.
(it also states that  it will use/var/log/libvirt/libvirtd.log if running as a daemon but I guess the first rule takes over if both are true.)

This explains why I could not reproduce locally - for some reason on my el6.5:

$ ll /run/systemd/journal/socket
ls: cannot access /run/systemd/journal/socket: No such file or directory

and on Nikolai's machine:
$ ll  /run/systemd/journal/socket
srw-rw-rw-. 1 root root 0 Sep 16 17:01 /run/systemd/journal/socket

Fix should be quick.

[1] http://gerrit.ovirt.org/#/q/I251c2534d3284aa95bffa56143c449d5de887008,n,z
[2] http://libvirt.org/logging.html

(In reply to Mooli Tayer from comment #5)
> Seems related to [1]
> 
> Specifically adding in /etc/libvirt/libvirtd.conf:
> log_outputs="1:file:/var/log/libvirt/libvirtd.log"
> Solves the problem.
> 
> Otherwise log file is journal (syslog):
> $ grep  -r 'libvirt version' /var/log/messages
> Sep 16 17:22:58 alma03 journal: libvirt version: 1.1.1, package: 29.el7_0.1
> (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-07-03-07:38:27,
> x86-017.build.eng.bos.redhat.com)
> 
> it seems that according to libvirt docs [2] if /run/systemd/journal/socket
> exists logging is done to journal.
> (it also states that  it will use/var/log/libvirt/libvirtd.log if running as
> a daemon but I guess the first rule takes over if both are true.)
> 
> This explains why I could not reproduce locally - for some reason on my
> el6.5:
> 
> $ ll /run/systemd/journal/socket
> ls: cannot access /run/systemd/journal/socket: No such file or directory
> 
> and on Nikolai's machine:
> $ ll  /run/systemd/journal/socket
> srw-rw-rw-. 1 root root 0 Sep 16 17:01 /run/systemd/journal/socket
> 
> Fix should be quick.
> 
> [1] http://gerrit.ovirt.org/#/q/I251c2534d3284aa95bffa56143c449d5de887008,n,z
> [2] http://libvirt.org/logging.html

Good catch, lets push on it, so it'll enter already to the closest build.

Why is this a bug? Are libvirt's logs are accessible via journalctl? If so, I see no reason to override libvirt's default.

However, we should make sure that our log collector takes libvirt's log properly.

Please check that it works also for the RHEL6.6 repos.

We must (In reply to Dan Kenigsberg from comment #7)
> Why is this a bug? Are libvirt's logs are accessible via journalctl? If so,
> I see no reason to override libvirt's default.

I think this should be communicated to users:
requested doc text on bug 1125237, comment 13

> 
> However, we should make sure that our log collector takes libvirt's log
> properly.

Nikolai: it seems that this change is intentional.
Could you test the collection of syslog in your environment?

(In reply to Mooli Tayer from comment #9)
> We must (In reply to Dan Kenigsberg from comment #7)
> > Why is this a bug? Are libvirt's logs are accessible via journalctl? If so,
> > I see no reason to override libvirt's default.
> 
> I think this should be communicated to users:
> requested doc text on bug 1125237, comment 13
> 
> > 
> > However, we should make sure that our log collector takes libvirt's log
> > properly.
> 
> Nikolai: it seems that this change is intentional.
> Could you test the collection of syslog in your environment?
If RHEL6.6 required, please contact to Ilanit Stein.

No need for 6.6.
The environment that you already have setup is fine.

As long as a host is logging to syslog.

(In reply to Mooli Tayer from comment #11)
> No need for 6.6.
> The environment that you already have setup is fine.
> 
> As long as a host is logging to syslog.

Attached a log file after I ran "rhevm-log-collector collect" on engine, please review regarding both almas only, as on them libvirtd.log is missing.

OK we got syslog with log collector:

tar -xf sosreport-LogCollector-20140917162219.tar.xz 
$ cd log-collector-data/alma03.qa.lab.tlv.redhat.com/
$ tar -xf alma03.qa.lab.tlv.redhat.com-sosreport-alma03.qa.lab.tlv.redhat.com-20140917162029.tar.xz 
$ cd sosreport-alma03.qa.lab.tlv.redhat.com-20140917162029/var/log/
$ cat messages | grep "libvirt version"| head -1
Sep 14 17:02:34 alma03 journal: libvirt version: 1.1.1, package: 29.el7_0.1 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-07-03-07:38:27, x86-017.build.eng.bos.redhat.com)

at sosreport-alma03.qa.lab.tlv.redhat.com-20140917162029/var/log/libvirt

Where we usually have libvirtd.log and qemu.log There is only qemu.log

Thanks Nikolai!

Closing this one as not a bug since usage of libvirt default logging
behavior is what we want since 3.5. 

requested doc-text on bug 1125237, comment 13 as I think this should be
documented.

(In reply to Mooli Tayer from comment #13)
> OK we got syslog with log collector:
> 
> tar -xf sosreport-LogCollector-20140917162219.tar.xz 
> $ cd log-collector-data/alma03.qa.lab.tlv.redhat.com/
> $ tar -xf
> alma03.qa.lab.tlv.redhat.com-sosreport-alma03.qa.lab.tlv.redhat.com-
> 20140917162029.tar.xz 
> $ cd sosreport-alma03.qa.lab.tlv.redhat.com-20140917162029/var/log/
> $ cat messages | grep "libvirt version"| head -1
> Sep 14 17:02:34 alma03 journal: libvirt version: 1.1.1, package: 29.el7_0.1
> (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-07-03-07:38:27,
> x86-017.build.eng.bos.redhat.com)
> 
> at sosreport-alma03.qa.lab.tlv.redhat.com-20140917162029/var/log/libvirt
> 
> Where we usually have libvirtd.log and qemu.log There is only qemu.log
> 
> Thanks Nikolai!
> 
> Closing this one as not a bug since usage of libvirt default logging
> behavior is what we want since 3.5. 
> 
> requested doc-text on bug 1125237, comment 13 as I think this should be
> documented.

So where can I get the libvirt log now, if it's not a bug?

use:

$ journalctl -u libvirtd

You will notice you get less output than what you are used to.

That is intentional, see bug 1125237.

If you need more verbose output or a different log location
you can update /etc/libvirt/libvirtd.conf

(In reply to Mooli Tayer from comment #15)
> use:
> 
> $ journalctl -u libvirtd
> 
> You will notice you get less output than what you are used to.
> 
> That is intentional, see bug 1125237.
> 
> If you need more verbose output or a different log location
> you can update /etc/libvirt/libvirtd.conf

journalctl -u libvirtd
-bash: journalctl: command not found
So the first option not working for me, please advise.

journalctl -u libvirtd
-bash: journalctl: command not found
So the first option not working for me, please advise.

I'm pretty sure I've tested it on one of the Alma0x servers.

I cant seem to reach them now but in the mean time you look
at the syslog or grep it at /var/log/messages.

what's going on with this NOTABUG issue?
I'm using libvirtd.log on an almost daily basis and now it's gone?

it will be a good decision to open this one.For example BZ #1147644 is a bug that demonstrate well ,to my opinion,the need for libvirtd logging,we shouldn't disregard this

There is still logging,
it has just moved to the default location/verbosity defined by libvirt[1].

If in any case you see different behavior that's a bug.

if you want different location put:
log_outputs "1:file:/var/log/libvirt/libvirtd.log"

into /etc/libvirt/libvirtd.log

if you want the old verbosity put:
log_outputs "1:file:/var/log/libvirt/libvirtd.log"

log_filters "3:virobject 3:virfile 2:virnetlink 3:cgroup 3:event 3:json 1:libvirt 1:util 1:qemu"

[1]
see 
http://libvirt.org/logging.html
"Logging in the daemon."

my two hosts do not contain libvirtd.log file at all

root@camel-vdsb / # noglob find -name *.log | grep virt
./var/log/libvirt/qemu/vm_test.log

root@camel-vdsc / # noglob find -name *.log | grep "virt"   
./var/log/libvirt/qemu/vm_test.log
./var/log/libvirt/qemu/vm____0.log
./var/log/libvirt/qemu/vm_snap.log
./var/log/libvirt/qemu/vm_ttt.log
./var/log/libvirt/qemu/vm.log
./var/log/libvirt/qemu/vm_sn.log

That's because on some situations libvirt uses journal.

See [1] at comment 22.

Closing.

See comment 7, comment 13.

To attain the preview situation see comment 22.

Hey Nikolai,
We remove vdsm configuration for libvirt log to let users use libvirt's default, which is to log priority 3 or more (i.e, info, warn, and error) to syslog (journald is preferred over syslog if present). if we want to change the default we can ask libvirt guys to consider this request.. but what is it ? to continue with vdsm specific conf? we prefer to avoid that.

(In reply to Yaniv Bronhaim from comment #27)
> Hey Nikolai,
> We remove vdsm configuration for libvirt log to let users use libvirt's
> default, which is to log priority 3 or more (i.e, info, warn, and error) to
> syslog (journald is preferred over syslog if present). if we want to change
> the default we can ask libvirt guys to consider this request.. but what is
> it ? to continue with vdsm specific conf? we prefer to avoid that.

I'd like that you'll take it with PM.

(In reply to Nikolai Sednev from comment #28)
> (In reply to Yaniv Bronhaim from comment #27)
> > Hey Nikolai,
> > We remove vdsm configuration for libvirt log to let users use libvirt's
> > default, which is to log priority 3 or more (i.e, info, warn, and error) to
> > syslog (journald is preferred over syslog if present). if we want to change
> > the default we can ask libvirt guys to consider this request.. but what is
> > it ? to continue with vdsm specific conf? we prefer to avoid that.
> 
> I'd like that you'll take it with PM.

It isn't relevant for PMs here. Also, customers/users don't even get to this log file anyway.
But if you insist then, Scott - can we close this one?

(In reply to Oved Ourfali from comment #29)
> (In reply to Nikolai Sednev from comment #28)
> > (In reply to Yaniv Bronhaim from comment #27)
> > > Hey Nikolai,
> > > We remove vdsm configuration for libvirt log to let users use libvirt's
> > > default, which is to log priority 3 or more (i.e, info, warn, and error) to
> > > syslog (journald is preferred over syslog if present). if we want to change
> > > the default we can ask libvirt guys to consider this request.. but what is
> > > it ? to continue with vdsm specific conf? we prefer to avoid that.
> > 
> > I'd like that you'll take it with PM.
> 
> It isn't relevant for PMs here. Also, customers/users don't even get to this
> log file anyway.
> But if you insist then, Scott - can we close this one?

Customers do use it and QE as a local representative of a customers uses it, especially when opening bugs to R&D and yes I insist that this one will be taken with PM for review.
Fixing performance issues should deal with enhancements in algorithms and code efficiency, but not disabling previously working functionalities.
IMHO.

It is easy enough to configure logging if you need it.

The cause of the efficiency problem WAS the logging, so no matter what
we improve in other areas we would still be hurt by IO writing logs.

I agreed with you at first but seeing Dan's point of view I abandoned this approach.
When you think about it, it does not make sense to turn on such a verbose log by default on a 3rd party package. It was set that way as a transition state only[1].

You are right that customers and GSS need to be informed, that's why we made a release note about this.

[1] From src:
# FIXME until we are confident with libvirt
#  integration, let us have a verbose log

To be honest,
there are 2 separate issues: verbosity and location.

While I'm 100% sure about verbosity, Not so sure about log location.
But than again, not my call.

Closing based on the fact that:

1) The default behavior was not optimal to the point that it caused a performance impact
2) There is a mechanism in place to collect libvirtd.log data using journalctl
3) There is a workaround in the event more verbose logging is required
4) There is proposed release note in the original BZ to notifiy uses of this change

Nikolai, please note that moving to ERROR by default was an explicit request of bug 1125237. If you believe that more information is needed for QE, please provide more details. Which information have you are missing now? Which verbosity level is needed by you, and why?

(In reply to Dan Kenigsberg from comment #34)
> Nikolai, please note that moving to ERROR by default was an explicit request
> of bug 1125237. If you believe that more information is needed for QE,
> please provide more details. Which information have you are missing now?
> Which verbosity level is needed by you, and why?

I need full verbosity to provide libvirt.log to R&D as they required, in case they need these logs and by default they do need them. Now I can't provide any logs without doing manual interventions to system, which makes a simple log gathering procedure of copy/paste, really complicated for the end customer.

why? you can use "journalctl -u libvirtd" as Mooli specified in comment #15
and it should also be included in the log collector output.

(In reply to Nikolai Sednev from comment #35)
> (In reply to Dan Kenigsberg from comment #34)
> > Nikolai, please note that moving to ERROR by default was an explicit request
> > of bug 1125237. If you believe that more information is needed for QE,
> > please provide more details. Which information have you are missing now?
> > Which verbosity level is needed by you, and why?
> 
> I need full verbosity to provide libvirt.log to R&D as they required, in
> case they need these logs and by default they do need them. Now I can't
> provide any logs without doing manual interventions to system, which makes a
> simple log gathering procedure of copy/paste, really complicated for the end
> customer.

I required a couple of times, to fully understand some misbehaviour of the host.
You're not the only one complaining about libvirt logs being now less practical to gather, and I think you have a point. That's why I opened
https://bugzilla.redhat.com/show_bug.cgi?id=1149656

I must also say, for the sake of completeness and transparency, that I ack'd the libvirtd logs reduction patch and I still fully support that move.
We just cannot go ahead forever with such verbose logging.

I also believe that for QE boxes re-enable fully verbose logging cause little harm and annoyance. I can help if needed.

(In reply to Francesco Romani from comment #37)
> (In reply to Nikolai Sednev from comment #35)
> > (In reply to Dan Kenigsberg from comment #34)
> > > Nikolai, please note that moving to ERROR by default was an explicit request
> > > of bug 1125237. If you believe that more information is needed for QE,
> > > please provide more details. Which information have you are missing now?
> > > Which verbosity level is needed by you, and why?
> > 
> > I need full verbosity to provide libvirt.log to R&D as they required, in
> > case they need these logs and by default they do need them. Now I can't
> > provide any logs without doing manual interventions to system, which makes a
> > simple log gathering procedure of copy/paste, really complicated for the end
> > customer.
> 
> I required a couple of times, to fully understand some misbehaviour of the
> host.
> You're not the only one complaining about libvirt logs being now less
> practical to gather, and I think you have a point. That's why I opened
> https://bugzilla.redhat.com/show_bug.cgi?id=1149656
> 
> I must also say, for the sake of completeness and transparency, that I ack'd
> the libvirtd logs reduction patch and I still fully support that move.
> We just cannot go ahead forever with such verbose logging.
> 
> I also believe that for QE boxes re-enable fully verbose logging cause
> little harm and annoyance. I can help if needed.

I never saw any Cisco or other vendor switch crashing because of the logs, only when debugging for all packets were enabled, I'm talking about PPC CPUs, here we have x86/x64 architecture, on top of that, logs may be asked to be directly reported to syslog server as a requirement from customer, then what?
If you enable verbose logging on a system, then you're turning it's performance automatically to low, so hosts with libvirt logs being collected should be also turned to maintenance mode?
If turning to maintenance, then what we have, non functional host under investigation=performance impact?

Understand me correctly, you decided to do things, it's your choice, but have you ever seen any system, in which logs influence performance? If so, then component may be not working correctly, as it reports too much problems?
IMHO.

its not that.. its just that vdsm should not configure external modules with unrelated to regular flow parameters under the legs. 
Vdsm should configure only what it must to have to work properly, and that should be also after manual confirmation (if doing it manually and not by ovirt-host-deploy).

therefore, configure libvirt log output is not part of the necessary parameters that vdsm needs to configure. it was wrong to configure it before, and it is wrong now even if you and other customers use to it. same as configuring libvirtd.conf on vdsmd start and not by vdsm-tool (if you're familiar with that discussion).

anyhow, this parameter should not be configured by vdsm automatically and can be modified by the user manually if required. I don't see any reason to change libvirt default. if you think the default is wrong its something else and this should be discussed in libvirt bug.