Short: NOTABUG

Long:

Even though the configuration looks empty, the *default* log_level is 3,
i.e. warnings and errors, this is redirected to to systemd journal (on
systems that are run systemd, and to syslog on older systems). You can
notice this when you invoke:

   $ systemctl status libvirtd
    libvirtd.service - Virtualization daemon
       Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
       Active: active (running) since Wed 2014-02-05 22:45:05 IST; 11s ago
     Main PID: 32521 (libvirtd)
       CGroup: /system.slice/libvirtd.service
               ├─  821 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
               └─32521 /usr/sbin/libvirtd

    Feb 05 22:45:05 foohost.com systemd[1]: Started Virtualization daemon.
    Feb 05 22:45:06 foohost.com dnsmasq[821]: read /etc/hosts - 2 addresses
    Feb 05 22:45:06 foohost.com dnsmasq[821]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
    Feb 05 22:45:06 foohost.com dnsmasq-dhcp[821]: read /var/lib/libvirt/dnsmasq/default.hostsfile


Furthermore, there's a couple of different ways to enable various log
levels based on filters, etc.

- If you want logs to be redirected to a file, that can be expressed in
  /etc/libvirt/libvirtd.conf. 

   To log _everything_ (this spews LOTS of details, fills up your disk),
   add these 

        log_level = 1
        log_outputs = 1:file:/var/tmp/libvirtd.log

   to /etc/libvirt/libvirtd.conf, restart libvirtd.

- Alternatively, you can set environment variables (if set, this will
  take precedence over values specified in the configuration file):

    $ export LIBVIRT_DEBUG=1
    $ export  LIBVIRT_TRACE=1

More extensive details on logging filters are available here:

     http://libvirt.org/logging.html

Yogev, can you clarify if you intended to make "packstack" to enable 
more elaborate libvirt logging other than the default (which is warnings
and errors).

You absolutely don't ever want to set LIBVIRT_DEBUG env var or log_level settings. The amount of data these generate destroys performance and is impossible to find the meaningul info in the GBs of irrelevant junk.

IMHO libvirt log level should be upto the host admin to decide upon.

I'm going to change the $subj of this bugzilla to see if it makes it more palatable to folks.

We could expose a new config option to packstack (cmdline/conf file) that allows you to have packstack increase the verbosity level of libvirt if you set it.  But it would always be a manual option to use that packstack flag/config option, and by default libvirt would use it's default logging verbosity settings

just to add that I disagree with Dan. 
1. currently, no libvirt log is created during installation and it used to do that in the past so something has changed.  
2. ovirt libvirt logs are in debug level and rotates the logs. keeping it in debug is helpful to dev/qe/gss and users. 
3. it's better to have libvirt events logged because not all events can be reproduced like they do in devel environment. 
4. an admin can decided to remove any log but if we configure logging of cinder/glance/nova and so on, why would we not log libvirt which is an important component? if the admin decides to stop debugging it it's their problem.

(In reply to Dafna Ron from comment #5)
> just to add that I disagree with Dan. 

A related debate happened recently, upstream, to revert a DevStack change that made 'log_level=1' (in the end, it was reverted to a more fine-grained filters).

  http://lists.openstack.org/pipermail/openstack-dev/2014-January/024414.html

Quoting DanpB:
'''
  it is complete insanity to set 'log_level=1' on *any* hosts.
  The level of debug info that generates is so enourmous that you'd
  never wanted to look at it. Debugging from that would be like trying
  to find a specific piece of hay in a haystack. That original devstack
  change should just be completely reverted.

  It would be useful if Jenkins machines had an elevated log level,
  but even there you definitely don't want to set log_level=1. I
  would suggest that for Jenkin's machines *only* we want to try

     log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 1:util"
     log_outputs="1:file:/var/log/libvirt/libvirtd.log"
'''

> 1. currently, no libvirt log is created during installation and it used to
> do that in the past so something has changed.  

Two things

 - You can do "tail -f foo.log" style monitoring, by default, this way:

        $  journalctl -f --unit=libvirtd

 - And, Some amount of guest specific logs go  /var/log/libvirt/qemu/guest.log


> 2. ovirt libvirt logs are in debug level and rotates the logs. keeping it in
> debug is helpful to dev/qe/gss and users. 

In my experience, more specific, targeted log filtering was helpful to narrow down issues.

> 3. it's better to have libvirt events logged because not all events can be
> reproduced like they do in devel environment. 

As you can notice above, "event" layer logs (and several other layers -
QEMU, util, security) can be easily configured.

> 4. an admin can decided to remove any log but if we configure logging of
> cinder/glance/nova and so on, why would we not log libvirt which is an
> important component? if the admin decides to stop debugging it it's their
> problem.

Even for Cinder/Glance, etc - VERBOSE/DEBUG  are not enabled by default you need to explicitly enable them.

I think you're saying you need a log file to be present by default with no need for explicit configuration -- that seems reasonably fair.

(In reply to Dafna Ron from comment #5)
> just to add that I disagree with Dan. 
> 1. currently, no libvirt log is created during installation and it used to
> do that in the past so something has changed.  

All logs go to the systemd journal by default now - syslog is being gracefully retired from life.

> 2. ovirt libvirt logs are in debug level and rotates the logs. keeping it in
> debug is helpful to dev/qe/gss and users. 

NB ovirt doesn't enable full libvirt debug level afaik - just some tailored subset - if it does that is very bad because it has a truely awful performance impact.

> 3. it's better to have libvirt events logged because not all events can be
> reproduced like they do in devel environment. 

'event' is an overloaded word - please clarify what you mean by "events" in this context

> 4. an admin can decided to remove any log but if we configure logging of
> cinder/glance/nova and so on, why would we not log libvirt which is an
> important component? if the admin decides to stop debugging it it's their
> problem.

IME it is sufficient to ask for specific debug options to be enabled at the time they're actually required, rather than having it turned on permanently. Taking logs at time of need means we can tailor the info collected to match what we actually need.

(In reply to Daniel Berrange from comment #7)
> (In reply to Dafna Ron from comment #5)
> > just to add that I disagree with Dan. 
> > 1. currently, no libvirt log is created during installation and it used to
> > do that in the past so something has changed.  
> 
> All logs go to the systemd journal by default now - syslog is being
> gracefully retired from life.

libvirtd.log was always created under /var/log/libvirt/ (until now). 
> 
> > 2. ovirt libvirt logs are in debug level and rotates the logs. keeping it in
> > debug is helpful to dev/qe/gss and users. 
> 
> NB ovirt doesn't enable full libvirt debug level afaik - just some tailored
> subset - if it does that is very bad because it has a truely awful
> performance impact.

I checked last night on an rhevm setup and logs are in debug (no one changed it manually, it's deployed in debug as it always have). 

> 
> > 3. it's better to have libvirt events logged because not all events can be
> > reproduced like they do in devel environment. 
> 
> 'event' is an overloaded word - please clarify what you mean by "events" in
> this context

event -> when a domain dies/fails to be migrated or any issue that involves the instances (qemu logs don't always have enough info). 

> 
> > 4. an admin can decided to remove any log but if we configure logging of
> > cinder/glance/nova and so on, why would we not log libvirt which is an
> > important component? if the admin decides to stop debugging it it's their
> > problem.
> 
> IME it is sufficient to ask for specific debug options to be enabled at the
> time they're actually required, rather than having it turned on permanently.
> Taking logs at time of need means we can tailor the info collected to match
> what we actually need.

(In reply to Dafna Ron from comment #8)
> (In reply to Daniel Berrange from comment #7)
> > (In reply to Dafna Ron from comment #5)
> > > just to add that I disagree with Dan. 
> > > 1. currently, no libvirt log is created during installation and it used to
> > > do that in the past so something has changed.  
> > 
> > All logs go to the systemd journal by default now - syslog is being
> > gracefully retired from life.
> 
> libvirtd.log was always created under /var/log/libvirt/ (until now). 

That's on legacy RHEL-6 platforms. On Fedora / RHEL-7 systemd journal is the preferred approach for any logging in the future because it gives you well structured data which can be more easily analysed/queried that plain text log files. Syslog / plain text logs are something to be avoided in any default configuration.

binary logs are for developers :) what about the rest of us?

(In reply to Dafna Ron from comment #10)
> binary logs are for developers :) what about the rest of us?

The journalctl tool for querying the logs displays them in plain text just fine.

(In reply to Daniel Berrange from comment #11)
> (In reply to Dafna Ron from comment #10)
> > binary logs are for developers :) what about the rest of us?
> 
> The journalctl tool for querying the logs displays them in plain text just
> fine.

I read some discussions on line that there are performance issues with it. 
but I think we are digressing here :) if Red Hat has decided to go towards systemd journal I think this is important information for future testing and support.

Adding Yaniv.

(In reply to Dafna Ron from comment #12)
> (In reply to Daniel Berrange from comment #11)
> > (In reply to Dafna Ron from comment #10)
> > > binary logs are for developers :) what about the rest of us?
> > 
> > The journalctl tool for querying the logs displays them in plain text just
> > fine.
> 
> I read some discussions on line that there are performance issues with it. 

Honestly, I wouldn't make "some discussions on line" as a basis to
conclude there are "performance issues" without any systematic
measurement to back it up with numbers.

> but I think we are digressing here :) if Red Hat has decided to go towards
> systemd journal I think this is important information for future testing and
> support.

[a] systemd journal already _is_ part of RHEL7:

    https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/System_Administrators_Guide/s1-Using_the_Journal.html


[b] From Fedora-20 and above, traditional syslog service will not be 
even _installed_ (you would have noticed there's no /var/log/messages
file on your F20 machine)

    https://fedoraproject.org/wiki/Changes/NoDefaultSyslog


(Finally, I redact my incorrect statement in a previous commentthat it's 
a reasonable request to ask for a default libvirt log file. Thanks Dan,
for correcting.)

I agree with  Daniel and Kashyap. Using journalctl is pretty straightforward and even QE should learn it. 

Nevertheless it seems that there are some people who need to run libvirtd in debug mode ( or complete insanity mode as was stated above :) ), so we can implement a special switch just for one service to run in debug mode (WITH BIG RED WARNING of course). But this is definitely not an urgent thing ...

Patch proposed upstream:

https://review.openstack.org/#/c/166345/

I think this can be closed as WONTFIX/NOTABUG or even

"How I learned to stop worrying and love the systemd"