Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1745003

Summary: [OSP-15] Configure log filter to capture interactions between libvirt and QEMU
Product: Red Hat OpenStack Reporter: Martin Schuppert <mschuppe>
Component: openstack-tripleo-heat-templatesAssignee: Martin Schuppert <mschuppe>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: medium Docs Contact:
Priority: medium    
Version: 15.0 (Stein)CC: gveitmic, jhakimra, kchamart, mburns, mschuppe, ratailor
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-nova-14.4.1-0.20190830110426.b28b01a.el8ost openstack-tripleo-heat-templates-10.6.2-0.20190913130451.e95a183.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1717359
: 1745005 (view as bug list) Environment:
Last Closed: 2020-03-05 11:59:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1717359    
Bug Blocks: 1745005, 1745007    

Description Martin Schuppert 2019-08-23 13:01:34 UTC 
+++ This bug was initially created as a clone of Bug #1717359 +++

What?
-----

On all compute nodes, by default, enable the libvirt log filter
("1:qemu") to capture the QMP (QEMU Machine Protocol) traffic.  This
will show the QMP request and response flow between libvirt and QEMU.


Why?
----

Often times, when debugging issues that involve interactions between
Nova, libvirt and QEMU, we need to understand the precise requests that
libvirt is sending to QEMU, and the corresponding responses from QEMU.
Since we don't capture these interactions by default, we request them[*]
_after_ a bug is reported, at which point it might not be possible to
capture them for various reasons, or worse, the machine might no longer
be available.

Having this log filter enabled can be extremely useful in debugging a
range of scenarios: live migration (with shared and non-shared storage),
live snapshots, "volume migration", and anything that involves libvirt
sending QMP commands to QEMU.

Note that we're only selectively enabling a targetted log filter
("1:qemu").  It will slightly increase the log file size, but nothing
crazy.  This slight increase in log file size is acceptable, given the
benefits.


How?
----

When deploying compute nodes, make sure that TripleO sets the below two
configuration attributes in `/etc/libvirt/libvirtd.conf`.  The "1:qemu" 
log filter will capture the QMP traffic between libvirt and QEMU in the
specified log file, `libvirtd.log`:

    ...
    log_filters="1:qemu"
    log_outputs="1:file:/var/log/libvirt/libvirtd.log"
    ...


[*] https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt
Comment 1 Rajesh Tailor 2019-09-03 09:54:47 UTC 
The patch [1] and [2] are already in RHOS-15 in puppet-nova and openstack-tripleo-heat-teamplates repos.

1) For puppet-nova repo:
    $ git branch --contains 749e4fbb25d71fb04883bb70f61676cca71df6de
    * rhos-15.0-trunk-patches

From RHOS-15 branch:

    $ git show 749e4fbb25d71fb04883bb70f61676cca71df6de
    commit 749e4fbb25d71fb04883bb70f61676cca71df6de
    Author: Martin Schuppert <mschuppert>
    Date:   Wed Aug 14 13:46:55 2019 +0200

        Adds log_filters parameter to nova::compute::libvirt
    
        Defines a filter to select a different logging level
        for a given category log outputs, as specified in
        https://libvirt.org/logging.html. Default undef
    
        Change-Id: Ic48afe856ae60654d6cebf627b12509881933c59
        Related-Bug: #1840140
        (cherry picked from commit 0b0bd400659e5e545173417816ec346249f4860e)

[...]

2) For opentack-tripleo-heat-templates repo:
    $ git branch --contains 78d16ded1e06b872b1ebda1c0f67ceebf53b665a
    * rhos-15.0-trunk-patches

From RHOS-15 branch:

    $ git show 78d16ded1e06b872b1ebda1c0f67ceebf53b665a
    commit 78d16ded1e06b872b1ebda1c0f67ceebf53b665a
    Author: Martin Schuppert <mschuppert>
    Date:   Wed Aug 14 14:05:30 2019 +0200

        Adds LibvirtLogFilters to define a libvirtd filter
    
        LibvirtLogFilters to select a different logging level for a given
        category log outputs, as specified in https://libvirt.org/logging.html .
        Default:
    
        '1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util'
    
        Conflicts:
              deployment/nova/nova-libvirt-container-puppet.yaml
    
        Depends-On: Ic48afe856ae60654d6cebf627b12509881933c59
        Change-Id: I85b09ddeb61b2f3563f81eb423e8f05b18fe984a
        Closes-Bug: 1840140
        (cherry picked from commit 71516f2816aa9705bbe74533957557269094fd32)

[...]
Comment 4 errata-xmlrpc 2020-03-05 11:59:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0643