+++ This bug was initially created as a clone of Bug #1717359 +++ What? ----- On all compute nodes, by default, enable the libvirt log filter ("1:qemu") to capture the QMP (QEMU Machine Protocol) traffic. This will show the QMP request and response flow between libvirt and QEMU. Why? ---- Often times, when debugging issues that involve interactions between Nova, libvirt and QEMU, we need to understand the precise requests that libvirt is sending to QEMU, and the corresponding responses from QEMU. Since we don't capture these interactions by default, we request them[*] _after_ a bug is reported, at which point it might not be possible to capture them for various reasons, or worse, the machine might no longer be available. Having this log filter enabled can be extremely useful in debugging a range of scenarios: live migration (with shared and non-shared storage), live snapshots, "volume migration", and anything that involves libvirt sending QMP commands to QEMU. Note that we're only selectively enabling a targetted log filter ("1:qemu"). It will slightly increase the log file size, but nothing crazy. This slight increase in log file size is acceptable, given the benefits. How? ---- When deploying compute nodes, make sure that TripleO sets the below two configuration attributes in `/etc/libvirt/libvirtd.conf`. The "1:qemu" log filter will capture the QMP traffic between libvirt and QEMU in the specified log file, `libvirtd.log`: ... log_filters="1:qemu" log_outputs="1:file:/var/log/libvirt/libvirtd.log" ... [*] https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt
The patch [1] and [2] are already in RHOS-15 in puppet-nova and openstack-tripleo-heat-teamplates repos. 1) For puppet-nova repo: $ git branch --contains 749e4fbb25d71fb04883bb70f61676cca71df6de * rhos-15.0-trunk-patches From RHOS-15 branch: $ git show 749e4fbb25d71fb04883bb70f61676cca71df6de commit 749e4fbb25d71fb04883bb70f61676cca71df6de Author: Martin Schuppert <mschuppert> Date: Wed Aug 14 13:46:55 2019 +0200 Adds log_filters parameter to nova::compute::libvirt Defines a filter to select a different logging level for a given category log outputs, as specified in https://libvirt.org/logging.html. Default undef Change-Id: Ic48afe856ae60654d6cebf627b12509881933c59 Related-Bug: #1840140 (cherry picked from commit 0b0bd400659e5e545173417816ec346249f4860e) [...] 2) For opentack-tripleo-heat-templates repo: $ git branch --contains 78d16ded1e06b872b1ebda1c0f67ceebf53b665a * rhos-15.0-trunk-patches From RHOS-15 branch: $ git show 78d16ded1e06b872b1ebda1c0f67ceebf53b665a commit 78d16ded1e06b872b1ebda1c0f67ceebf53b665a Author: Martin Schuppert <mschuppert> Date: Wed Aug 14 14:05:30 2019 +0200 Adds LibvirtLogFilters to define a libvirtd filter LibvirtLogFilters to select a different logging level for a given category log outputs, as specified in https://libvirt.org/logging.html . Default: '1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util' Conflicts: deployment/nova/nova-libvirt-container-puppet.yaml Depends-On: Ic48afe856ae60654d6cebf627b12509881933c59 Change-Id: I85b09ddeb61b2f3563f81eb423e8f05b18fe984a Closes-Bug: 1840140 (cherry picked from commit 71516f2816aa9705bbe74533957557269094fd32) [...]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0643