The `virt-admin` tool is now available for you to use to capture logs for reporting RHOSP bugs. This tool is useful for troubleshooting all libvirt and QEMU problems, as the logs provide the communications between libvirt and QEMU on the Compute nodes. You can use `virt-admin` to set the libvirt and QEMU debug log filters dynamically, without having to restart the `nova_libvirt` container.
Perform the following steps to enable libvirt and QEMU log filters on a Compute node:
1. Log into the `nova_libvirt` container on the Compute node:
----
$ sudo podman exec -it nova_libvirt /bin/bash
----
2. Specify the name and location of the log file to send `virt-admin` output to:
----
$ virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log"
----
3. Configure the filters you want to collect logs for:
----
$ virt-admin daemon-log-filters \
"1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util"
----
NOTE: When debugging issues with live migration, you must configure these filters on all source and destination Compute nodes.
5. Repeat your test. After debugging is complete, upload the `libvirtd.log` to a bug.
6. Disable the libvirt and QEMU log filters on the Compute nodes:
----
$ virt-admin daemon-log-filters ""
----
7. To confirm that the filters are removed, enter the following command:
----
$ virt-admin daemon-log-filters
----
This command returns an empty list when you have successfully removed the filters.
DescriptionKashyap Chamarthy
2020-08-19 14:08:37 UTC
What?
-----
Install the 'libvirt-admin' RPM by default in the 'nova_libvirt'
container: it brings us the useful `virt-admin` tool.
Why?
----
The `virt-admin` tool lets you set the libvirt debug log filters
dynamically, *without* having to the disruptive process of restarting
the 'libvirtd' process (i.e. restart of the 'nova_libvirt' container).
This is extremely useful for troubleshooting all libvirt/QEMU problems.
How?
----
Here is how to configure libvirt log filters dynamically:
a. Install the `virt-admin` tool, provided by 'libvirt-admin'
package:
$> dnf install libvirt-admin -y
b. Configure the libvirt log file:
$> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log"
c. Configure the most useful log filters. (They also let you
observe the communication between libvirt and QEMU -- e.g. QMP
commands libvirt sends to QEMU. They are useful to debug bugs
involving interactions between libvirt and QEMU):
$> virt-admin daemon-log-filters \
"1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util"
Ensure that the filters are actually applied by running
`virt-admin daemon-log-filters` (with no extra args), which
should show the log filters that are currently applied.
That's it; no need to restart the libvirt daemon. The
/var/log/libvirt/libvirtd.log will now start capturing the debug
logs with the applied filters.
d. Now, repeat the test.
e. Then attach the libvirtd.log (from all relevant hosts) as plain
text to the bug.
To turn off the dynamic log filters from the command-line:
$> virt-admin daemon-log-filters ""
Check that the filters are actually removed, by running `virt-admin
daemon-log-filters` (with no extra args), which should show an empty
list.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Red Hat OpenStack Platform 16.1.4 director bug fix advisory), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2021:0817
What? ----- Install the 'libvirt-admin' RPM by default in the 'nova_libvirt' container: it brings us the useful `virt-admin` tool. Why? ---- The `virt-admin` tool lets you set the libvirt debug log filters dynamically, *without* having to the disruptive process of restarting the 'libvirtd' process (i.e. restart of the 'nova_libvirt' container). This is extremely useful for troubleshooting all libvirt/QEMU problems. How? ---- Here is how to configure libvirt log filters dynamically: a. Install the `virt-admin` tool, provided by 'libvirt-admin' package: $> dnf install libvirt-admin -y b. Configure the libvirt log file: $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" c. Configure the most useful log filters. (They also let you observe the communication between libvirt and QEMU -- e.g. QMP commands libvirt sends to QEMU. They are useful to debug bugs involving interactions between libvirt and QEMU): $> virt-admin daemon-log-filters \ "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" Ensure that the filters are actually applied by running `virt-admin daemon-log-filters` (with no extra args), which should show the log filters that are currently applied. That's it; no need to restart the libvirt daemon. The /var/log/libvirt/libvirtd.log will now start capturing the debug logs with the applied filters. d. Now, repeat the test. e. Then attach the libvirtd.log (from all relevant hosts) as plain text to the bug. To turn off the dynamic log filters from the command-line: $> virt-admin daemon-log-filters "" Check that the filters are actually removed, by running `virt-admin daemon-log-filters` (with no extra args), which should show an empty list.