Bug 2069619

I checked 16.2.2 vs 16.2.1 - no changes in libvirt or qemu-kvm are relevant.  (I realized later this was a change in behavior since 16.1, which is substantially different from 16.2; it would take some time to isolate those changes)

Okay, I'm running out of ideas here.  What have we ruled out so far:

(1) It's not a QEMU issue

(2) It is also not a libvirt issue

(3) it's not SELinux -- the machine is already running in "permissive" mode

(4) I have also tried changing the SELinux "type" from 'container_ro_file_t' to 'virt_log_t' on /var/log/libvirt/qemu/ directory: 

    # Inside 'nova_libvirt' container, as "root"
    $> cd /var/log/libvirt/
    $> chcon -R -t virt_log_t qemu/
    $> ls -lsrtZ /var/log/libvirt/  | grep qemu
    0 drwxr-xr-x. 2 root root system_u:object_r:virt_log_t:s0                 6 Mar 21 15:26 qemu  

But that doesn't help either, as we're already running in permissive.  So I changed it back to "container_ro_file_t":

    $> chcon -R -t container_ro_file_t qemu/                                                                                                                        

(5) The _file_ permissions on /var/log/libvirt/qemu itself look fine, with "root:" and 755.  (Although on my F35 box, qemu/ directory has 700 permissions)


So, overall the issue is "somewhere" in the way we're launching the 'nova_libvirt' container.

Let's check with Martin Schuppert for some more input here.

The logs are in virtlogd container and this container miss the host volume mount:
[root@computeamdsev-1 ~]# podman exec -it nova_virtlogd /bin/bash
[root@computeamdsev-1 /]# ls -la /var/log/libvirt/qemu/ | head -n 10
total 1484
drwx------. 1 root root  8192 Mar 28 18:03 .
drwx------. 1 root root    18 Mar 10 23:47 ..
-rw-------. 1 root root  8592 Mar 29 13:56 el9vm1.log
-rw-------. 1 root root  6135 Mar 25 12:43 instance-0000007c.log
-rw-------. 1 root root  6135 Mar 25 12:43 instance-0000007d.log
-rw-------. 1 root root  6134 Mar 25 12:44 instance-00000080.log
-rw-------. 1 root root  6135 Mar 25 12:43 instance-00000082.log
-rw-------. 1 root root  6135 Mar 25 12:43 instance-00000083.log
-rw-------. 1 root root  7206 Mar 25 12:44 instance-00000085.log

libvirt container:
            {
                "Type": "bind",
                "Source": "/var/log/containers/libvirt",
                "Destination": "/var/log/libvirt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "shared"
            },

the virtlogd container is missing this.

probably related to https://github.com/openstack/tripleo-heat-templates/commit/41eca11495a94cbcab876e323761e6d947ae0bf2

(In reply to Martin Schuppert from comment #6)
> probably related to
> https://github.com/openstack/tripleo-heat-templates/commit/
> 41eca11495a94cbcab876e323761e6d947ae0bf2

It looks like the bind-mount was accidentally in this patch[1].  See the line that was removed in nova-libvirt-container-puppet.yaml file[2]:

    - /var/log/libvirt/qemu:/var/log/libvirt/qemu:ro

We need to add this back.

And ship a fix as quickly as possible.  Because even though the logs are in "nova_virtlogd" container, you will lose the logs
once you restart the container -- which is terrible from a debugging point of view, as we lose valuable logs.

[1] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/817733/
    Fix libvirt/qemu logs location

[2] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/817733/1/deployment/nova/nova-libvirt-container-puppet.yaml

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4793