Description of problem:

When setting up log forwarding [0] and octavia log offload [1] at the same time, the log file in which octavia dumps amphora logs is not included on the default set of files that are forwarded to Splunk/Elastic/etc.

Similar to what I reported earlier for Horizon [2] and HAproxy, but the difference here is that the amphora log is optional and only exists if log offloading is enabled.

Version-Release number of selected component (if applicable):

16.x

How reproducible:

Always reproducible.

Steps to Reproduce:
1. Deploy RHOSP 16 with log forwarding and octavia log offloading

2. Check that the octavia_rsyslog container is running:

~~~
(overcloud) [stack@director ~]$ ansible -i inventory.yaml -m shell -a 'podman ps -f name=octavia_rsyslog' -b Controller
overcloud-controller-0 | CHANGED | rc=0 >>
CONTAINER ID  IMAGE                                                                  COMMAND      CREATED            STATUS                PORTS   NAMES
16ba34bd959e  director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-rsyslog:16.2  kolla_start  About an hour ago  Up About an hour ago          octavia_rsyslog
overcloud-controller-1 | CHANGED | rc=0 >>
CONTAINER ID  IMAGE                                                                  COMMAND      CREATED            STATUS                PORTS   NAMES
f5c0f9a4cdf2  director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-rsyslog:16.2  kolla_start  About an hour ago  Up About an hour ago          octavia_rsyslog
overcloud-controller-2 | CHANGED | rc=0 >>
CONTAINER ID  IMAGE                                                                  COMMAND      CREATED            STATUS                PORTS   NAMES
84c4d5df210f  director.ctlplane.localdomain:8787/rhosp-rhel8/openstack-rsyslog:16.2  kolla_start  About an hour ago  Up About an hour ago          octavia_rsyslog
~~~

3. Create an LB

~~~
(overcloud) [stack@director ~]$ openstack loadbalancer create --flavor activestandby --name testlb-stdby-01 --vip-subnet-id external
~~~

3. After some minutes, the amphora log file shows some content:

~~~
(overcloud) [stack@director ~]$ ansible -i inventory.yaml -m shell -a 'ls -l /var/log/containers/octavia-amphorae/' -b Controller
overcloud-controller-2 | CHANGED | rc=0 >>
total 0
overcloud-controller-0 | CHANGED | rc=0 >>
total 28
-rw-r--r--. 1 root root 28385 Jan 13 16:05 octavia-amphora.log
overcloud-controller-1 | CHANGED | rc=0 >>
total 0
~~~

4. However, this file is not included on the default set of files that are forwarded (but the rest of the octavia files are):

~~~
[root@overcloud-controller-0 ~]# egrep file= /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf
  file="/var/log/containers/cinder/cinder-scheduler.log"
  file="/var/log/containers/cinder/cinder-volume.log"
  file="/var/log/containers/glance/api.log"
  file="/var/log/containers/haproxy/haproxy.log"
  file="/var/log/containers/heat/heat_api_cfn.log"
  file="/var/log/containers/heat/heat_api.log"
  file="/var/log/containers/heat/heat-engine.log"
  file="/var/log/containers/horizon/horizon.log"
  file="/var/log/containers/keystone/keystone.log"
  file="/var/log/containers/neutron/server.log"
  file="/var/log/containers/nova/nova-api.log"
  file="/var/log/containers/nova/nova-conductor.log"
  file="/var/log/containers/nova/nova-metadata-api.log"
  file="/var/log/containers/nova/nova-scheduler.log"
  file="/var/log/containers/nova/nova-novncproxy.log"
  file="/var/log/containers/octavia/api.log"
  file="/var/log/containers/octavia/health-manager.log"
  file="/var/log/containers/octavia/housekeeping.log"
  file="/var/log/containers/octavia/worker.log"
  file="/var/log/pacemaker/pacemaker.log"
  file="/var/log/host/messages"
    errorfile="/var/log/rsyslog/omelasticsearch.log"
[root@overcloud-controller-0 ~]# 

Actual results:

/var/log/containers/octavia-amphorae/octavia-amphora.log is not on the list of files to be forwarded by rsyslog.

Expected results:

/var/log/containers/octavia-amphorae/octavia-amphora.log comes from a container deployed by TripleO, so it should be forwarded in the same way that they other logs from the rest of the containers, especially since the rest of the octavia files are already included.

Additional info:



[0] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/logging_monitoring_and_troubleshooting_guide/index#enabling-centralized-logging-during-deployment_installing-and-configuring-the-logs-service
[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/using_octavia_for_load_balancing-as-a-service/index#enable-lb-service-instance-admin-log-offload_manage-lb-service-instance-logs
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2034730
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2026290