Bug 1939964

I was able to verify that reopenOnTruncate="on" works for me with rsyslog-8.1911.0-6.el8.x86_64. I found out that it did not use to work properly ([1]) and had been patched in the past. Not sure which rsyslog version is used in 16.1 container. Can you please get me a NVR used so I can check that the patch is in version used?

[1] https://github.com/rsyslog/rsyslog/issues/1090

I was able to reproduce this in OSP16.1.6 with the following process, even after setting `reopenOnTruncate="on"` in `/var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf`.

The problem is not a fault of rsyslog, and the `reopenOnTruncate` is the correct option to set to ensure the expected behaviour. However, adjustments to the mentioned config file are not being mounted into the rsyslog container when they are made manually, and thus `reopenOnTruncate` is never set in `/etc/rsyslog.d/50_openstack_logs.conf` within the container.

Relevant upstream patches:
https://review.opendev.org/c/openstack/puppet-tripleo/+/790154
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/790177

Try the same workaround as mentioned above: 

1. Within /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf on each node, add the reopenOnTruncate="on" parameter to each input block.

For example:

> input(type="imfile"
>   file="/var/log/containers/glance/api.log"
>   tag="openstack.glance.api"
>   startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) "
>   reopenOnTruncate="on"  # <------------ new parameter
> )

2. Restart the rsyslog container with

$ podman restart rsyslog

3. Verify that the configuration change took affect in the container:

$ podman exec rsyslog cat /etc/rsyslog.d/50_openstack_logs.conf

I created the following Ansible playbook to insert the change in an idempotent way, since it needs to be done after each overcloud update for the time being:

[stack.lab ~]$ cat ./insert-rsyslog-fix.yaml 
- name: This playbooks adds a custom line to the rsyslog configuration file
  hosts: Controller
  gather_facts: False
  become: True

  tasks:
    - name: Insert reopenOnTruncate on 50_openstack_logs.conf
      ansible.builtin.replace:
        path: /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf
        regexp: '^(  file=.*$\n)(?!  reopenOnTruncate="on")'
        replace: '\1  reopenOnTruncate="on"\n'
      register: fix

    - name: Restart rsyslog container
      shell: podman restart rsyslog
      when: fix.changed


Run it from the undercloud using the inventory generated by tripleo:

[stack.lab ~]$ ansible-playbook -i inventory.yaml ./insert-rsyslog-fix.yaml

PLAY [This playbooks adds a custom line to the rsyslog configuration file] *********************************************************************************************************************************************************************************

TASK [Insert reopenOnTruncate on 50_openstack_logs.conf] ***************************************************************************************************************************************************************************************************
changed: [overcloud-controller-0]
changed: [overcloud-controller-2]
changed: [overcloud-controller-1]

TASK [Restart rsyslog container] ***************************************************************************************************************************************************************************************************************************
changed: [overcloud-controller-2]
changed: [overcloud-controller-0]
changed: [overcloud-controller-1]

PLAY RECAP *************************************************************************************************************************************************************************************************************************************************
overcloud-controller-0     : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
overcloud-controller-1     : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
overcloud-controller-2     : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[stack.lab ~]$

(In reply to Eric Nothen from comment #28)
> I created the following Ansible playbook to insert the change in an
> idempotent way, since it needs to be done after each overcloud update for
> the time being:
> 
> [stack.lab ~]$ cat ./insert-rsyslog-fix.yaml 
> - name: This playbooks adds a custom line to the rsyslog configuration file
>   hosts: Controller
> 

hosts should be:

  hosts: Compute:Controller

tested on 16.1.8

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0986