I was able to verify that reopenOnTruncate="on" works for me with rsyslog-8.1911.0-6.el8.x86_64. I found out that it did not use to work properly ([1]) and had been patched in the past. Not sure which rsyslog version is used in 16.1 container. Can you please get me a NVR used so I can check that the patch is in version used? [1] https://github.com/rsyslog/rsyslog/issues/1090
I was able to reproduce this in OSP16.1.6 with the following process, even after setting `reopenOnTruncate="on"` in `/var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf`. The problem is not a fault of rsyslog, and the `reopenOnTruncate` is the correct option to set to ensure the expected behaviour. However, adjustments to the mentioned config file are not being mounted into the rsyslog container when they are made manually, and thus `reopenOnTruncate` is never set in `/etc/rsyslog.d/50_openstack_logs.conf` within the container.
Relevant upstream patches: https://review.opendev.org/c/openstack/puppet-tripleo/+/790154 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/790177
Try the same workaround as mentioned above: 1. Within /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf on each node, add the reopenOnTruncate="on" parameter to each input block. For example: > input(type="imfile" > file="/var/log/containers/glance/api.log" > tag="openstack.glance.api" > startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) " > reopenOnTruncate="on" # <------------ new parameter > ) 2. Restart the rsyslog container with $ podman restart rsyslog 3. Verify that the configuration change took affect in the container: $ podman exec rsyslog cat /etc/rsyslog.d/50_openstack_logs.conf
I created the following Ansible playbook to insert the change in an idempotent way, since it needs to be done after each overcloud update for the time being: [stack.lab ~]$ cat ./insert-rsyslog-fix.yaml - name: This playbooks adds a custom line to the rsyslog configuration file hosts: Controller gather_facts: False become: True tasks: - name: Insert reopenOnTruncate on 50_openstack_logs.conf ansible.builtin.replace: path: /var/lib/config-data/puppet-generated/rsyslog/etc/rsyslog.d/50_openstack_logs.conf regexp: '^( file=.*$\n)(?! reopenOnTruncate="on")' replace: '\1 reopenOnTruncate="on"\n' register: fix - name: Restart rsyslog container shell: podman restart rsyslog when: fix.changed Run it from the undercloud using the inventory generated by tripleo: [stack.lab ~]$ ansible-playbook -i inventory.yaml ./insert-rsyslog-fix.yaml PLAY [This playbooks adds a custom line to the rsyslog configuration file] ********************************************************************************************************************************************************************************* TASK [Insert reopenOnTruncate on 50_openstack_logs.conf] *************************************************************************************************************************************************************************************************** changed: [overcloud-controller-0] changed: [overcloud-controller-2] changed: [overcloud-controller-1] TASK [Restart rsyslog container] *************************************************************************************************************************************************************************************************************************** changed: [overcloud-controller-2] changed: [overcloud-controller-0] changed: [overcloud-controller-1] PLAY RECAP ************************************************************************************************************************************************************************************************************************************************* overcloud-controller-0 : ok=2 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 overcloud-controller-1 : ok=2 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 overcloud-controller-2 : ok=2 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 [stack.lab ~]$
(In reply to Eric Nothen from comment #28) > I created the following Ansible playbook to insert the change in an > idempotent way, since it needs to be done after each overcloud update for > the time being: > > [stack.lab ~]$ cat ./insert-rsyslog-fix.yaml > - name: This playbooks adds a custom line to the rsyslog configuration file > hosts: Controller > hosts should be: hosts: Compute:Controller
tested on 16.1.8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0986