Bug 2169831

Disk on controller0 fills up with syslog logs in /var/log/messages*:

[root@leaf1-controller-1 log]# du -a /var/log | sort -n -r | head -n 10
9936860 /var/log
2684312 /var/log/messages-20230212
2683540 /var/log/messages-20230205
2310812 /var/log/messages-20230129
940728  /var/log/messages
650404  /var/log/messages-20230122
354744  /var/log/containers
143412  /var/log/containers/httpd
67392   /var/log/sa
49436   /var/log/journal/357b886700b24d5bb84c8e12b910bf6e

Logrotate config on the controller (/etc/logrotate.d/rsyslog):

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
    endscript
}

This should be very easy to reproduce:
1. Install undercloud
2. Provision networks
3. Provision VIPs
4. Import and inspect nodes
5. Provision nodes
6. Deploy Ceph
7. Deploy overcloud
8. Let it run for a month or so

When the syslog logs in /var/log/messages* eventually fill up the LVM volume mounted on /var/log, the controller will crash.

So, you're saying the size of the /var/log partition is too small right? You can adjust that:
https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/provisioning/baremetal_provision.html#grow-volumes-playbook

I'll move this BZ to the hardprov team to decide if the default size of that partition should be adjusted or not.

We didnt have this problem on OSP 16.

The recommended way to address this would be to tweak logrotate config so the /var/log/ partition does not fill up without further tweaking.

Alternate solution would be to update documentation with a big warning that the default size of the /var/log partition is really small and will fill up with logs pretty fast.

Closing this as a duplicate of Bz2165470.

* We are adding to documentation.
* In 17.1 thin pools are used so in that case it will be easier to increase the volume size after the fact.
* Logging can be tuned using THT parameters.

I do have some questions regarding your environment:
 - Is this running with debug logging enabled? (this is not the default, but most environments deployed for testing by tools such as infrared does have debug logging enabled by default)
   I belive ceph logs to /var/log/messages, and in debug mode it logs _alot_.

*** This bug has been marked as a duplicate of bug 2165470 ***