The rabbitmq-server logs are not rotated properly /var/log/rabbitmq/*.log { weekly missingok rotate 20 compress delaycompress notifempty sharedscripts postrotate /usr/sbin/rabbitmqctl -q rotate_logs endscript } After the rotation using fuser shows the rabbitmq is still using the rabbit [root@overcloud-controller-0 ~]# fuser -v /var/log/rabbitmq/* USER PID ACCESS COMMAND /var/log/rabbitmq/rabbit: rabbitmq 22869 F.... beam.smp /var/log/rabbitmq/rabbit: rabbitmq 22869 F.... beam.smp /var/log/rabbitmq/startup_err: root 22842 F.... rabbitmq-server root 22867 F.... su rabbitmq 22869 F.... beam.smp rabbitmq 23245 F.... inet_gethost rabbitmq 23246 F.... inet_gethost /var/log/rabbitmq/startup_log: root 22842 F.... rabbitmq-server root 22867 F.... su rabbitmq 22869 F.... beam.smp Version-Release number of selected component (if applicable): rabbitmq-server-3.3.5-16.el7ost.noarch How reproducible: always Steps to Reproduce: 1. wait until logs are rotated 2. 3. Actual results: Wrong file is use /var/log/rabbitmq/rabbit Expected results: Correct file in use /var/log/rabbitmq/rabbit
The same problem can also be observed in OSP 8, with rabbitmq-server-3.3.5-22.el7ost. To reproduce, just deploy an OpenStack using OSP-d and then leave to matute for a week....
Cannot confirm this. II've just tried to rotate logs with /usr/sbin/logrotate /etc/logrotate.conf And everything was rotated just fine (rabbitmq continues with a new log and no PID is holding older log).
I've also had prblems reproducing this when I manually try to trigger the log rotation. If I however deploy an OverCloud from OSP-d 8 and then let it sit for a week, this problem does occur every time.
Hello All! Please try this build ( rabbitmq-server-3.3.5-26.el7ost ) - it fixes related issues in logging subsystem and very likely fixes this issue as well.
(In reply to Peter Lemenkov from comment #11) > Hello All! > > Please try this build ( rabbitmq-server-3.3.5-26.el7ost ) - it fixes related > issues in logging subsystem and very likely fixes this issue as well. Hey Peter!! :) I don't think that's just available yet. At least I don't see it in our Portal: https://access.redhat.com/downloads/content/rabbitmq-server/3.3.5-22.el7ost/noarch/fd431d51/package Do you have any info about that? Best, Irina PS Many thanks for taking a look.
Verified, OSPD-8 using rabbitmq-server-3.3.5-27.el7ost.noarch [root@controller-2 ~]# fuser -v /var/log/rabbitmq/* USER PID ACCESS COMMAND /var/log/rabbitmq/rabbit: rabbitmq 20819 F.... beam.smp /var/log/rabbitmq/rabbit: rabbitmq 20819 F.... beam.smp /var/log/rabbitmq/startup_err: root 20794 F.... rabbitmq-server root 20811 F.... su rabbitmq 20819 F.... beam.smp rabbitmq 20954 F.... inet_gethost rabbitmq 20955 F.... inet_gethost /var/log/rabbitmq/startup_log: root 20794 F.... rabbitmq-server root 20811 F.... su rabbitmq 20819 F.... beam.smp [root@controller-2 rabbitmq]# rabbitmqctl rotate_logs .1 Rotating logs to files with suffix ".1" ... ...done. [root@controller-2 rabbitmq]# ll total 236 -rw-r-----. 1 rabbitmq rabbitmq 0 Oct 9 04:11 rabbit -rw-r-----. 1 rabbitmq rabbitmq 43304 Oct 9 04:11 rabbit.1 -rw-r-----. 1 rabbitmq rabbitmq 189340 Oct 6 16:41 rabbit -rw-r-----. 1 rabbitmq rabbitmq 0 Oct 9 04:11 rabbit -rw-r-----. 1 rabbitmq rabbitmq 0 Oct 9 04:11 rabbit.1 -rw-r-----. 1 root root 0 Oct 9 03:36 startup_err -rw-r-----. 1 root root 698 Oct 9 03:36 startup_log [root@controller-2 rabbitmq]# fuser -v /var/log/rabbitmq/* USER PID ACCESS COMMAND /var/log/rabbitmq/rabbit: rabbitmq 20819 F.... beam.smp /var/log/rabbitmq/rabbit: rabbitmq 20819 F.... beam.smp /var/log/rabbitmq/startup_err: root 20794 F.... rabbitmq-server root 20811 F.... su rabbitmq 20819 F.... beam.smp rabbitmq 20954 F.... inet_gethost rabbitmq 20955 F.... inet_gethost /var/log/rabbitmq/startup_log: root 20794 F.... rabbitmq-server root 20811 F.... su rabbitmq 20819 F.... beam.smp [root@controller-2 rabbitmq]# rabbitmqctl rotate_logs .2 Rotating logs to files with suffix ".2" ... ...done. [root@controller-2 rabbitmq]# fuser -v /var/log/rabbitmq/* USER PID ACCESS COMMAND /var/log/rabbitmq/rabbit: rabbitmq 20819 F.... beam.smp /var/log/rabbitmq/rabbit: rabbitmq 20819 F.... beam.smp /var/log/rabbitmq/startup_err: root 20794 F.... rabbitmq-server root 20811 F.... su rabbitmq 20819 F.... beam.smp rabbitmq 20954 F.... inet_gethost rabbitmq 20955 F.... inet_gethost /var/log/rabbitmq/startup_log: root 20794 F.... rabbitmq-server root 20811 F.... su rabbitmq 20819 F.... beam.smp [root@controller-2 rabbitmq]# ll total 240 -rw-r-----. 1 rabbitmq rabbitmq 0 Oct 9 04:13 rabbit -rw-r-----. 1 rabbitmq rabbitmq 43304 Oct 9 04:11 rabbit.1 -rw-r-----. 1 rabbitmq rabbitmq 74 Oct 9 04:13 rabbit.2 -rw-r-----. 1 rabbitmq rabbitmq 189340 Oct 6 16:41 rabbit -rw-r-----. 1 rabbitmq rabbitmq 0 Oct 9 04:13 rabbit -rw-r-----. 1 rabbitmq rabbitmq 0 Oct 9 04:11 rabbit.1 -rw-r-----. 1 rabbitmq rabbitmq 0 Oct 9 04:13 rabbit.2 -rw-r-----. 1 root root 0 Oct 9 03:36 startup_err -rw-r-----. 1 root root 698 Oct 9 03:36 startup_log [root@controller-2 rabbitmq]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2712.html