Bug 1462144

Summary: RabbitMQ logs are not rotated properly
Product: Red Hat OpenStack Reporter: Pablo Iranzo Gómez <pablo.iranzo>
Component: rabbitmq-serverAssignee: Peter Lemenkov <plemenko>
Status: CLOSED CURRENTRELEASE QA Contact: Udi Shkalim <ushkalim>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0 (Kilo)CC: ahirshbe, apevec, djuran, ipetrova, jbuchta, jeckersb, lhh, madgupta, mburns, michele, pablo.iranzo, pbarta, plemenko, rcernin, sclewis, srevivo, ushkalim
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1405211
: 1673827 (view as bug list) Environment:
Last Closed: 2018-07-20 12:15:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1339224, 1405211    
Bug Blocks: 1371521, 1673827    

Description Pablo Iranzo Gómez 2017-06-16 10:09:54 UTC
Original bug:


The rabbitmq-server logs are not rotated properly

/var/log/rabbitmq/*.log {
        weekly
        missingok
        rotate 20
        compress
        delaycompress
        notifempty
        sharedscripts
        postrotate
            /usr/sbin/rabbitmqctl -q rotate_logs
        endscript
}

After the rotation using fuser shows the rabbitmq is still using the rabbit 

[root@overcloud-controller-0 ~]# fuser -v /var/log/rabbitmq/*
                     USER        PID ACCESS COMMAND
/var/log/rabbitmq/rabbit:
                     rabbitmq  22869 F.... beam.smp
/var/log/rabbitmq/rabbit:
                     rabbitmq  22869 F.... beam.smp
/var/log/rabbitmq/startup_err:
                     root      22842 F.... rabbitmq-server
                     root      22867 F.... su
                     rabbitmq  22869 F.... beam.smp
                     rabbitmq  23245 F.... inet_gethost
                     rabbitmq  23246 F.... inet_gethost
/var/log/rabbitmq/startup_log:
                     root      22842 F.... rabbitmq-server
                     root      22867 F.... su
                     rabbitmq  22869 F.... beam.smp

After the errata on masterbug (1405211) there is still issues:


Rabbit MQ version: 3.3.5-31

Status:
Logrorate hasn't changed and still process using the rotated files instead of new ones:


~~~~
rabbitmq  9516  9510  4 May13 ?        1-01:09:08 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@esah-ost1-cc01p -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true},{raw,6,18,<<5000:64/native>>}] -kernel inet_default_listen_options [{raw,6,18,<<5000:64/native>>}] -rabbit tcp_listeners [{"172.24.13.16",5672}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@esah-ost1-cc01p-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@esah-ost1-cc01p"
root     11870 29338  0 12:07 pts/0    00:00:00 grep --color=auto rabbit

[root@esah-ost1-cc01p rabbitmq]# lsof rabbit
[root@esah-ost1-cc01p rabbitmq]# lsof rabbit
[root@esah-ost1-cc01p rabbitmq]# lsof rabbit
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
beam.smp 9516 rabbitmq   10w   REG    8,2  2445512 17689604 rabbit
[root@esah-ost1-cc01p rabbitmq]# lsof rabbit
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
beam.smp 9516 rabbitmq   11w   REG    8,2    13766 17698260 rabbit

Comment 1 Red Hat Bugzilla Rules Engine 2017-06-16 10:10:13 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 2 Red Hat Bugzilla Rules Engine 2017-06-16 10:13:54 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 8 Peter Lemenkov 2017-10-02 13:45:37 UTC
As far as I know the only remaining issue related to logrotation is one related to selinux. See bug 1426600 (Fedora EPEL) or bug 1413775 (RHOS9, RHOS10). Could yoyu please check that openstack-selinux is installed? I also believe that switching SELinux to permissive work will help but this might not be exactly what you wanted.

I'm working on this SElinux issue.