Bug 998943 - Dist-geo-rep: logrotate utility command rotates the geo-replication log file but doesn't open a new one to write.
Dist-geo-rep: logrotate utility command rotates the geo-replication log file ...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: geo-replication (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Aravinda VK
M S Vishwanath Bhat
: ZStream
Depends On:
Blocks: 1012776
  Show dependency treegraph
 
Reported: 2013-08-20 07:51 EDT by M S Vishwanath Bhat
Modified: 2016-05-31 21:56 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the geo-replication log-rotate utility rotated the existing log files, but did not open a new file for writing further logs, leading the further logs to be missed. With this update release, the geo-replication log-rotate module creates a new file after rotating existing file, thus not loosing any logs.
Story Points: ---
Clone Of:
: 1012776 (view as bug list)
Environment:
Last Closed: 2013-11-27 10:31:57 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description M S Vishwanath Bhat 2013-08-20 07:51:11 EDT
Description of problem:
rotate the geo-replication log using the logrotate command. This rotates the geo-repliaction log file but doesn't open a new file to write to.

Version-Release number of selected component (if applicable):
[root@harrier ~]# rpm -q glusterfs
glusterfs-3.4.0.20rhs-2.el6rhs.x86_64


How reproducible:
Always


Steps to Reproduce:
1. Setup a geo-replication session and keep syncing some data, so the the log is being written to.
2. On any node, run logrotate -f /etc/logrotate.d/glusterfs-geoprep

Actual results:
The log files are rotated but new files are not being opened for writing.

Expected results:
The log file should be rotated and new files should be opened for writing.

Additional info:


There is one more issue which happens when logrotate(8) is run. That is the node on which this is being executed will go to N/A in the status/status detail.

But that is being taken care in other bug https://bugzilla.redhat.com/show_bug.cgi?id=982471
Comment 2 Venky Shankar 2013-08-20 08:19:45 EDT
MS,

Could you do the operations by hand and see if the logs are rotated and gsyncd works as expected. Maybe that will figure out if it's something with logrotate or with gsyncd handling rotation signals.
Comment 3 M S Vishwanath Bhat 2013-08-21 08:04:36 EDT
It doesn't work as expected even when I try by sending the signal SIGSTOP and SIGCONT manually. The log will be rotated but new file is not being opened for writing. Seems there is some issue with gsyncd while handling signal.
Comment 4 Amar Tumballi 2013-09-16 08:19:54 EDT
Aravinda, can you look into this?
Comment 5 Amar Tumballi 2013-09-19 02:41:24 EDT
https://code.engineering.redhat.com/gerrit/#/c/13056
Comment 6 Gowrishankar Rajaiyan 2013-10-08 04:41:09 EDT
Fixed in version please.
Comment 7 M S Vishwanath Bhat 2013-10-18 05:23:56 EDT
There will be two logs in the geo-replication log directory. One will be by geo-rep monitor and worker processes and other is by the auxiliary mount process used by geo-rep.

As of now in master volume, both the log files are being rotated. When I say rotated, they are renamed from *.log to *.log.1) But only one of them (gsyncd log file) is being re-opened for writing. The other one (auxiliary gluster client log) is not being re-opened. I am not sure the reason is because there is bug or if the process is Idle and there is nothing to write to the log file.

At the slave side, the log files are rotated but are not opened again. None of the log files are being re-opened at the slave side. So re-opening the bug.
Comment 8 Aravinda VK 2013-10-18 07:40:26 EDT
*slave.gluster.log file is used by glusterfs --aux-gfid-mount, so need to send message to glusterfs process. I will update the /etc/logrotate.d/glusterfs-georep file.

I am still looking into the logrotate issue of *slave.log.
Comment 10 Aravinda VK 2013-10-30 07:48:19 EDT
gsync spawns glusterfs aux-mount and python logrotate handling will not work for external processes. SIGHUP is re for auxiliary mount process. 

Patch sent by Vishwanath Bhat.
https://code.engineering.redhat.com/gerrit/#/c/14265/
Comment 11 M S Vishwanath Bhat 2013-11-05 01:00:15 EST
Now this issue is fixed...

Tested in version: glusterfs-3.4.0.38rhs-1.el6rhs.x86_64

Before executing the logrotate command 

[root@spitfire ~]# find /var/log//glusterfs/geo-replication* -type f
/var/log//glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.42.224%3Agluster%3A%2F%2F127.0.0.1%3Aslave.log
/var/log//glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.42.224%3Agluster%3A%2F%2F127.0.0.1%3Aslave.%2Frhs%2Fbricks%2Fbrick0.gluster.log
/var/log//glusterfs/geo-replication-slaves/slave.log


After executing the logrotate command and waiting for some time (till there is something for gsync to write to the log file)

[root@spitfire glusterfs-deploy-scripts]# find /var/log/glusterfs/geo-replication/master/
/var/log/glusterfs/geo-replication/master/
/var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.42.224%3Agluster%3A%2F%2F127.0.0.1%3Aslave.%2Frhs%2Fbricks%2Fbrick0.gluster.log.1
/var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.42.224%3Agluster%3A%2F%2F127.0.0.1%3Aslave.log
/var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.42.224%3Agluster%3A%2F%2F127.0.0.1%3Aslave.log.1
/var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.42.224%3Agluster%3A%2F%2F127.0.0.1%3Aslave.%2Frhs%2Fbricks%2Fbrick0.gluster.log


The original log files have been rotated and new log files are being opened.

This works fine for all the logs in master. But in slave only the logfile opened by the glusterfs process will be rotated. Other log file opened by python process will not be rotated.


Amar, Is this Okay? Not rotating the log file in slave.
Comment 12 Amar Tumballi 2013-11-11 04:59:56 EST
Can we open another bug for slave log-rotate behavior and VERIFY this bug?
Comment 13 M S Vishwanath Bhat 2013-11-11 11:25:22 EST
(In reply to Amar Tumballi from comment #12)
> Can we open another bug for slave log-rotate behavior and VERIFY this bug?

From my discussion with Aravinda, it seems like slave files are being written only once during the initial setting up. But they won't be logged later since there will be no gsync process running.

So moving this bug to verified. Will open a new bug if required.

Tested in version: glusterfs-3.4.0.38rhs-1.el6rhs.x86_64
Comment 15 errata-xmlrpc 2013-11-27 10:31:57 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html

Note You need to log in before you can comment on or make changes to this bug.