Bug 1367665 - rotated FUSE mount log is using to populate the information after log rotate.
Summary: rotated FUSE mount log is using to populate the information after log rotate.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: build
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Milind Changire
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-17 07:36 UTC by Milind Changire
Modified: 2017-03-27 18:28 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.9.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1327952
Environment:
Last Closed: 2017-03-27 18:28:06 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Milind Changire 2016-08-17 07:36:00 UTC
+++ This bug was initially created as a clone of Bug #1327952 +++

Description of problem:
=======================
rotated FUSE mount log is using to populate the information instead of newly created file after log rotate.


Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.7.9-1.el7.x86_64


How reproducible:
=================
One time observed.// i think it's 100% reproducible.

Steps to Reproduce:
===================
1. FUSE mount one of the sample volume on the client 
2. Log rotate the volume mount log dynamically (OR)  wait for log rotate to happen // check log file has rotated and new log file has created.
2. Do some volume set operations for the mounted volume
3. Check volume set information is populated in newly created mount log file // file will be empty.

Actual results:
===============
Logs are populating to rotated log instead of writing to new log file.


Expected results:
=================
Once log rotated, next log info should populate to new log file.


Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-18 02:51:44 EDT ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Atin Mukherjee on 2016-05-05 07:43:33 EDT ---

I tried reproducing it by running logrotate command manually with
logrotate --force /etc/logrotate.d/glusterfs

But I could see the further logs get redirected to new mount log, not the rotated log file.

Please cross check.

And also if you hit it again, upload the sosreport.

--- Additional comment from Atin Mukherjee on 2016-05-09 23:53:46 EDT ---

Given that I've waited for few days for QE to get back, I am closing this bug since the same is not reproducible at our end. Please feel free to reopen if you hit the issue and attach relevant logs.

--- Additional comment from Byreddy on 2016-08-16 01:06 EDT ---



--- Additional comment from Byreddy on 2016-08-16 01:08:36 EDT ---

This issue is reproducible always.
I am reopening this bug and sos report is attached to debug the issue.

--- Additional comment from Byreddy on 2016-08-16 02:18:46 EDT ---

I am proposing this bug for 3.2 release because currently it's using single log file to populate the info even though log rotation happens, it's  not picking the rotated log to populate the info.

--- Additional comment from Atin Mukherjee on 2016-08-16 03:12:31 EDT ---

I retried the scenario as mentioned in comment 2 with both upstream and downstream latest bits and couldn't reproduce this issue. The only difference I see here is that I am not waiting logrotate to trigger the rotation gracefully, instead going for a force mode, but IMO that shouldn't make any difference. 

With that, may I ask the maintainer(s) of logging (libglusterfs) to chime in with their inputs?

--- Additional comment from Byreddy on 2016-08-16 06:27:42 EDT ---

"killall"  command package ( psmisc ) is not getting installed by default / not pulling during client packages installation, which is needed for  the gluster log files rotation.



Below is some info:

[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# yum install glusterfs glusterfs-fuse
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Resolving Dependencies
--> Running transaction check
---> Package glusterfs.x86_64 0:3.7.9-10.el7 will be installed
--> Processing Dependency: glusterfs-libs(x86-64) = 3.7.9-10.el7 for package: glusterfs-3.7.9-10.el7.x86_64
--> Processing Dependency: libgfxdr.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64
--> Processing Dependency: libglusterfs.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64
--> Processing Dependency: libgfrpc.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64
---> Package glusterfs-fuse.x86_64 0:3.7.9-10.el7 will be installed
--> Processing Dependency: glusterfs-client-xlators(x86-64) = 3.7.9-10.el7 for package: glusterfs-fuse-3.7.9-10.el7.x86_64
--> Processing Dependency: attr for package: glusterfs-fuse-3.7.9-10.el7.x86_64
--> Running transaction check
---> Package attr.x86_64 0:2.4.46-12.el7 will be installed
---> Package glusterfs-client-xlators.x86_64 0:3.7.9-10.el7 will be installed
---> Package glusterfs-libs.x86_64 0:3.7.9-10.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                          Arch                           Version                               Repository                                                        Size
==============================================================================================================================================================================================
Installing:
 glusterfs                                        x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         461 k
 glusterfs-fuse                                   x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         108 k
Installing for dependencies:
 attr                                             x86_64                         2.4.46-12.el7                         rhel-x86_64-server-7                                              66 k
 glusterfs-client-xlators                         x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         836 k
 glusterfs-libs                                   x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         331 k

Transaction Summary
==============================================================================================================================================================================================
Install  2 Packages (+3 Dependent packages)

Total download size: 1.8 M
Installed size: 7.1 M
Is this ok [y/d/N]: y
Downloading packages:
(1/5): attr-2.4.46-12.el7.x86_64.rpm                                                                                                                                   |  66 kB  00:00:00     
(2/5): glusterfs-3.7.9-10.el7.x86_64.rpm                                                                                                                               | 461 kB  00:00:00     
(3/5): glusterfs-client-xlators-3.7.9-10.el7.x86_64.rpm                                                                                                                | 836 kB  00:00:00     
(4/5): glusterfs-fuse-3.7.9-10.el7.x86_64.rpm                                                                                                                          | 108 kB  00:00:00     
(5/5): glusterfs-libs-3.7.9-10.el7.x86_64.rpm                                                                                                                          | 331 kB  00:00:00     
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         3.7 MB/s | 1.8 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : glusterfs-libs-3.7.9-10.el7.x86_64                                                                                                                                         1/5 
  Installing : glusterfs-client-xlators-3.7.9-10.el7.x86_64                                                                                                                               2/5 
  Installing : glusterfs-3.7.9-10.el7.x86_64                                                                                                                                              3/5 
  Installing : attr-2.4.46-12.el7.x86_64                                                                                                                                                  4/5 
  Installing : glusterfs-fuse-3.7.9-10.el7.x86_64                                                                                                                                         5/5 
  Verifying  : glusterfs-client-xlators-3.7.9-10.el7.x86_64                                                                                                                               1/5 
  Verifying  : glusterfs-libs-3.7.9-10.el7.x86_64                                                                                                                                         2/5 
  Verifying  : glusterfs-fuse-3.7.9-10.el7.x86_64                                                                                                                                         3/5 
  Verifying  : glusterfs-3.7.9-10.el7.x86_64                                                                                                                                              4/5 
  Verifying  : attr-2.4.46-12.el7.x86_64                                                                                                                                                  5/5 

Installed:
  glusterfs.x86_64 0:3.7.9-10.el7                                                             glusterfs-fuse.x86_64 0:3.7.9-10.el7                                                            

Dependency Installed:
  attr.x86_64 0:2.4.46-12.el7                          glusterfs-client-xlators.x86_64 0:3.7.9-10.el7                          glusterfs-libs.x86_64 0:3.7.9-10.el7                         

Complete!
[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# cat /etc/logrotate.d/glusterfs 
# Rotate client logs
/var/log/glusterfs/*.log {
  sharedscripts
  weekly
  rotate 52
  missingok
  compress
  delaycompress
  notifempty
  postrotate
  /usr/bin/killall -HUP glusterfs > /dev/null 2>&1 || true
  /usr/bin/killall -HUP glusterd > /dev/null 2>&1 || true
  endscript
}

# Rotate server logs
/var/log/glusterfs/bricks/*.log {
  sharedscripts
  weekly
  rotate 52
  missingok
  compress
  delaycompress
  notifempty
  postrotate
  /usr/bin/killall -HUP glusterfsd > /dev/null 2>&1 || true
  endscript
}
[root@dhcp37-82 ~]# which killall
/usr/bin/which: no killall in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# rpm -qa |grep psmisc
[root@dhcp37-82 ~]# 


psmisc package is there in the rhel7 server channel

Comment 1 Vijay Bellur 2016-08-17 07:41:15 UTC
REVIEW: http://review.gluster.org/15184 (build: add dependency for psmisc for fuse) posted (#1) for review on master by Milind Changire (mchangir)

Comment 2 Worker Ant 2016-08-22 14:00:53 UTC
REVIEW: http://review.gluster.org/15184 (build: add dependency for psmisc for fuse) posted (#2) for review on master by Milind Changire (mchangir)

Comment 3 Worker Ant 2016-08-22 14:54:01 UTC
REVIEW: http://review.gluster.org/15184 (build: add dependency for psmisc for fuse) posted (#3) for review on master by Milind Changire (mchangir)

Comment 4 Worker Ant 2016-08-23 14:17:10 UTC
COMMIT: http://review.gluster.org/15184 committed in master by Niels de Vos (ndevos) 
------
commit 1c4c75c358df745cc73b73bf2ee08e5c5d0b598f
Author: Milind Changire <mchangir>
Date:   Wed Aug 17 13:08:41 2016 +0530

    build: add dependency for psmisc for fuse
    
    Add package dependency for psmisc for client package fuse
    for the killall utility. killall is referenced in the gluster
    logrotate config file.
    
    Change-Id: I4cc95fe4aaa52faf8f384b9834654d3d928c252d
    BUG: 1367665
    Signed-off-by: Milind Changire <mchangir>
    Reviewed-on: http://review.gluster.org/15184
    Reviewed-by: Niels de Vos <ndevos>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>

Comment 5 Shyamsundar 2017-03-27 18:28:06 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report.

glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.