Bug 1327952 - rotated FUSE mount log is using to populate the information after log rotate.
Summary: rotated FUSE mount log is using to populate the information after log rotate.
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: build
Version: rhgs-3.1
Hardware: x86_64
OS: Linux
Target Milestone: ---
: RHGS 3.2.0
Assignee: Milind Changire
QA Contact: Byreddy
Depends On:
Blocks: 1351528
TreeView+ depends on / blocked
Reported: 2016-04-18 06:51 UTC by Byreddy
Modified: 2018-09-03 11:39 UTC (History)
9 users (show)

Fixed In Version: glusterfs-3.8.4-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1367665 1369517 (view as bug list)
Last Closed: 2017-03-23 05:28:45 UTC

Attachments (Terms of Use)
FUSE Client SOS Report (6.65 MB, application/x-xz)
2016-08-16 05:06 UTC, Byreddy
no flags Details

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Byreddy 2016-04-18 06:51:38 UTC
Description of problem:
rotated FUSE mount log is using to populate the information instead of newly created file after log rotate.

Version-Release number of selected component (if applicable):

How reproducible:
One time observed.// i think it's 100% reproducible.

Steps to Reproduce:
1. FUSE mount one of the sample volume on the client 
2. Log rotate the volume mount log dynamically (OR)  wait for log rotate to happen // check log file has rotated and new log file has created.
2. Do some volume set operations for the mounted volume
3. Check volume set information is populated in newly created mount log file // file will be empty.

Actual results:
Logs are populating to rotated log instead of writing to new log file.

Expected results:
Once log rotated, next log info should populate to new log file.

Additional info:

Comment 2 Atin Mukherjee 2016-05-05 11:43:33 UTC
I tried reproducing it by running logrotate command manually with
logrotate --force /etc/logrotate.d/glusterfs

But I could see the further logs get redirected to new mount log, not the rotated log file.

Please cross check.

And also if you hit it again, upload the sosreport.

Comment 3 Atin Mukherjee 2016-05-10 03:53:46 UTC
Given that I've waited for few days for QE to get back, I am closing this bug since the same is not reproducible at our end. Please feel free to reopen if you hit the issue and attach relevant logs.

Comment 4 Byreddy 2016-08-16 05:06:58 UTC
Created attachment 1191060 [details]
FUSE Client SOS Report

Comment 5 Byreddy 2016-08-16 05:08:36 UTC
This issue is reproducible always.
I am reopening this bug and sos report is attached to debug the issue.

Comment 7 Atin Mukherjee 2016-08-16 07:12:31 UTC
I retried the scenario as mentioned in comment 2 with both upstream and downstream latest bits and couldn't reproduce this issue. The only difference I see here is that I am not waiting logrotate to trigger the rotation gracefully, instead going for a force mode, but IMO that shouldn't make any difference. 

With that, may I ask the maintainer(s) of logging (libglusterfs) to chime in with their inputs?

Comment 8 Byreddy 2016-08-16 10:27:42 UTC
"killall"  command package ( psmisc ) is not getting installed by default / not pulling during client packages installation, which is needed for  the gluster log files rotation.

Below is some info:

[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# yum install glusterfs glusterfs-fuse
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Resolving Dependencies
--> Running transaction check
---> Package glusterfs.x86_64 0:3.7.9-10.el7 will be installed
--> Processing Dependency: glusterfs-libs(x86-64) = 3.7.9-10.el7 for package: glusterfs-3.7.9-10.el7.x86_64
--> Processing Dependency: libgfxdr.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64
--> Processing Dependency: libglusterfs.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64
--> Processing Dependency: libgfrpc.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64
---> Package glusterfs-fuse.x86_64 0:3.7.9-10.el7 will be installed
--> Processing Dependency: glusterfs-client-xlators(x86-64) = 3.7.9-10.el7 for package: glusterfs-fuse-3.7.9-10.el7.x86_64
--> Processing Dependency: attr for package: glusterfs-fuse-3.7.9-10.el7.x86_64
--> Running transaction check
---> Package attr.x86_64 0:2.4.46-12.el7 will be installed
---> Package glusterfs-client-xlators.x86_64 0:3.7.9-10.el7 will be installed
---> Package glusterfs-libs.x86_64 0:3.7.9-10.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

 Package                                          Arch                           Version                               Repository                                                        Size
 glusterfs                                        x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         461 k
 glusterfs-fuse                                   x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         108 k
Installing for dependencies:
 attr                                             x86_64                         2.4.46-12.el7                         rhel-x86_64-server-7                                              66 k
 glusterfs-client-xlators                         x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         836 k
 glusterfs-libs                                   x86_64                         3.7.9-10.el7                          rhel-x86_64-server-7-rh-gluster-3-client                         331 k

Transaction Summary
Install  2 Packages (+3 Dependent packages)

Total download size: 1.8 M
Installed size: 7.1 M
Is this ok [y/d/N]: y
Downloading packages:
(1/5): attr-2.4.46-12.el7.x86_64.rpm                                                                                                                                   |  66 kB  00:00:00     
(2/5): glusterfs-3.7.9-10.el7.x86_64.rpm                                                                                                                               | 461 kB  00:00:00     
(3/5): glusterfs-client-xlators-3.7.9-10.el7.x86_64.rpm                                                                                                                | 836 kB  00:00:00     
(4/5): glusterfs-fuse-3.7.9-10.el7.x86_64.rpm                                                                                                                          | 108 kB  00:00:00     
(5/5): glusterfs-libs-3.7.9-10.el7.x86_64.rpm                                                                                                                          | 331 kB  00:00:00     
Total                                                                                                                                                         3.7 MB/s | 1.8 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : glusterfs-libs-3.7.9-10.el7.x86_64                                                                                                                                         1/5 
  Installing : glusterfs-client-xlators-3.7.9-10.el7.x86_64                                                                                                                               2/5 
  Installing : glusterfs-3.7.9-10.el7.x86_64                                                                                                                                              3/5 
  Installing : attr-2.4.46-12.el7.x86_64                                                                                                                                                  4/5 
  Installing : glusterfs-fuse-3.7.9-10.el7.x86_64                                                                                                                                         5/5 
  Verifying  : glusterfs-client-xlators-3.7.9-10.el7.x86_64                                                                                                                               1/5 
  Verifying  : glusterfs-libs-3.7.9-10.el7.x86_64                                                                                                                                         2/5 
  Verifying  : glusterfs-fuse-3.7.9-10.el7.x86_64                                                                                                                                         3/5 
  Verifying  : glusterfs-3.7.9-10.el7.x86_64                                                                                                                                              4/5 
  Verifying  : attr-2.4.46-12.el7.x86_64                                                                                                                                                  5/5 

  glusterfs.x86_64 0:3.7.9-10.el7                                                             glusterfs-fuse.x86_64 0:3.7.9-10.el7                                                            

Dependency Installed:
  attr.x86_64 0:2.4.46-12.el7                          glusterfs-client-xlators.x86_64 0:3.7.9-10.el7                          glusterfs-libs.x86_64 0:3.7.9-10.el7                         

[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# cat /etc/logrotate.d/glusterfs 
# Rotate client logs
/var/log/glusterfs/*.log {
  rotate 52
  /usr/bin/killall -HUP glusterfs > /dev/null 2>&1 || true
  /usr/bin/killall -HUP glusterd > /dev/null 2>&1 || true

# Rotate server logs
/var/log/glusterfs/bricks/*.log {
  rotate 52
  /usr/bin/killall -HUP glusterfsd > /dev/null 2>&1 || true
[root@dhcp37-82 ~]# which killall
/usr/bin/which: no killall in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# 
[root@dhcp37-82 ~]# rpm -qa |grep psmisc
[root@dhcp37-82 ~]# 

psmisc package is there in the rhel7 server channel

Comment 9 Atin Mukherjee 2016-09-19 13:02:03 UTC
Upstream mainline : http://review.gluster.org/15184
Upstream 3.8 : http://review.gluster.org/15297

And the fix is available in rhgs-3.2.0 as part of rebase to GlusterFS 3.8.4.

Comment 14 Byreddy 2016-11-24 05:01:30 UTC
Verified this bug using build - glusterfs-3.8.4-5.el7.x86_64.

I am seeing log rotation happening and killall command which is needed for the log rotation is installed using the package " psmisc-22.20-11.el7.x86_64".
This package is pulled while installing the rhel7 rhgs client packages.

]# rpm -qf /usr/bin/killall

psmisc-22.20-11.el7.x86_64 package got pulled while installing client packages.

 Package                                               Arch                                Version                                    Repository                                         Size
 glusterfs                                             x86_64                              3.8.4-5.el7                                RHSClient-RHEL-7                                  489 k
 glusterfs-fuse                                        x86_64                              3.8.4-5.el7                                RHSClient-RHEL-7                                  109 k
Installing for dependencies:
 glusterfs-client-xlators                              x86_64                              3.8.4-5.el7                                RHSClient-RHEL-7                                  762 k
 glusterfs-libs                                        x86_64                              3.8.4-5.el7                                RHSClient-RHEL-7                                  356 k
 psmisc                                                x86_64                              22.20-11.el7                               rhel-x86_64-server-7                              141 k

Transaction Summary

Moving to verified state.

Comment 16 errata-xmlrpc 2017-03-23 05:28:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.