Description of problem: ======================= rotated FUSE mount log is using to populate the information instead of newly created file after log rotate. Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.9-1.el7.x86_64 How reproducible: ================= One time observed.// i think it's 100% reproducible. Steps to Reproduce: =================== 1. FUSE mount one of the sample volume on the client 2. Log rotate the volume mount log dynamically (OR) wait for log rotate to happen // check log file has rotated and new log file has created. 2. Do some volume set operations for the mounted volume 3. Check volume set information is populated in newly created mount log file // file will be empty. Actual results: =============== Logs are populating to rotated log instead of writing to new log file. Expected results: ================= Once log rotated, next log info should populate to new log file. Additional info:
I tried reproducing it by running logrotate command manually with logrotate --force /etc/logrotate.d/glusterfs But I could see the further logs get redirected to new mount log, not the rotated log file. Please cross check. And also if you hit it again, upload the sosreport.
Given that I've waited for few days for QE to get back, I am closing this bug since the same is not reproducible at our end. Please feel free to reopen if you hit the issue and attach relevant logs.
Created attachment 1191060 [details] FUSE Client SOS Report
This issue is reproducible always. I am reopening this bug and sos report is attached to debug the issue.
I retried the scenario as mentioned in comment 2 with both upstream and downstream latest bits and couldn't reproduce this issue. The only difference I see here is that I am not waiting logrotate to trigger the rotation gracefully, instead going for a force mode, but IMO that shouldn't make any difference. With that, may I ask the maintainer(s) of logging (libglusterfs) to chime in with their inputs?
"killall" command package ( psmisc ) is not getting installed by default / not pulling during client packages installation, which is needed for the gluster log files rotation. Below is some info: [root@dhcp37-82 ~]# [root@dhcp37-82 ~]# yum install glusterfs glusterfs-fuse Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager This system is receiving updates from RHN Classic or Red Hat Satellite. Resolving Dependencies --> Running transaction check ---> Package glusterfs.x86_64 0:3.7.9-10.el7 will be installed --> Processing Dependency: glusterfs-libs(x86-64) = 3.7.9-10.el7 for package: glusterfs-3.7.9-10.el7.x86_64 --> Processing Dependency: libgfxdr.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64 --> Processing Dependency: libglusterfs.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64 --> Processing Dependency: libgfrpc.so.0()(64bit) for package: glusterfs-3.7.9-10.el7.x86_64 ---> Package glusterfs-fuse.x86_64 0:3.7.9-10.el7 will be installed --> Processing Dependency: glusterfs-client-xlators(x86-64) = 3.7.9-10.el7 for package: glusterfs-fuse-3.7.9-10.el7.x86_64 --> Processing Dependency: attr for package: glusterfs-fuse-3.7.9-10.el7.x86_64 --> Running transaction check ---> Package attr.x86_64 0:2.4.46-12.el7 will be installed ---> Package glusterfs-client-xlators.x86_64 0:3.7.9-10.el7 will be installed ---> Package glusterfs-libs.x86_64 0:3.7.9-10.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================== Installing: glusterfs x86_64 3.7.9-10.el7 rhel-x86_64-server-7-rh-gluster-3-client 461 k glusterfs-fuse x86_64 3.7.9-10.el7 rhel-x86_64-server-7-rh-gluster-3-client 108 k Installing for dependencies: attr x86_64 2.4.46-12.el7 rhel-x86_64-server-7 66 k glusterfs-client-xlators x86_64 3.7.9-10.el7 rhel-x86_64-server-7-rh-gluster-3-client 836 k glusterfs-libs x86_64 3.7.9-10.el7 rhel-x86_64-server-7-rh-gluster-3-client 331 k Transaction Summary ============================================================================================================================================================================================== Install 2 Packages (+3 Dependent packages) Total download size: 1.8 M Installed size: 7.1 M Is this ok [y/d/N]: y Downloading packages: (1/5): attr-2.4.46-12.el7.x86_64.rpm | 66 kB 00:00:00 (2/5): glusterfs-3.7.9-10.el7.x86_64.rpm | 461 kB 00:00:00 (3/5): glusterfs-client-xlators-3.7.9-10.el7.x86_64.rpm | 836 kB 00:00:00 (4/5): glusterfs-fuse-3.7.9-10.el7.x86_64.rpm | 108 kB 00:00:00 (5/5): glusterfs-libs-3.7.9-10.el7.x86_64.rpm | 331 kB 00:00:00 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 3.7 MB/s | 1.8 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : glusterfs-libs-3.7.9-10.el7.x86_64 1/5 Installing : glusterfs-client-xlators-3.7.9-10.el7.x86_64 2/5 Installing : glusterfs-3.7.9-10.el7.x86_64 3/5 Installing : attr-2.4.46-12.el7.x86_64 4/5 Installing : glusterfs-fuse-3.7.9-10.el7.x86_64 5/5 Verifying : glusterfs-client-xlators-3.7.9-10.el7.x86_64 1/5 Verifying : glusterfs-libs-3.7.9-10.el7.x86_64 2/5 Verifying : glusterfs-fuse-3.7.9-10.el7.x86_64 3/5 Verifying : glusterfs-3.7.9-10.el7.x86_64 4/5 Verifying : attr-2.4.46-12.el7.x86_64 5/5 Installed: glusterfs.x86_64 0:3.7.9-10.el7 glusterfs-fuse.x86_64 0:3.7.9-10.el7 Dependency Installed: attr.x86_64 0:2.4.46-12.el7 glusterfs-client-xlators.x86_64 0:3.7.9-10.el7 glusterfs-libs.x86_64 0:3.7.9-10.el7 Complete! [root@dhcp37-82 ~]# [root@dhcp37-82 ~]# cat /etc/logrotate.d/glusterfs # Rotate client logs /var/log/glusterfs/*.log { sharedscripts weekly rotate 52 missingok compress delaycompress notifempty postrotate /usr/bin/killall -HUP glusterfs > /dev/null 2>&1 || true /usr/bin/killall -HUP glusterd > /dev/null 2>&1 || true endscript } # Rotate server logs /var/log/glusterfs/bricks/*.log { sharedscripts weekly rotate 52 missingok compress delaycompress notifempty postrotate /usr/bin/killall -HUP glusterfsd > /dev/null 2>&1 || true endscript } [root@dhcp37-82 ~]# which killall /usr/bin/which: no killall in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) [root@dhcp37-82 ~]# [root@dhcp37-82 ~]# [root@dhcp37-82 ~]# rpm -qa |grep psmisc [root@dhcp37-82 ~]# psmisc package is there in the rhel7 server channel
Upstream mainline : http://review.gluster.org/15184 Upstream 3.8 : http://review.gluster.org/15297 And the fix is available in rhgs-3.2.0 as part of rebase to GlusterFS 3.8.4.
Verified this bug using build - glusterfs-3.8.4-5.el7.x86_64. I am seeing log rotation happening and killall command which is needed for the log rotation is installed using the package " psmisc-22.20-11.el7.x86_64". This package is pulled while installing the rhel7 rhgs client packages. ]# rpm -qf /usr/bin/killall psmisc-22.20-11.el7.x86_64 psmisc-22.20-11.el7.x86_64 package got pulled while installing client packages. ============================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================== Installing: glusterfs x86_64 3.8.4-5.el7 RHSClient-RHEL-7 489 k glusterfs-fuse x86_64 3.8.4-5.el7 RHSClient-RHEL-7 109 k Installing for dependencies: glusterfs-client-xlators x86_64 3.8.4-5.el7 RHSClient-RHEL-7 762 k glusterfs-libs x86_64 3.8.4-5.el7 RHSClient-RHEL-7 356 k psmisc x86_64 22.20-11.el7 rhel-x86_64-server-7 141 k Transaction Summary Moving to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html