+++ This bug was initially created as a clone of Bug #1412689 +++ Description of problem: Slave mount log file is cluttered by logs of multiple active mounts Geo-rep worker mounts the slave volume on the slave node. If multiple worker connects to same slave node, all workers share the same mount log file. This is very difficult to debug as logs are cluttered from different mounts The location of log file is /var/log/glusterfs/geo-replication-slaves/*.gluster.log Version-Release number of selected component (if applicable): mainline How reproducible: Always Steps to Reproduce: 1. Create master volume with two bricks 2. Create slave volume with single brick 3. Establish geo-rep sessio between them. 6. Now geo-rep will have two slave mounts per master brick. Both will log into single file. Actual results: Multiple mount logs to same file Expected results: Each mount should log to separate file Additional info: --- Additional comment from Worker Ant on 2017-01-12 09:57:57 EST --- REVIEW: http://review.gluster.org/16384 (geo-rep: Separate slave mount logs for each connection) posted (#1) for review on master by Kotresh HR (khiremat) --- Additional comment from Aravinda VK on 2017-01-12 23:49:56 EST --- --- Additional comment from Worker Ant on 2017-01-18 03:47:05 EST --- COMMIT: http://review.gluster.org/16384 committed in master by Aravinda VK (avishwan) ------ commit ff5e91a60887d22934fcb5f8a15dd36019d6e09a Author: Kotresh HR <khiremat> Date: Tue Jan 10 15:39:55 2017 -0500 geo-rep: Separate slave mount logs for each connection Geo-rep worker mounts the slave volume on the slave node. If multiple worker connects to same slave node, all workers share the same mount log file. This is very difficult to debug as logs are cluttered from different mounts. Hence creating separate mount log file for each connection from worker. Each connection from worker is identified uniquely using 'mastervol uuid', 'master host', 'master brickpath', 'salve vol'. The log file name will be combination of the above. Change-Id: I67871dc8e8ea5864e2ad55e2a82063be0138bf0c BUG: 1412689 Signed-off-by: Kotresh HR <khiremat> Reviewed-on: http://review.gluster.org/16384 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Aravinda VK <avishwan>
upstream patch : http://review.gluster.org/16384
It's in upstream 3.10 as part of branch out from master
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/101292/
Verified with the build: glusterfs-geo-replication-3.8.4-22.el7rhgs.x86_64 With latest changes the log files contains 'mastervol uuid', 'master host', 'master brickpath', 'salve vol'. In 3.2.0: ========= [root@dhcp37-155 geo-replication-slaves]# ls d0beefb8-495e-41ae-b545-1e819f7e3b01:gluster%3A%2F%2F127.0.0.1%3Aslave.gluster.log d0beefb8-495e-41ae-b545-1e819f7e3b01:gluster%3A%2F%2F127.0.0.1%3Aslave.log mbr [root@dhcp37-155 geo-replication-slaves]# ps -eaf | grep gsyncd | grep tmp root 1768 1 0 18:42 ? 00:00:00 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication-slaves/d0beefb8-495e-41ae-b545-1e819f7e3b01:gluster%3A%2F%2F127.0.0.1%3Aslave.gluster.log --volfile-server=localhost --volfile-id=slave --client-pid=-1 /tmp/gsyncd-aux-mount-s3F0xe root 1806 1 0 18:42 ? 00:00:00 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication-slaves/d0beefb8-495e-41ae-b545-1e819f7e3b01:gluster%3A%2F%2F127.0.0.1%3Aslave.gluster.log --volfile-server=localhost --volfile-id=slave --client-pid=-1 /tmp/gsyncd-aux-mount-C1KsAJ [root@dhcp37-155 geo-replication-slaves]# In 3.3.0: ========= [root@dhcp43-94 geo-replication-slaves]# ps -eaf | grep gsyncd | grep tmp root 19977 1 0 13:05 ? 00:00:00 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication-slaves/e3e9e535-8938-49b8-8ef1-42488b13c6b9:10.70.43.91.%2Frhs%2Fbrick1%2Fb1.slave.gluster.log --volfile-server=localhost --volfile-id=slave --client-pid=-1 /tmp/gsyncd-aux-mount-GPp4F0 root 20015 1 0 13:05 ? 00:00:00 /usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication-slaves/e3e9e535-8938-49b8-8ef1-42488b13c6b9:10.70.43.93.%2Frhs%2Fbrick1%2Fb2.slave.gluster.log --volfile-server=localhost --volfile-id=slave --client-pid=-1 /tmp/gsyncd-aux-mount-lYyEBg [root@dhcp43-94 geo-replication-slaves]# [root@dhcp43-94 geo-replication-slaves]# [root@dhcp43-94 geo-replication-slaves]# ls e3e9e535-8938-49b8-8ef1-42488b13c6b9:10.70.43.91.%2Frhs%2Fbrick1%2Fb1.slave.gluster.log e3e9e535-8938-49b8-8ef1-42488b13c6b9:10.70.43.91.%2Frhs%2Fbrick1%2Fb1.slave.log e3e9e535-8938-49b8-8ef1-42488b13c6b9:10.70.43.93.%2Frhs%2Fbrick1%2Fb2.slave.gluster.log e3e9e535-8938-49b8-8ef1-42488b13c6b9:10.70.43.93.%2Frhs%2Fbrick1%2Fb2.slave.log mbr [root@dhcp43-94 geo-replication-slaves]# Moving this RFE to verified state, any further issues will be tracked by separate bugs.
Nit: It's slave mount logs, master mount logs were fine. Geo-replication workers now have separate slave mount logs to make debugging easier. Log files are named according to the following format: '<mastervol-uuid>:<master-host>:<master brickpath>:<slavevol>.gluster.log'.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774