Description of problem: geo-rep workers crashing on non-root geo-rep setup with "OSError: [Errno 13] Permission denied" when mkdir and mv is executed from 3 clients. This was observed only on Distributed-Disperse and Distributed-Replicated volumes. ################################################################################ [root@dhcp43-72 ~]# gluster v geo-rep status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- dhcp43-72.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-82.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:45 dhcp43-72.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:50 dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:00:22 dhcp43-72.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep N/A Initializing... N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Initializing... N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:53 dhcp43-139.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Faulty N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-82.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:59 dhcp43-139.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Active History Crawl 2019-10-19 18:54:52 dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep N/A Faulty N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Faulty N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slaverep dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Initializing... N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-153.lab.eng.blr.redhat.com Active N/A ################################################################################ gsyncd.log for Distributed-Replicated volume. ################################################################################ [2019-10-19 13:34:06.631305] E [syncdutils(worker /bricks/brick5/masterdistrep):338:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 331, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1297, in service_loop g3.crawlwrap(oneshot=True) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap self.crawl() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1593, in crawl self.changelogs_batch_process(changes) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1493, in changelogs_batch_process self.process(batch) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1328, in process self.process_change(change, done, retry) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1222, in process_change failures = self.slave.server.entry_ops(entries) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: [Errno 13] Permission denied: '/bricks/brick5/slavedistrep/.glusterfs/27/3c/273c06af-527d-4f7b-ba0c-affc1d5c6a90' ################################################################################ ################################################################################ gsyncd.log for Distributed-Disperse volume. ################################################################################ [2019-10-19 13:39:38.680366] E [syncdutils(worker /bricks/brick3/masterdisp):338:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 331, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1297, in service_loop g3.crawlwrap(oneshot=True) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap self.crawl() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1593, in crawl self.changelogs_batch_process(changes) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1493, in changelogs_batch_process self.process(batch) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1328, in process self.process_change(change, done, retry) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1222, in process_change failures = self.slave.server.entry_ops(entries) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: [Errno 13] Permission denied: '/bricks/brick3/slavedisp/.glusterfs/91/0e/910e8317-67ed-4229-9338-8335a4d2ba9c' ################################################################################ Version-Release number of selected component (if applicable): glusterfs-6.0-19.el7rhgs.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.Create master and slave cluster of six nodes each. 2.Create one volume of each type and start on both clusters 3.Create and start non-root geo-rep sessions between all the master and slave volumes. 4. Mount the volumes on 3 clinets and perform the below given IO. From one client: for i in {1..1999}; do mkdir $i ; sleep 1 ; mv $i rs.$i ; done From second client: for i in {1..1000}; do mkdir dir.$i;mv dir.$i rename_dir.$i; done From third client: for i in {1..500}; do mkdir h.$i ; mv h.$i rsh.$i ; done Actual results: geo-rep workers crash. Expected results: geo-rep workers shouldn't crash. Additional info: Was the geo-rep setup stale? [root@dhcp43-72 ~]# gluster v geo-rep status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- dhcp43-72.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-82.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:40 dhcp43-72.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:59 dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:59 dhcp43-72.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:38:01 dhcp43-128.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:45 dhcp43-125.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:47 dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:38:06 dhcp43-125.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slaverep dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249