Bug 1763412
| Summary: | [geo-rep] Workers crashing for non-root geo-rep sessions with OSError: [Errno 13] Permission denied when mkdir and mv is executed | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Kshithij Iyer <kiyer> |
| Component: | geo-replication | Assignee: | Kotresh HR <khiremat> |
| Status: | CLOSED ERRATA | QA Contact: | Kshithij Iyer <kiyer> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rhgs-3.5 | CC: | aspandey, avishwan, csaba, khiremat, rhs-bugs, rkothiya, sheggodu, storage-qa-internal, vdas |
| Target Milestone: | --- | Flags: | aspandey:
needinfo-
|
| Target Release: | RHGS 3.5.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | glusterfs-6.0-20 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-10-30 12:23:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1696809 | ||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249 |
Description of problem: geo-rep workers crashing on non-root geo-rep setup with "OSError: [Errno 13] Permission denied" when mkdir and mv is executed from 3 clients. This was observed only on Distributed-Disperse and Distributed-Replicated volumes. ################################################################################ [root@dhcp43-72 ~]# gluster v geo-rep status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- dhcp43-72.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-82.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:45 dhcp43-72.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:50 dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:00:22 dhcp43-72.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep N/A Initializing... N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Initializing... N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:53 dhcp43-139.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Faulty N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-82.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:01:59 dhcp43-139.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Active History Crawl 2019-10-19 18:54:52 dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep N/A Faulty N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Faulty N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slaverep dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip N/A Initializing... N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-153.lab.eng.blr.redhat.com Active N/A ################################################################################ gsyncd.log for Distributed-Replicated volume. ################################################################################ [2019-10-19 13:34:06.631305] E [syncdutils(worker /bricks/brick5/masterdistrep):338:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 331, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1297, in service_loop g3.crawlwrap(oneshot=True) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap self.crawl() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1593, in crawl self.changelogs_batch_process(changes) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1493, in changelogs_batch_process self.process(batch) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1328, in process self.process_change(change, done, retry) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1222, in process_change failures = self.slave.server.entry_ops(entries) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: [Errno 13] Permission denied: '/bricks/brick5/slavedistrep/.glusterfs/27/3c/273c06af-527d-4f7b-ba0c-affc1d5c6a90' ################################################################################ ################################################################################ gsyncd.log for Distributed-Disperse volume. ################################################################################ [2019-10-19 13:39:38.680366] E [syncdutils(worker /bricks/brick3/masterdisp):338:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 331, in main func(args) File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker local.service_loop(remote) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1297, in service_loop g3.crawlwrap(oneshot=True) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap self.crawl() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1593, in crawl self.changelogs_batch_process(changes) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1493, in changelogs_batch_process self.process(batch) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1328, in process self.process_change(change, done, retry) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1222, in process_change failures = self.slave.server.entry_ops(entries) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__ raise res OSError: [Errno 13] Permission denied: '/bricks/brick3/slavedisp/.glusterfs/91/0e/910e8317-67ed-4229-9338-8335a4d2ba9c' ################################################################################ Version-Release number of selected component (if applicable): glusterfs-6.0-19.el7rhgs.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.Create master and slave cluster of six nodes each. 2.Create one volume of each type and start on both clusters 3.Create and start non-root geo-rep sessions between all the master and slave volumes. 4. Mount the volumes on 3 clinets and perform the below given IO. From one client: for i in {1..1999}; do mkdir $i ; sleep 1 ; mv $i rs.$i ; done From second client: for i in {1..1000}; do mkdir dir.$i;mv dir.$i rename_dir.$i; done From third client: for i in {1..500}; do mkdir h.$i ; mv h.$i rsh.$i ; done Actual results: geo-rep workers crash. Expected results: geo-rep workers shouldn't crash. Additional info: Was the geo-rep setup stale? [root@dhcp43-72 ~]# gluster v geo-rep status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- dhcp43-72.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-82.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:40 dhcp43-72.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:59 dhcp43-72.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:59 dhcp43-72.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-176.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:38:01 dhcp43-128.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-128.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:45 dhcp43-125.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:37:47 dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-162.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-125.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-162.lab.eng.blr.redhat.com Active Changelog Crawl 2019-10-19 19:38:06 dhcp43-125.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slaverep dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-82.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-139.lab.eng.blr.redhat.com masterrep /bricks/brick1/masterrep geoaccount ssh://geoaccount.43.82::slaverep dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-167.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-114.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-153.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterarb /bricks/brick1/masterarb geoaccount ssh://geoaccount.43.82::slavearb dhcp43-176.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdip /bricks/brick2/masterdisp geoaccount ssh://geoaccount.43.82::slavedip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick3/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistdip /bricks/brick4/masterdisp geoaccount ssh://geoaccount.43.82::slavedistdip dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A dhcp43-103.lab.eng.blr.redhat.com masterdistrep /bricks/brick5/masterdistrep geoaccount ssh://geoaccount.43.82::slavedistrep dhcp43-168.lab.eng.blr.redhat.com Passive N/A N/A