Bug 1763439

Summary: [GSS] geo-rep entering into faulty state with OSError: [Errno 13] Permission denied
Product: [Community] GlusterFS Reporter: Kotresh HR <khiremat>
Component: geo-replicationAssignee: Kotresh HR <khiremat>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: avishwan, csaba, khiremat, olim, rhinduja, rhs-bugs, storage-qa-internal, sunkumar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1762127
: 1764030 1764183 (view as bug list) Environment:
Last Closed: 2019-10-21 16:31:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1762127    
Bug Blocks: 1764030, 1764183    

Description Kotresh HR 2019-10-19 20:02:35 UTC 
+++ This bug was initially created as a clone of Bug #1762127 +++

Issue description:


Customer have a secured geo-replication setup, and am encountering the following issue:


[2019-10-13 17:17:32.126867] I [master(/opt/gluster/siteh4/brick2):1407:process] _GMaster: Batch Completed      changelog_end=1570949937        entry_stime=(1570950011, 0)     changelog_start=1570949937    stime=(1570949936, 0)    duration=0.0858 num_changelogs=1        mode=history_changelog
[2019-10-13 17:17:32.133430] I [master(/opt/gluster/siteh4/brick2):1550:crawl] _GMaster: slave's time   stime=(1570949936, 0)
[2019-10-13 17:17:32.309490] E [repce(/opt/gluster/siteh4/brick2):209:__call__] RepceClient: call failed        call=9419:140374089455424:1570987052.15 method=entry_ops        error=OSError
[2019-10-13 17:17:32.309892] E [syncdutils(/opt/gluster/siteh4/brick2):355:log_raise_exception] <top>: FAIL:
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 210, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 805, in main_i
    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1592, in service_loop
    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 597, in crawlwrap
    self.crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1559, in crawl
    self.changelogs_batch_process(changes)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1459, in changelogs_batch_process
    self.process(batch)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1293, in process
    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1189, in process_change
    failures = self.slave.server.entry_ops(entries)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 228, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 210, in __call__
    raise res
OSError: [Errno 13] Permission denied: '/opt/gluster/siteh4/brick2/.glusterfs/7d/eb/7deb5f05-b8a4-4d5e-92de-b979c0d17960'
[2019-10-13 17:17:32.329540] I [syncdutils(/opt/gluster/siteh4/brick2):295:finalize] <top>: exiting.
[2019-10-13 17:17:32.334484] I [repce(/opt/gluster/siteh4/brick2):92:service_loop] RepceServer: terminating on reaching EOF.
[2019-10-13 17:17:32.334897] I [syncdutils(/opt/gluster/siteh4/brick2):295:finalize] <top>: exiting.
Comment 1 Worker Ant 2019-10-19 20:08:09 UTC 
REVIEW: https://review.gluster.org/23570 (geo-rep: Fix Permission denied traceback on non root setup) posted (#1) for review on master by Kotresh HR
Comment 2 Worker Ant 2019-10-21 16:31:07 UTC 
REVIEW: https://review.gluster.org/23570 (geo-rep: Fix Permission denied traceback on non root setup) merged (#3) on master by Kotresh HR