Description of problem: gsyncd worker crashed in syncdutils with "OSError: [Errno 22] Invalid argument" Python backtrace >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2014-05-16 12:52:17.532416] W [master(/bricks/master_brick5):278:regjob] <top>: Rsync: .gfid/a71da453-578d-45c7-9974-1db81dfb387c [errcode: 23] [2014-05-16 12:52:17.533325] E [syncdutils(/bricks/master_brick1):270:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 164, in main main_i() File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 633, in main_i local.service_loop(*[r for r in [remote] if r]) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1298, in service_loop g3.crawlwrap(oneshot=True) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 501, in crawlwrap self.crawl() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1191, in crawl self.process(changes) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 904, in process if self.syncdata_wait(): File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 282, in syncdata_wait if self.wait(self.FLAT_DIR_HIERARCHY, None): File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 689, in wait ret = j[-1]() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 671, in <lambda> self.jobtab[path].append((label, a, lambda: job(*a, **kw))) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 274, in regjob st = lstat(se) File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 487, in lstat return os.lstat(e) OSError: [Errno 22] Invalid argument: '.gfid/8a95a636-d68a-443e-a607-abe32c21c4c5' [2014-05-16 12:52:17.536465] I [syncdutils(/bricks/master_brick1):214:finalize] <top>: exiting. [2014-05-16 12:52:17.547113] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF. [2014-05-16 12:52:17.547662] I [syncdutils(agent):214:finalize] <top>: exiting. [2014-05-16 12:52:17.551680] W [master(/bricks/master_brick5):278:regjob] <top>: Rsync: .gfid/14eb0c3c-584d-4bbd-bebb-e64e72406da4 [errcode: 23] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Version-Release number of selected component (if applicable): glusterfs-3.6.0.2-1.el6rhs How reproducible: Didn't try to reproduce. Steps to Reproduce: 1.create and start a geo-rep relationship between master and slave. 2.start creating files on master using the command "crefi -n 10 -T 10 --multi -b 10 -d 10 --random --max=50K --min=1K /mnt/master" 3. check if geo-rep log file has such crash. Actual results: worker crashed in syncdutils "OSError: [Errno 22] Invalid argument" . Expected results: worker shouldn't crash. Additional info:
I see an OSError: [Errno 22] Invalid argument on glusterfs3.6.0.51-1rhs. But the traceback is different: [2015-03-13 20:15:15.838018] E [syncdutils(/bricks/brick0/master_brick0):270:log_raise_exception] <top>: FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 164, in main main_i() File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 645, in main_i local.service_loop(*[r for r in [remote] if r]) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1348, in service_loop g2.crawlwrap() File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 553, in crawlwrap self.crawl(no_stime_update=no_stime_update) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1413, in crawl self.process([item[1]], 0) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1017, in process self.process_change(change, done, retry) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 981, in process_change self.slave.server.entry_ops(entries) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 226, in __call__ return self.ins(self.meth, *a) File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 208, in __call__ raise res OSError: [Errno 22] Invalid argument [2015-03-13 20:15:15.841619] I [syncdutils(/bricks/brick0/master_brick0):214:finalize] <top>: exiting. [2015-03-13 20:15:15.850374] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF. [2015-03-13 20:15:15.850802] I [syncdutils(agent):214:finalize] <top>: exiting. [2015-03-13 20:15:15.859649] I [monitor(monitor):146:set_state] Monitor: new state: faulty [2015-03-13 20:15:26.214607] I [monitor(monitor):220:monitor] Monitor: ------------------------------------------------------------ [2015-03-13 20:15:26.215138] I [monitor(monitor):221:monitor] Monitor: starting gsyncd worker