Bug 1098426 - Dist-geo-rep : gsyncd worker crashed in syncdutils with "OSError: [Errno 22] Invalid argument"
Summary: Dist-geo-rep : gsyncd worker crashed in syncdutils with "OSError: [Errno 22] ...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: geo-replication
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Bug Updates Notification Mailing List
QA Contact: amainkar
URL:
Whiteboard: consistency
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-16 07:45 UTC by Vijaykumar Koppad
Modified: 2018-04-16 15:56 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-16 15:56:44 UTC
Embargoed:


Attachments (Terms of Use)

Description Vijaykumar Koppad 2014-05-16 07:45:08 UTC
Description of problem: gsyncd worker crashed in syncdutils with "OSError: [Errno 22] Invalid argument"
Python backtrace
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[2014-05-16 12:52:17.532416] W [master(/bricks/master_brick5):278:regjob] <top>: Rsync: .gfid/a71da453-578d-45c7-9974-1db81dfb387c [errcode: 23]
[2014-05-16 12:52:17.533325] E [syncdutils(/bricks/master_brick1):270:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 164, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 633, in main_i
    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1298, in service_loop
    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 501, in crawlwrap
    self.crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1191, in crawl
    self.process(changes)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 904, in process
    if self.syncdata_wait():
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 282, in syncdata_wait
    if self.wait(self.FLAT_DIR_HIERARCHY, None):
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 689, in wait
    ret = j[-1]()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 671, in <lambda>
    self.jobtab[path].append((label, a, lambda: job(*a, **kw)))
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 274, in regjob
    st = lstat(se)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 487, in lstat
    return os.lstat(e)
OSError: [Errno 22] Invalid argument: '.gfid/8a95a636-d68a-443e-a607-abe32c21c4c5'
[2014-05-16 12:52:17.536465] I [syncdutils(/bricks/master_brick1):214:finalize] <top>: exiting.
[2014-05-16 12:52:17.547113] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.
[2014-05-16 12:52:17.547662] I [syncdutils(agent):214:finalize] <top>: exiting.
[2014-05-16 12:52:17.551680] W [master(/bricks/master_brick5):278:regjob] <top>: Rsync: .gfid/14eb0c3c-584d-4bbd-bebb-e64e72406da4 [errcode: 23]

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>


Version-Release number of selected component (if applicable): glusterfs-3.6.0.2-1.el6rhs


How reproducible: Didn't try to reproduce.


Steps to Reproduce:
1.create and start a geo-rep relationship between master and slave.
2.start creating files on master using the command "crefi -n 10 -T 10 --multi -b 10 -d 10 --random --max=50K --min=1K  /mnt/master"
3. check if  geo-rep log file has such crash. 

Actual results: worker crashed in syncdutils "OSError: [Errno 22] Invalid argument" .


Expected results: worker shouldn't crash. 


Additional info:

Comment 6 shilpa 2015-03-13 17:26:44 UTC
I see an OSError: [Errno 22] Invalid argument on glusterfs3.6.0.51-1rhs.
But the traceback is different:

[2015-03-13 20:15:15.838018] E [syncdutils(/bricks/brick0/master_brick0):270:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 164, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 645, in main_i
    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1348, in service_loop
    g2.crawlwrap()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 553, in crawlwrap
    self.crawl(no_stime_update=no_stime_update)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1413, in crawl
    self.process([item[1]], 0)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1017, in process
    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 981, in process_change
    self.slave.server.entry_ops(entries)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 226, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 208, in __call__
    raise res
OSError: [Errno 22] Invalid argument
[2015-03-13 20:15:15.841619] I [syncdutils(/bricks/brick0/master_brick0):214:finalize] <top>: exiting.
[2015-03-13 20:15:15.850374] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.
[2015-03-13 20:15:15.850802] I [syncdutils(agent):214:finalize] <top>: exiting.
[2015-03-13 20:15:15.859649] I [monitor(monitor):146:set_state] Monitor: new state: faulty
[2015-03-13 20:15:26.214607] I [monitor(monitor):220:monitor] Monitor: ------------------------------------------------------------
[2015-03-13 20:15:26.215138] I [monitor(monitor):221:monitor] Monitor: starting gsyncd worker


Note You need to log in before you can comment on or make changes to this bug.