Bug 1221175

Description of problem:
=======================

geo-rep session is going into faulty state with the below traceback:

[2015-05-13 17:47:26.465090] W [master(/rhs/brick1/b1):792:log_failures] _GMaster: META FAILED: ({'go': '.gfid/8db3cbe6-946e-45cb-bf74-d233c4091003', 'stat': {'atime': 1431518244.1599989, 'gid': 0, 'mtime': 1431518665.03, 'mode': 16877, 'uid': 0}, 'op': 'META'}, 2)
[2015-05-13 17:47:26.669015] E [repce(/rhs/brick1/b1):207:__call__] RepceClient: call 23144:139914667464448:1431519446.51 (entry_ops) failed on peer with OSError
[2015-05-13 17:47:26.669274] E [syncdutils(/rhs/brick1/b1):276:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 165, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 659, in main_i
    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1440, in service_loop
    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 580, in crawlwrap
    self.crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1150, in crawl
    self.changelogs_batch_process(changes)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1059, in changelogs_batch_process
    self.process(batch)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 946, in process
    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 902, in process_change
    failures = self.slave.server.entry_ops(entries)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 226, in __call__
    return self.ins(self.meth, *a)
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 208, in __call__
    raise res
OSError: [Errno 12] Cannot allocate memory
[2015-05-13 17:47:26.670993] I [syncdutils(/rhs/brick1/b1):220:finalize] <top>: exiting.
[2015-05-13 17:47:26.674686] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.
[2015-05-13 17:47:26.675065] I [syncdutils(agent):220:finalize] <top>: exiting.
[2015-05-13 17:47:27.299925] I [monitor(monitor):282:monitor] Monitor: worker(/rhs/brick1/b1) died in startup phase


I was doing delete from fuse and NFS mount with the trash translator ON. The session is configured to use meta volume.


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.0beta2-0.0.el6.x86_64


How reproducible:
=================

Could reproduce twice in as many attempts.


Steps Carried:
==============
1. Created master cluster 
2. Created and started master volume
3. Created shared volume (gluster_shared_storage)
4. Mounted the shared volume on /var/run/gluster/shared_storage
5. Created Slave cluster
6. Created and Started slave volume
7. Created geo-rep session between master and slave
8. Configured use_meta_volume true
9. Started geo-rep
10. Mounted master volume over Fuse and NFS to client
11. Copied files /etc{1..10} from fuse mount
12. Copied files /etc{11.20} from NFS mount
13. Sync completed successfully
14. Set the option features.trash ON for master and slave volume
15. Removed the files etc.2 from fuse and etc.12 from NFS
16. From Fuse and NFS it errored "Directory not empty". Another try was successful in removing.
17. Looked into the geo-rep session it was faulty 
18. Looked into the logs, it showed continuous traceback

Actual results:
===============

Worker died


Expected results:
================

Worker should run and sync the deleted files to slave


Additional info:
================

geo-rep session goes to faulty state. There was a similar bug with rename  and it was closed having ID 1144428. which did not involve trash translator and meta volume