Bug 1217938

Summary: Dist-geo-rep: Too many "remote operation failed: No such file or directory" warning messages in auxilary mount log on slave while executing "rm -rf"
Product: [Community] GlusterFS Reporter: Kotresh HR <khiremat>
Component: geo-replicationAssignee: Kotresh HR <khiremat>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3.7.0CC: aavati, avishwan, bugs, csaba, gluster-bugs, rhs-bugs, storage-qa-internal, vagarwal, vbhat
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: consistency
Fixed In Version: glusterfs-3.7.0beta2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1210562 Environment:
Last Closed: 2015-05-14 17:27:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 990558, 1210562    
Bug Blocks: 1199352    

Description Kotresh HR 2015-05-03 07:17:38 UTC 
+++ This bug was initially created as a clone of Bug #1210562 +++

Description of problem:
I was running rm -rf * on the master mount point and saw lots of Warning messages in the auxiliary mount logs in the slave.

The messages were 

[2013-07-31 11:11:59.688460] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.688548] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.690008] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691900] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691959] I [afr-lk-common.c:1085:afr_lock_blocking] 0-hosa_slave-replicate-1: unable to lock on even one child
[2013-07-31 11:11:59.691993] I [afr-transaction.c:1063:afr_post_blocking_inodelk_cbk] 0-hosa_slave-replicate-1: Blocking inodelks failed.
[2013-07-31 11:11:59.693990] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory. Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools (dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.694328] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory. Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools (dce9238f-e5c7-49c1-9741-73388f92f050)


Version-Release number of selected component (if applicable):
mailine

How reproducible:
Hit once. Not sure about reproducibility.

Steps to Reproduce:
1. Create and start a geo-rep session between master and slave.
2. Untar the linux kernel twice in 2 different locations on the master mountpoint.
3. Now run from master mount point run, rm -rf *

Actual results:
 hosa_slave-client-2
[2013-07-31 11:11:58.042628] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:58.042946] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:58.087397] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-0: remote operation failed: No such file or directory
[2013-07-31 11:11:58.088347] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-1: remote operation failed: No such file or directory
[2013-07-31 11:11:58.088422] W [fuse-bridge.c:1624:fuse_unlink_cbk] 0-glusterfs-fuse: 1528415: RMDIR() <gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/vhost_test => -1 (No such file or directory)
[2013-07-31 11:11:58.291267] E [afr-self-heal-entry.c:2325:afr_sh_post_nonblocking_entry_cbk] 0-hosa_slave-replicate-1: Non Blocking entrylks failed for <gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace.
[2013-07-31 11:11:58.291409] E [afr-self-heal-common.c:2722:afr_log_self_heal_completion_status] 0-hosa_slave-replicate-1: foreground gfid or missing entry self heal  is not attempted, medatadata self heal  is not attempted, data self heal  is not attempted, entry self heal  failed on  <gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace
[2013-07-31 11:11:58.296013] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:58.296128] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:58.324498] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-0: remote operation failed: No such file or directory
[2013-07-31 11:11:58.325157] W [client-rpc-fops.c:695:client3_3_rmdir_cbk] 0-hosa_slave-client-1: remote operation failed: No such file or directory
[2013-07-31 11:11:58.325237] W [fuse-bridge.c:1624:fuse_unlink_cbk] 0-glusterfs-fuse: 1581366: RMDIR() <gfid:69f85589-6525-4cdb-a308-9dae4d6c6935>/virtio-trace => -1 (No such file or directory)
[2013-07-31 11:11:59.613599] I [dht-layout.c:636:dht_layout_normalize] 0-hosa_slave-dht: found anomalies in <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools. holes=1 overlaps=1 missing=1 down=0 misc=0
[2013-07-31 11:11:59.674199] W [client-rpc-fops.c:1983:client3_3_setattr_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.674292] W [client-rpc-fops.c:1983:client3_3_setattr_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.675293] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk] 0-hosa_slave-client-2: remote operation failed: Success. Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools (dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.675997] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk] 0-hosa_slave-client-3: remote operation failed: Success. Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools (dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.688460] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.688548] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.690008] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691900] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory
[2013-07-31 11:11:59.691959] I [afr-lk-common.c:1085:afr_lock_blocking] 0-hosa_slave-replicate-1: unable to lock on even one child
[2013-07-31 11:11:59.691993] I [afr-transaction.c:1063:afr_post_blocking_inodelk_cbk] 0-hosa_slave-replicate-1: Blocking inodelks failed.
[2013-07-31 11:11:59.693990] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-hosa_slave-client-2: remote operation failed: No such file or directory. Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools (dce9238f-e5c7-49c1-9741-73388f92f050)
[2013-07-31 11:11:59.694328] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-hosa_slave-client-3: remote operation failed: No such file or directory. Path: <gfid:ec6152e9-e1e9-498c-9e05-46edb9d0bc07>/tools (dce9238f-e5c7-49c1-9741-73388f92f050)


Expected results:
Not so much of warning and error logs.
Comment 1 Anand Avati 2015-05-03 07:20:35 UTC 
REVIEW: http://review.gluster.org/10501 (geo-rep: Don't log geo-rep safe errors in mount logs) posted (#1) for review on release-3.7 by Kotresh HR (khiremat)
Comment 2 Anand Avati 2015-05-04 18:25:12 UTC 
REVIEW: http://review.gluster.org/10533 (geo-rep: Fix ignoring geo-rep safe errors) posted (#1) for review on release-3.7 by Kotresh HR (khiremat)
Comment 3 Anand Avati 2015-05-05 06:15:26 UTC 
COMMIT: http://review.gluster.org/10533 committed in release-3.7 by Vijay Bellur (vbellur) 
------
commit 3933109fa34d2e405364b57103f8b6a427acc8ec
Author: Kotresh HR <khiremat>
Date:   Tue Apr 28 18:09:29 2015 +0530

    geo-rep: Fix ignoring geo-rep safe errors
    
    Fix ignoring geo-rep safe errors in fuse layer
    and also ignore logging in client translator
    for mknod. Though it is rare, to happen with
    mknod, it might happen with history crawl on
    overlapping changelogs replay.
    
    BUG: 1217938
    Change-Id: If06f7a6b6f86a315b4e033e294d6f6be67135cb8
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: http://review.gluster.org/10422
    Reviewed-on: http://review.gluster.org/10533
    Reviewed-by: Aravinda VK <avishwan>
    Tested-by: NetBSD Build System
    Tested-by: Gluster Build System <jenkins.com>
Comment 4 Niels de Vos 2015-05-14 17:27:29 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 5 Niels de Vos 2015-05-14 17:28:51 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 6 Niels de Vos 2015-05-14 17:35:21 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user