Bug 1500396 - [geo-rep]: Observed "Operation not supported" error with traceback on slave log
Summary: [geo-rep]: Observed "Operation not supported" error with traceback on slave log
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: geo-replication
Version: 3.12
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Kotresh HR
QA Contact:
URL:
Whiteboard:
Depends On: 1308892 1499180
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-10 13:48 UTC by Kotresh HR
Modified: 2017-10-13 12:47 UTC (History)
10 users (show)

Fixed In Version: glusterfs-glusterfs-3.12.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1499180
Environment:
Last Closed: 2017-10-13 12:47:15 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Kotresh HR 2017-10-10 13:48:41 UTC
+++ This bug was initially created as a clone of Bug #1499180 +++

+++ This bug was initially created as a clone of Bug #1308892 +++


Description of problem:
=======================

During the automation run found a traceback with Operation not supported errors as follows: 

[2016-02-12 10:06:56.109858] I [repce(slave):92:service_loop] RepceServer: terminating on reaching EOF.
[2016-02-12 10:06:56.110126] I [syncdutils(slave):220:finalize] <top>: exiting.
[2016-02-12 10:07:02.720356] E [repce(slave):117:worker] <top>: call failed: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 113, in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 685, in entry_ops
    [], [ENOTEMPTY, ESTALE, ENODATA])
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 475, in errno_wrap
    return call(*arg)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 662, in recursive_rmdir
    recursive_rmdir(gfid, entry, fullname)
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 650, in recursive_rmdir
    names = errno_wrap(os.listdir, [path], [ENOENT], [ESTALE])
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 475, in errno_wrap
    return call(*arg)
OSError: [Errno 95] Operation not supported: '.gfid/63d4c437-7990-4b4b-a6fa-c0c6ef942d3f/level88/symlink_to_files'


Other logs reported during readdir as:


[2016-02-12 09:57:51.995660] W [MSGID: 114031]
[client-rpc-fops.c:2669:client3_3_readdirp_cbk] 0-slave-client-8: remote
operation failed [Operation not permitted]
[2016-02-12 09:57:51.996228] W [MSGID: 114031]
[client-rpc-fops.c:2669:client3_3_readdirp_cbk] 0-slave-client-11:
remote operation failed [Operation not permitted]
[2016-02-12 09:57:51.996475] W [MSGID: 114031]
[client-rpc-fops.c:2669:client3_3_readdirp_cbk] 0-slave-client-9: remote
operation failed [Operation not permitted]
[2016-02-12 09:57:51.996700] W [MSGID: 114031]
[client-rpc-fops.c:2669:client3_3_readdirp_cbk] 0-slave-client-4: remote
operation failed [Operation not permitted]

Syncing is done to slave and arequal checksum matches.

Version-Release number of selected component (if applicable):
=============================================================

mainline

--- Additional comment from Worker Ant on 2017-10-06 06:12:24 EDT ---

REVIEW: https://review.gluster.org/18443 (geo-rep: Add ENOTSUP error to retry list) posted (#1) for review on master by Kotresh HR (khiremat)

--- Additional comment from Worker Ant on 2017-10-10 02:41:50 EDT ---

REVIEW: https://review.gluster.org/18443 (geo-rep: Add ENOTSUP error to retry list) posted (#2) for review on master by Kotresh HR (khiremat)

--- Additional comment from Worker Ant on 2017-10-10 09:46:40 EDT ---

COMMIT: https://review.gluster.org/18443 committed in master by Kotresh HR (khiremat) 
------
commit 3afbe4201d864d2e886509f5633468aa344fa444
Author: Kotresh HR <khiremat>
Date:   Fri Oct 6 06:02:56 2017 -0400

    geo-rep: Add ENOTSUP error to retry list
    
    os.listdir gives ENOTSUP on gfid path occasionally
    which is not persistant. Adding it to retry list
    to avoid worker to crash if it's transient error.
    
    Change-Id: Ic795dd1f02a27c9e5d901e20722ee32451838feb
    BUG: 1499180
    Signed-off-by: Kotresh HR <khiremat>

Comment 1 Worker Ant 2017-10-10 13:50:23 UTC
REVIEW: https://review.gluster.org/18469 (geo-rep: Add ENOTSUP error to retry list) posted (#1) for review on release-3.12 by Kotresh HR (khiremat)

Comment 2 Worker Ant 2017-10-12 18:27:18 UTC
COMMIT: https://review.gluster.org/18469 committed in release-3.12 by jiffin tony Thottan (jthottan) 
------
commit e57c81fa658d180e4f70c68bea205a89b7497615
Author: Kotresh HR <khiremat>
Date:   Fri Oct 6 06:02:56 2017 -0400

    geo-rep: Add ENOTSUP error to retry list
    
    os.listdir gives ENOTSUP on gfid path occasionally
    which is not persistant. Adding it to retry list
    to avoid worker to crash if it's transient error.
    
    > Change-Id: Ic795dd1f02a27c9e5d901e20722ee32451838feb
    > BUG: 1499180
    > Signed-off-by: Kotresh HR <khiremat>
    (cherry picked from commit 3afbe4201d864d2e886509f5633468aa344fa444)
    
    Change-Id: Ic795dd1f02a27c9e5d901e20722ee32451838feb
    BUG: 1500396
    Signed-off-by: Kotresh HR <khiremat>

Comment 3 Jiffin 2017-10-13 12:47:15 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-glusterfs-3.12.2, please open a new bug report.

glusterfs-glusterfs-3.12.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-October/032684.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.