Bug 858943 - "remote operation failed: No such file or directory" warning messages reported continuously in bulk around 4839 times
"remote operation failed: No such file or directory" warning messages reporte...
Status: CLOSED WONTFIX
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: replicate (Show other bugs)
2.0
x86_64 Unspecified
medium Severity high
: ---
: ---
Assigned To: Pranith Kumar K
Rahul Hinduja
:
Depends On:
Blocks: 872490
  Show dependency treegraph
 
Reported: 2012-09-20 02:14 EDT by Rahul Hinduja
Modified: 2016-09-19 18:06 EDT (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0qa6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 872490 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
glustershd.log file (690.62 KB, text/x-log)
2012-09-20 02:14 EDT, Rahul Hinduja
no flags Details
New glustershd logs (4.02 MB, text/x-log)
2012-09-21 06:28 EDT, Rahul Hinduja
no flags Details

  None (edit)
Description Rahul Hinduja 2012-09-20 02:14:37 EDT
Created attachment 614742 [details]
glustershd.log file

Description of problem:

"W [client3_1-fops.c:529:client3_1_readlink_cbk] 0-vol0-client-1: remote operation failed: No such file or directory" messages are logged in bulk for around 4839 times during self heal operation.

Version-Release number of selected component (if applicable):

[root@darrel ~]# gluster --version 
glusterfs 3.3.0rhs built on Sep 10 2012 00:49:11

(glusterfs-3.3.0rhs-28.el6rhs.x86_64)

How reproducible:


Steps Carried:
1. Created 2 replica setup (10.70.34.115:/b1 and 10.70.34.119:/b2).
2. Bring down brick b2 ( I performed network stop on 10.70.34.119)
3. Created 20420 entries (files and directories) from the mount point
4. Brought back the brick b2 up (restarted 10.70.34.119)
5. After 10 min Checked the xattr which were all zero(s) and bricks were self healed. 
6. But glustershd.log from 10.70.34.115 continuously reported these warning messages. This message is logged around 4839 times from line 67 to 4906  of the attached logs.

Actual results:

[2012-09-18 04:52:48.805426] W [client3_1-fops.c:529:client3_1_readlink_cbk] 0-vol0-client-1: remote operation failed: No such file or directory
[2012-09-18 04:52:48.808220] W [client3_1-fops.c:529:client3_1_readlink_cbk] 0-vol0-client-1: remote operation failed: No such file or directory
[2012-09-18 04:52:48.810325] W [client3_1-fops.c:529:client3_1_readlink_cbk] 0-vol0-client-1: remote operation failed: No such file or directory
[2012-09-18 04:52:48.810508] W [client3_1-fops.c:529:client3_1_readlink_cbk] 0-vol0-client-1: remote operation failed: No such file or directory

Log file is attached.


Expected results:


Additional info
Comment 2 Rahul Hinduja 2012-09-21 06:28:09 EDT
Created attachment 615361 [details]
New glustershd logs
Comment 3 Rahul Hinduja 2012-09-21 06:30:59 EDT
Raising the severity of this bug. Tried the same scenario with distributed-replicate setup and observed this message for 28962 times continuously.

Total number of lines reported in glustershd.log : 29088
Number of lines reported this message continuously: 28962

Attached the new log
Comment 4 Amar Tumballi 2012-09-25 02:07:18 EDT
Pranith,

Wanted to check if a explicit readlink() is called from self-heal code? otherwise, this issue looks strange to me, because readlink() seems to be called on an non-existing file.
Comment 5 Raghavendra Bhat 2012-11-07 02:24:34 EST
http://review.gluster.org/#change,4153 has been submitted for review.
Comment 6 Amar Tumballi 2012-12-26 04:23:24 EST
ON_QA for rhs-2.1.0 branch.
Comment 9 Raghavendra Bhat 2013-07-02 05:47:46 EDT
Let us use this bug itslef for handling similar ENOENT errors in the protocol/client xlator.
Comment 10 Vivek Agarwal 2015-03-23 03:40:41 EDT
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html
Comment 11 Vivek Agarwal 2015-03-23 03:40:51 EDT
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html

Note You need to log in before you can comment on or make changes to this bug.