Bug 1473968

Summary: rda-cache-limit filled with (null) value after use parallel-readdir
Product: [Community] GlusterFS Reporter: Vitaly Lipatov <lav>
Component: glusterdAssignee: bugs <bugs>
Status: CLOSED WORKSFORME QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: mainlineCC: amukherj, atumball, bugs, lav, pgurusid
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-14 12:04:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vitaly Lipatov 2017-07-22 20:26:48 UTC
Description of problem:

Any operations failed with rda-cache-limit related error:
# gluster volume set ftp-pvt performance.rda-cache-limit 10M
volume set: failed: invalid number format "5242880(null)" in option "rda-cache-limit"

Version-Release number of selected component (if applicable):
3.10.1, 3.10.3

How reproducible:

I did on 3.10.1
# gluster volume set ftp-pub performance.parallel-readdir on
volume set: success

After it I can't disable it:
# gluster volume set ftp-pub performance.parallel-readdir off
volume set: failed: Staging failed on euclid. Error: invalid number format "5242880(null)" in option "rda-cache-limit"
Staging failed on asu. Error: invalid number format "5242880(null)" in option "rda-cache-limit"
Staging failed on nun. Error: invalid number format "5242880(null)" in option "rda-cache-limit"
Staging failed on cellar. Error: invalid number format "5242880(null)" in option "rda-cache-limit"
Staging failed on multi. Error: invalid number format "5242880(null)" in option "rda-cache-limit"
Staging failed on server. Error: invalid number format "5242880(null)" in option "rda-cache-limit"


Actual results:
files like /var/lib/glusterd/vols/ftp-pub/trusted-ftp-pub.tcp-fuse.vol contains

volume ftp-pvt-readdir-ahead-0
    type performance/readdir-ahead
    option rda-cache-limit 5242880(null)
    option rda-request-size 131072
    option parallel-readdir on
    subvolumes ftp-pvt-replicate-0
end-volume



Expected results:
correct value (without (null))
    option rda-cache-limit 5242880


Additional info:

Comment 1 Atin Mukherjee 2017-07-24 04:54:43 UTC
I believe https://review.gluster.org/17338 fixes this problem.

Poornima - can you please confirm?

Comment 2 Poornima G 2017-07-24 05:04:27 UTC
Yes, this bug is fixed. Could you please try it on 3.11.X latest version?

Comment 3 Vitaly Lipatov 2017-07-24 14:30:55 UTC
I can test 3.10.4 with patch from https://review.gluster.org/17338
but it seems there are no fixes against strange (null) string after number in
in volume info like option rda-cache-limit 5242880(null)

Comment 4 Vitaly Lipatov 2017-07-26 20:58:25 UTC
That patch not about this problem and did not fix it (I have adopted it for 3.10.4 and tried with it).

The problem is left after I ran 
# find /var/lib/glusterd/vols -type f | xargs -n1 subst "s|readdir\([= ]\)on|readdir\1off|g"
on all gluster's servers and restart it.

After it I got
# gluster volume set ftp-pub performance.rda-cache-limit 5MB
volume set: success

Comment 5 Atin Mukherjee 2017-08-08 15:44:35 UTC
Poornima - any comments?

Comment 6 Poornima G 2017-08-09 05:08:07 UTC
I think in 3.10 parallel readdir was only experimental feature, after 3.10 lots of fixes have gone in parallel readdir. Could you please check if the bug is seen on 3.11? If you are doing source install on a test machine, 3.12 will also be a good candidate to test with. Will be interested in knowing if its seen on 3.12/11?

Comment 7 Vitaly Lipatov 2017-11-29 23:46:41 UTC
I just got the same with 3.12.3 after
# gluster volume set ftp-pvt performance.parallel-readdir on


[root@server ftp]# gluster volume set ftp-pvt server.root-squash on
volume set: failed: invalid number format "2621440(null)" in option "rda-cache-limit"

[root@server ftp]# gluster volume get ftp-pvt rda-cache-limit
Option                                  Value                                   
------                                  -----                                   
performance.rda-cache-limit             5MB   

# gluster volume set ftp-pvt rda-cache-limit 10MB
volume set: failed: invalid number format "5242880(null)" in option "rda-cache-limit"

Comment 8 Vitaly Lipatov 2017-11-29 23:53:14 UTC
It seem, the problem near 32 bit system.

vol file on 32 bit host in cluster:
# grep -A5 readdir /var/lib/glusterd/vols/ftp-pvt/trusted-ftp-pvt.tcp-fuse.vol
volume ftp-pvt-readdir-ahead-0
    type performance/readdir-ahead
    option rda-cache-limit 2621440(null)
    option rda-request-size 131072
    option parallel-readdir on
    subvolumes ftp-pvt-replicate-0
end-volume

vol file on 64 bit host in cluster:
# grep -A5 readdir /var/lib/glusterd/vols/ftp-pvt/trusted-ftp-pvt.tcp-fuse.vol
volume ftp-pvt-readdir-ahead-0
    type performance/readdir-ahead
    option rda-cache-limit 2621440B
    option rda-request-size 131072
    option parallel-readdir on
    subvolumes ftp-pvt-replicate-0
end-volume

Comment 9 Shyamsundar 2018-06-20 18:26:11 UTC
This bug reported is against a version of Gluster that is no longer maintained (or has been EOL'd). See https://www.gluster.org/release-schedule/ for the versions currently maintained.

As a result this bug is being closed.

If the bug persists on a maintained version of gluster or against the mainline gluster repository, request that it be reopened and the Version field be marked appropriately.

Comment 10 Vitaly Lipatov 2018-06-20 19:05:07 UTC
Last report was about 3.12.x.

Comment 11 Shyamsundar 2018-10-23 14:54:15 UTC
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.

Comment 12 Amar Tumballi 2019-06-14 08:53:16 UTC
Hi Vitaly, Sorry for being late. (That too late by an year). Is it the issue with glusterfs-6.x releases? We haven't heard the same in bugzilla or mailing list in quite sometime now. 

If not happening for you with latter releases, would like to close the bug.

Comment 13 Amar Tumballi 2019-06-14 08:54:45 UTC
We also fixed many formatting issues with 32bit compiler, and now a CI job runs to validate that none of our patches break 32bit systems. I hope the issues would have got fixed as of th

Comment 14 Vitaly Lipatov 2019-06-14 12:04:08 UTC
(In reply to Amar Tumballi from comment #13)
> We also fixed many formatting issues with 32bit compiler, and now a CI job
> runs to validate that none of our patches break 32bit systems. I hope the
> issues would have got fixed as of th
I've dropped out all 32bit instances of glusterfs, so I believe it runs perfectly after formatting issues have been fixed.