Bug 1596040

Summary:	eagerlock seems to lose effect when the file being written is also being read
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Nag Pavan Chilakam <nchilaka>
Component:	replicate	Assignee:	Ravishankar N <ravishankar>
Status:	CLOSED ERRATA	QA Contact:	Anees Patel <anepatel>
Severity:	high	Docs Contact:
Priority:	high
Version:	rhgs-3.3	CC:	amukherj, jahernan, nchilaka, pkarampu, ravishankar, rhs-bugs, sanandpa, sankarshan, sheggodu, storage-qa-internal
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.4.z Batch Update 3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.12.2-33	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-02-04 07:41:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1630688
Bug Blocks:

Description Nag Pavan Chilakam 2018-06-28 06:40:22 UTC

Description of problem:
=========================
If a file being written to is being read from another client, I see that finodelk count is increasing.

read doesn't require locks in afr and also if we read a file (no other operation on this file), we don't see any finodelk in profiling

 eager locking is lost if we read a file being written to, which may need to be improvised


Version-Release number of selected component (if applicable):
-----------
3.8.4-54.12

How reproducible:
==============
always

Steps to Reproduce:
1.create a 1x3 volume enable profiling 
2. mount vol on 2 clients
3. write to a file using dd say " dd if=/dev/urandom of=prof bs=1024 count=1000000"
4. from another client do a md5sum on same file



Actual results:
-----------
finodelk count would be huge

Expected results:
----------
findoelk count should be almost zero

Additional info:
----------------
FYI(may not be useful)
you can even check that dirty bit is incrementing in a  rangebound  sequence and then being reset ie it increase from about 0-5 and then again goes back to zero

Comment 10 Ravishankar N 2018-09-19 06:45:27 UTC

1630688 makes eager locking for data transactions independent of fd count, which should fix this BZ. Hence marking this as dependent on it.

Comment 11 Atin Mukherjee 2018-11-11 19:40:26 UTC

So isn't this a good candidate for qualification for 3.4.3 considering the content is already part of the latest release?

Comment 12 Ravishankar N 2018-11-15 05:11:31 UTC

(In reply to Atin Mukherjee from comment #11)
> So isn't this a good candidate for qualification for 3.4.3 considering the
> content is already part of the latest release?

Agree. Nag, Irrespective of the broader changes mentioned in comment#5 and 7, Pranith's eager lock patch does fix this particular bug. i.e. eager lock won't be lost when we read a file that is being written to simultaneously. Please provide ack for 3.4.3 if there is agreement to test it.

Comment 13 Nag Pavan Chilakam 2018-12-03 08:41:45 UTC

(In reply to Ravishankar N from comment #12)
> (In reply to Atin Mukherjee from comment #11)
> > So isn't this a good candidate for qualification for 3.4.3 considering the
> > content is already part of the latest release?
> 
> Agree. Nag, Irrespective of the broader changes mentioned in comment#5 and
> 7, Pranith's eager lock patch does fix this particular bug. i.e. eager lock
> won't be lost when we read a file that is being written to simultaneously.
> Please provide ack for 3.4.3 if there is agreement to test it.

I believe this can be taken in 3.4.3 only after assessing other bugs and timelines.
Kindly go ahead and propose for 3.4.3 from dev side, and then QE would look into this if it is possible to take in 3.4.3 or not

Comment 16 Sunil Kumar Acharya 2018-12-17 09:37:13 UTC

Fix is already part of downstream code as mentioned in comment #10

Comment 19 Anees Patel 2018-12-26 05:21:57 UTC

Hi Ravi,

I have created a test-plan to verify the fix,
https://docs.google.com/spreadsheets/d/1bX1EyPNnVUU_3PJa-HfvzJe4stMWQix7G02YSSPAROE/edit?usp=sharing
Kindly go thru the test-plan and please provide your feedback

Comment 20 Anees Patel 2018-12-26 05:23:07 UTC

Setting need_info on Ravi for the above comment

Comment 21 Ravishankar N 2018-12-27 12:09:07 UTC

Hi Anees, The first 3 tests look fine. The 4th one seems to be multiple writers to the same file and no reads, which does not seem to relate to this specific bug. You might experience more inodelks/xattrops with multiple writers than with single writer.

Comment 22 Anees Patel 2018-12-28 10:23:44 UTC

Hi Ravi,

Thank-you for the feed-back,

I verified the fix on the latest build
# rpm -qa | grep gluster
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-rdma-3.12.2-34.el7rhgs.x86_64
glusterfs-server-3.12.2-34.el7rhgs.x86_64
glusterfs-client-xlators-3.12.2-34.el7rhgs.x86_64
glusterfs-fuse-3.12.2-34.el7rhgs.x86_64
glusterfs-events-3.12.2-34.el7rhgs.x86_64
glusterfs-devel-3.12.2-34.el7rhgs.x86_64


finodelks were well close to one or two with the above test-cases.
The max count that I was able to get with single writer was 3.

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              2     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              8  RELEASEDIR
      0.00      36.00 us      30.00 us      42.00 us              2       FLUSH
      0.00     181.00 us     181.00 us     181.00 us              1        OPEN
      0.00     101.50 us      70.00 us     133.00 us              2     INODELK
      0.00      72.67 us      53.00 us     108.00 us              3    FINODELK
      0.00     180.67 us     140.00 us     209.00 us              3    FXATTROP
      0.00     308.50 us     281.00 us     336.00 us              2     XATTROP
      0.00     317.00 us     282.00 us     352.00 us              2      RENAME
      0.00      86.88 us      38.00 us     155.00 us              8     ENTRYLK
      0.00     142.12 us     103.00 us     204.00 us              8     OPENDIR
      0.02     175.77 us      81.00 us     477.00 us             26      LOOKUP
      0.34  101170.00 us  101170.00 us  101170.00 us              1    TRUNCATE
     99.63     168.52 us      67.00 us 1083136.00 us         176836       WRITE


Hence setting this bug to verified

Comment 24 errata-xmlrpc 2019-02-04 07:41:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0263