Bug 869724

Summary: smbtorture's raw.ping-pong test fails against GlusterFS share
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Jose A. Rivera <jarrpa>
Component: glusterfsAssignee: krishnan parthasarathi <kparthas>
Status: CLOSED ERRATA QA Contact: Ujjwala <ujjwala>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.0CC: amarts, crh, divya, nsathyan, rhs-bugs, sdharane, shaines, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.3.0.5rhs-40 Doc Type: Bug Fix
Doc Text:
Cause: As per fcntl(3) man pages on receiving F_GETLK cmd, the filesystem is expected to get the first lock which blocks the lock description pointed to by the third argument to fcntl(). Prior to this fix, GlusterFS was not doing this. Consequence: smbtorture //server/share raw.ping-pong -U user%pass --option=torture:filename=file --option=torture:num_locks=N --option=torture:read=true --option=torture:write=true Where: server - Samba server share - Samba share user - Samba user (if needed) pass - Samba user's password (if needed) file - Junk file to which to write, preferably not yet extant N - Number of instances of ping_pong that will be run + 1, or greater The above test fails with with STATUS_FILE_LOCK_CONFLICT. Fix: The fix ensures that we return the first lock that blocks the lock, that was supplied as an argument to fcntl(3). Result: The test mentioned in 'Consequence' section doesn't fail with STATUS_FILE_LOCK_CONFLICT.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-28 22:26:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Network capture of failed smbtorture test. none

Description Jose A. Rivera 2012-10-24 15:53:44 UTC
Created attachment 632876 [details]
Network capture of failed smbtorture test.

Description of problem:
smbtorture's raw.ping-pong test fails against a GlusterFS share with STATUS_FILE_LOCK_CONFLICT.

Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
Always

Steps to Reproduce:
1. Create a gluster volume and share it over SMB with CTDB.
2. On the machine running the test:
2.1. Install the smbtorture test suite
2.2. Run the following command:

smbtorture //server/share raw.ping-pong -U user%pass --option=torture:filename=file --option=torture:num_locks=N --option=torture:read=true --option=torture:write=true

Where:

    server - Samba server
    share - Samba share
    user - Samba user (if needed)
    pass - Samba user's password (if needed)
    file - Junk file to which to write, preferably not yet extant
    N - Number of instances of ping_pong that will be run + 1, or greater
3. The test should fail almost immediately.
  
Actual results:
Fails with STATUS_FILE_LOCK_CONFLICT.

Expected results:
Test should pass.

Additional info:
I've attached a network capture showing the SMB traffic, including the failure.

Comment 3 Jose A. Rivera 2012-11-02 19:53:14 UTC
Some further research findings and notes:

 * Single-node Samba server w/o CTDB over a pure XFS filesystem passes the ping-pong test.
 * If Samba is configured to not use POSIX locks ("posix locking = no" in smb.conf), the smbtorture raw.ping-pong test passes against a single-node Samba server.
   * NOTE: This configuration is incompatible with "clustering = yes".

Comment 4 Jose A. Rivera 2012-11-02 20:00:39 UTC
(In reply to comment #3)
> Some further research findings and notes:
> 
>  * Single-node Samba server w/o CTDB over a pure XFS filesystem passes the
> ping-pong test.
>  * If Samba is configured to not use POSIX locks ("posix locking = no" in
> smb.conf), the smbtorture raw.ping-pong test passes against a single-node
> Samba server.
>    * NOTE: This configuration is incompatible with "clustering = yes".

A correction: This configuration is NOT RECOMMENDED to be used in conjunction with "clustering = yes".

Comment 6 Amar Tumballi 2012-11-07 10:33:08 UTC
Patch submitted for review http://review.gluster.org/4164

Comment 9 Jose A. Rivera 2012-11-15 17:55:01 UTC
Success! The proposed patch fixes the locking issue with the smbtorture ping-pong test.

Comment 10 Vijay Bellur 2012-11-19 06:44:10 UTC
CHANGE: http://review.gluster.org/4164 (features/locks: fcntl(3) on F_GETLK must return first conflicting lock) merged in master by Vijay Bellur (vbellur)

Comment 15 Ujjwala 2013-01-10 09:30:38 UTC
Tested it on the build glusterfs-3.3.0.5rhs-40.

Configuration:
CTDB setup on a 4 node cluster.

Steps:
Followed the steps mentioned in the description.

Results:
The single instance of the smbtorture ping_pong runs fine.

1st instance:
--------------------------------------------------------------------------------
[root@RHEL6 ~]# smbtorture //10.16.159.226/gluster-gs raw.ping-pong -U root%redhat --option=torture:filename=file2 --option=torture:num_locks=2 --option=torture:read=true --option=torture:write=true
Using seed 1357809112
time: 2013-01-10 04:11:52.037311
test: ping-pong
time: 2013-01-10 04:11:52.038489
data increment = 1
     522 locks/sec
-------------------------------------------------------------------------------

When I start the 2nd instance,  data increment and locks/sec is not displayed for the 2nd instance.
On the 1st instance, locks/sec remains constant.

2nd instance:
-------------------------------------------------------------------------------
[root@RHEL6 ~]# smbtorture //10.16.159.225/gluster-gs raw.ping-pong -U root%redhat --option=torture:filename=file2 --option=torture:num_locks=2 --option=torture:read=true --option=torture:write=true
Using seed 1357809140
time: 2013-01-10 04:12:20.966015
test: ping-pong
time: 2013-01-10 04:12:20.967518
------------------------------------------------------------------------------

Once the 2nd instance is terminated, the 1st instance continues properly.
Since there is another BZ 888580 for this issue, moving the bug to verified.

Comment 16 Divya 2013-02-12 10:39:21 UTC
KP,

This bug has been added to Update 4 errata. Could you provide your inputs in doc text field which will enable me to update errata??

Thanks,
Divya

Comment 18 errata-xmlrpc 2013-03-28 22:26:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0691.html