DescriptionRiyas Abdulrasak
2016-01-25 08:04:36 UTC
Description of problem:
File creation fails intermittently on a distributed volume, while trying to write concurrently from two clients at the same time.
Version-Release number of selected component (if applicable):
glusterfs-3.7.1-11.el6rhs.x86_64
glusterfs-server-3.7.1-11.el6rhs.x86_64
Red Hat Gluster Storage Server 3.1
How reproducible:
Not always , Intermittent
Steps to Reproduce:
1. Create a distributed volume having 60 bricks or a relative high number of bricks on different nodes.
2. Mount the volume on two different clients.
3. Try to create directories/files from simultaneously from both the clients using the below script.
for i in {1..9}
do
mkdir /mnt/test/$i
cd /mnt/test/$i
touch $HOSTNAME.$i
done
Actual results:
For some files , it fails to create from one of the server. At times it gives , the "No such file/directory" for the file , like below.
[root@server2 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/1': File exists
mkdir: cannot create directory `/mnt/test/2': File exists
mkdir: cannot create directory `/mnt/test/5': File exists
mkdir: cannot create directory `/mnt/test/6': File exists
touch: cannot touch `server2.6': No such file or directory
mkdir: cannot create directory `/mnt/test/7': File exists
mkdir: cannot create directory `/mnt/test/9': File exists
[root@server1 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/3': File exists
mkdir: cannot create directory `/mnt/test/4': File exists
mkdir: cannot create directory `/mnt/test/8': File exists
And at some different point of time it failed giving , stale file handle.
[root@server2 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/4': File exists
touch: cannot touch `server2.4': Stale file handle
mkdir: cannot create directory `/mnt/test/6': File exists
mkdir: cannot create directory `/mnt/test/7': File exists
mkdir: cannot create directory `/mnt/test/8': File exists
mkdir: cannot create directory `/mnt/test/9': File exists
[root@server1 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/1': File exists
mkdir: cannot create directory `/mnt/test/2': File exists
mkdir: cannot create directory `/mnt/test/3': File exists
mkdir: cannot create directory `/mnt/test/5': File exists
Expected results:
The file creation should work without any issue.
Additional info:
[2016-01-16 02:19:01.799335] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-01-16 03:00:36.570750] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 936606817: MKDIR() /pathtofile/0006/897 => -1 (File exists)
[2016-01-16 03:00:36.667744] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 29-volname-dht: Found anomalies in /pathtofile/0006/897 (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2016-01-16 03:00:36.679985] W [MSGID: 114031] [client-rpc-fops.c:2322:client3_3_setattr_cbk] 29-volname-client-62: remote operation failed [Stale file handle]
[2016-01-16 03:00:36.679982] W [MSGID: 114031] [client-rpc-fops.c:2322:client3_3_setattr_cbk] 29-volname-client-63: remote operation failed [Stale file handle]
Verified this BZ on glusterfs version 3.12.2-11.el7rhgs.x86_64.
The script mentioned in the description is tested on a pure distribute - 120 bricks and 8 x 3 volume. Ran the script for 1hr from 4 different clients, didn't see any fail creation failures during this time.
Hence, moving this BZ to Verified state.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2018:2607
Description of problem: File creation fails intermittently on a distributed volume, while trying to write concurrently from two clients at the same time. Version-Release number of selected component (if applicable): glusterfs-3.7.1-11.el6rhs.x86_64 glusterfs-server-3.7.1-11.el6rhs.x86_64 Red Hat Gluster Storage Server 3.1 How reproducible: Not always , Intermittent Steps to Reproduce: 1. Create a distributed volume having 60 bricks or a relative high number of bricks on different nodes. 2. Mount the volume on two different clients. 3. Try to create directories/files from simultaneously from both the clients using the below script. for i in {1..9} do mkdir /mnt/test/$i cd /mnt/test/$i touch $HOSTNAME.$i done Actual results: For some files , it fails to create from one of the server. At times it gives , the "No such file/directory" for the file , like below. [root@server2 ~]# ./test.sh mkdir: cannot create directory `/mnt/test/1': File exists mkdir: cannot create directory `/mnt/test/2': File exists mkdir: cannot create directory `/mnt/test/5': File exists mkdir: cannot create directory `/mnt/test/6': File exists touch: cannot touch `server2.6': No such file or directory mkdir: cannot create directory `/mnt/test/7': File exists mkdir: cannot create directory `/mnt/test/9': File exists [root@server1 ~]# ./test.sh mkdir: cannot create directory `/mnt/test/3': File exists mkdir: cannot create directory `/mnt/test/4': File exists mkdir: cannot create directory `/mnt/test/8': File exists And at some different point of time it failed giving , stale file handle. [root@server2 ~]# ./test.sh mkdir: cannot create directory `/mnt/test/4': File exists touch: cannot touch `server2.4': Stale file handle mkdir: cannot create directory `/mnt/test/6': File exists mkdir: cannot create directory `/mnt/test/7': File exists mkdir: cannot create directory `/mnt/test/8': File exists mkdir: cannot create directory `/mnt/test/9': File exists [root@server1 ~]# ./test.sh mkdir: cannot create directory `/mnt/test/1': File exists mkdir: cannot create directory `/mnt/test/2': File exists mkdir: cannot create directory `/mnt/test/3': File exists mkdir: cannot create directory `/mnt/test/5': File exists Expected results: The file creation should work without any issue. Additional info: [2016-01-16 02:19:01.799335] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2016-01-16 03:00:36.570750] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 936606817: MKDIR() /pathtofile/0006/897 => -1 (File exists) [2016-01-16 03:00:36.667744] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 29-volname-dht: Found anomalies in /pathtofile/0006/897 (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 [2016-01-16 03:00:36.679985] W [MSGID: 114031] [client-rpc-fops.c:2322:client3_3_setattr_cbk] 29-volname-client-62: remote operation failed [Stale file handle] [2016-01-16 03:00:36.679982] W [MSGID: 114031] [client-rpc-fops.c:2322:client3_3_setattr_cbk] 29-volname-client-63: remote operation failed [Stale file handle]