Bug 808056 - when a new brick is added to volume, dd fails to create few files on nfs mount
when a new brick is added to volume, dd fails to create few files on nfs mount
Status: CLOSED DUPLICATE of bug 802233
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
mainline
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: shishir gowda
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-29 09:05 EDT by Shwetha Panduranga
Modified: 2015-12-01 11:45 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-07 02:05:19 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Attaching nfs server log (6.93 MB, text/x-log)
2012-03-29 09:05 EDT, Shwetha Panduranga
no flags Details

  None (edit)
Description Shwetha Panduranga 2012-03-29 09:05:48 EDT
Created attachment 573669 [details]
Attaching nfs server log

Description of problem:

When dd was running on nfs mount and performed add-brick operation, dd failed
to create a file reporting the following error and later after some time
continued to create other files.

dd: opening `file.4': Invalid argument
dd: opening `file.5': Invalid argument

nfs server reports:-
---------------------
[2012-03-29 21:47:13.557274] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:ce816c18-ab7b-42b6-b765-8521d5a6e775>: Invalid argument
[2012-03-29 21:47:13.557409] E [nfs3.c:1516:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.2.34:733) dstore : ce816c18-ab7b-42b6-b765-8521d5a6e775
[2012-03-29 21:47:13.569428] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: a93627b9, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)
[2012-03-29 21:47:13.583653] E [nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:ce816c18-ab7b-42b6-b765-8521d5a6e775>: Invalid argument
[2012-03-29 21:47:13.583749] E [nfs3.c:1516:nfs3_access_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.2.34:733) dstore : ce816c18-ab7b-42b6-b765-8521d5a6e775
[2012-03-29 21:47:13.583800] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: aa3627b9, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)

Version-Release number of selected component (if applicable):
mainline

How reproducible:
often

script:-
-----
#!/bin/bash

mountpoint=`pwd`

for i in {1..5}
do 
	level1_dir=$mountpoint/nfs1.$i
	mkdir $level1_dir
	cd $level1_dir

	for j in {1..5}
	do
		level2_dir=dir.$j
		mkdir $level2_dir
		cd $level2_dir
 
		for k in {1..5}
		do 
			dd if=/dev/zero of=file.$k bs="$k"M count=1024 
		done
		cd $level1_dir
	done
	cd $mountpoint
done


Steps to Reproduce:
1.create distribute-replicate volume(2 X 3). start the volume.
2.create nfs mount from client. 
3.run script from nfs mount.
4.add-brick to the volume while the script is still in progress.
  
Actual results:
[03/29/12 - 21:34:18 root@APP-CLIENT1 nfsc1]# /nfsc1.sh 
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 80.854 s, 13.3 MB/s
1024+0 records in
1024+0 records out
2147483648 bytes (2.1 GB) copied, 89.2179 s, 24.1 MB/s
1024+0 records in
1024+0 records out
3221225472 bytes (3.2 GB) copied, 588.557 s, 5.5 MB/s

dd: opening `file.4': Invalid argument
dd: opening `file.5': Invalid argument

1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 38.3277 s, 28.0 MB/s


Expected results:
shouldn't fail to create any files. 

Additional info:
Comment 1 Krishna Srinivas 2012-04-04 09:07:30 EDT
[2012-03-29 21:47:13.557274] E
[nfs3-helpers.c:3603:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup
failed: <gfid:ce816c18-ab7b-42b6-b765-8521d5a6e775>: Invalid argument
[2012-03-29 21:47:13.557409] E [nfs3.c:1516:nfs3_access_resume] 0-nfs-nfsv3:
Unable to resolve FH: (192.168.2.34:733) dstore :
ce816c18-ab7b-42b6-b765-8521d5a6e775

NFS gets error from the translators below (DHR or AFR) for lookups. Most likely DHT is returning error because of the addition of new brick.
Comment 2 shishir gowda 2012-04-27 02:19:30 EDT
I am not able to reproduce this issue after fix for bug 802233.
Can you please confirm?
Comment 3 shishir gowda 2012-05-07 02:05:19 EDT

*** This bug has been marked as a duplicate of bug 802233 ***

Note You need to log in before you can comment on or make changes to this bug.