Bug 762551 - (GLUSTER-819) NFS server hangs on dd run
NFS server hangs on dd run
Status: CLOSED WORKSFORME
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
nfs-alpha
All Linux
low Severity medium
: ---
: ---
Assigned To: Shehjar Tikoo
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-04-12 03:31 EDT by Shehjar Tikoo
Modified: 2015-12-01 11:45 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: RTNR
Mount Type: nfs
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
GlusterFS server log (5.13 KB, application/octet-stream)
2010-04-12 10:50 EDT, Chad Richards
no flags Details
nfs server log part 1 (839.28 KB, application/x-gzip)
2010-04-12 10:58 EDT, Chad Richards
no flags Details
nfs server log part 2 (835.83 KB, application/x-gzip)
2010-04-12 10:59 EDT, Chad Richards
no flags Details

  None (edit)
Description Shehjar Tikoo 2010-04-12 03:31:27 EDT
Reported on gluster-users:
chadr wrote:
> I ran the same dd tests from KnowYourNFSAlpha-1.pdf and performance is
> inconsistent and causes the server to become unresponsive.
> 
> My server freezes every time when I run the following command:
> dd if=/dev/zero of=garb bs=256k count=64000
> 
> I would also like to mount a path like: /volume/some/random/dir 
> 
> # mount host:/gluster/tmp /mnt/test
> mount: host:/gluster/tmp failed, reason given by server: No such file or
> directory
> 
> I can mount it up host:/volume_name and /mnt/test/tmp exists
> 
> dd if=/dev/zero of=garb bs=64K count=100
> 100+0 records in
> 100+0 records out
> 6553600 bytes (6.6 MB) copied, 0.068906 seconds, 95.1 MB/s
> 
> dd of=garb if=/dev/zero bs=64K count=100
> 100+0 records in
> 100+0 records out
> 6553600 bytes (6.6 MB) copied, 0.057207 seconds, 115 MB/s
> 
> dd if=/dev/zero of=garb bs=64K count=1000
> 1000+0 records in
> 1000+0 records out
> 65536000 bytes (66 MB) copied, 0.523117 seconds, 125 MB/s
> 
> dd of=garb if=/dev/zero bs=64K count=1000
> 1000+0 records in
> 1000+0 records out
> 65536000 bytes (66 MB) copied, 1.04666 seconds, 62.6 MB/s
> 
> dd if=/dev/zero of=garb bs=64K count=10000
> 10000+0 records in
> 10000+0 records out
> 655360000 bytes (655 MB) copied, 10.9809 seconds, 59.7 MB/s
> 
> dd of=garb if=/dev/zero bs=64K count=10000
> 10000+0 records in
> 10000+0 records out
> 655360000 bytes (655 MB) copied, 11.3515 seconds, 57.7 MB/s
> 
> dd if=/dev/zero of=garb bs=128K count=100
> 100+0 records in
> 100+0 records out
> 13107200 bytes (13 MB) copied, 0.105364 seconds, 124 MB/s
> 
> dd of=garb if=/dev/zero bs=128K count=100
> 100+0 records in
> 100+0 records out
> 13107200 bytes (13 MB) copied, 0.254225 seconds, 51.6 MB/s
> 
> dd if=/dev/zero of=garb bs=128K count=1000
> 1000+0 records in
> 1000+0 records out
> 131072000 bytes (131 MB) copied, 60.1008 seconds, 2.2 MB/s
> 
> dd of=garb if=/dev/zero bs=128K count=1000
> 1000+0 records in
> 1000+0 records out
> 131072000 bytes (131 MB) copied, 1.51868 seconds, 86.3 MB/s
> 
> dd if=/dev/zero of=garb bs=128K count=10000
> 10000+0 records in
> 10000+0 records out
> 1310720000 bytes (1.3 GB) copied, 18.7755 seconds, 69.8 MB/s
> 
> dd of=garb if=/dev/zero bs=128K count=10000
> 10000+0 records in
> 10000+0 records out
> 1310720000 bytes (1.3 GB) copied, 18.9837 seconds, 69.0 MB/s
> 
> dd if=/dev/zero of=garb bs=256k count=64000
> My server freezes.
> 
> 
> Here is the recent nfs log when the server froze:
> 
> [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3:
> XID: 6f68c85f, WRITE: args: FH: hashcount 2, xlid 0, gen
> 5458285267163021319, ino 11856898, offset: 1129578496,  count: 65536,
> UNSTABLE
> [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service:
> RPC XID: 7068c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3:
> XID: 7068c85f, WRITE: args: FH: hashcount 2, xlid 0, gen
> 5458285267163021319, ino 11856898, offset: 1129644032,  count: 65536,
> UNSTABLE
> [2010-04-09 23:37:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service:
> RPC XID: 7168c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service:
> Actor found: NFS3 - WRITE
> [2010-04-09 23:37:33] D [nfs3-helpers.c:2114:nfs3_log_rw_call] nfs-nfsv3:
> XID: 7168c85f, WRITE: args: FH: hashcount 2, xlid 0, gen
> 5458285267163021319, ino 11856898, offset: 1129709568,  count: 65536,
> UNSTABLE
> [2010-04-09 23:38:33] D [rpcsvc.c:1790:rpcsvc_request_create] rpc-service:
> RPC XID: 6268c85f, Ver: 2, Program: 100003, ProgVers: 3, Proc: 7
> 
> Thanks,
> Chad Richards
> _________________________
Comment 1 Chad Richards 2010-04-12 10:50:18 EDT
Created attachment 176 [details]
RH6.x doesnt work. same code
Comment 2 Chad Richards 2010-04-12 10:58:40 EDT
Created attachment 177 [details]
Fixed EMU10k1 patch. There's only 1 modification SUB_DIRS -> MOD_SUB_DIRS
Comment 3 Chad Richards 2010-04-12 10:59:02 EDT
Created attachment 178 [details]
trivial #ifdef STAT64 => #ifdef HAVE_STAT64
Comment 4 Shehjar Tikoo 2010-04-20 23:55:18 EDT
Chad, I've looked at the NFS server log and it is receiving and serving the write request just fine, right till the end of the log provided. Knowing that the server also freezes up under a unfs+fuse load, I'd suggest testing the same dd load against the NFS translator on a different machine, just to be sure that the problem is unrelated to the kernel freeze.

Thanks
Comment 5 Shehjar Tikoo 2010-05-04 03:13:02 EDT
Closing bug on account of no response from user.

Note You need to log in before you can comment on or make changes to this bug.