Bug 763180 (GLUSTER-1448)

Summary: afr - dd command hangs
Product: [Community] GlusterFS Reporter: Lakshmipathi G <lakshmipathi>
Component: replicateAssignee: Raghavendra G <raghavendra>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 3.1-alphaCC: gluster-bugs, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: fuse
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Lakshmipathi G 2010-08-26 14:40:29 UTC 
setup:
3.1.0qa9 ,glfs 4 servers with afr, while running dd to create a 50 mb file ,hangs.but creation of 10000 empty files and fileop works fine. 
Here is server log-
----------------------------

[2010-08-26 07:28:31.735985] D [server-resolve.c:118:resolve_loc_touchup] : return value inode_path 17
[2010-08-26 07:28:31.736150] T [rpcsvc.c:1280:rpcsvc_submit_generic] rpc-service: Tx message: 24
[2010-08-26 07:28:31.736172] T [rpcsvc.c:1108:rpcsvc_record_build_header] rpc-service: Reply fraglen 48, payload: 24, rpc hdr: 24
[2010-08-26 07:28:44.278750] T [rpcsvc-auth.c:276:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_GLUSTERFS
[2010-08-26 07:28:44.278785] T [rpcsvc.c:880:rpcsvc_request_create] rpc-service: RPC XID: 1bd9b6, Ver: 2, Program: 1298437, ProgVers: 310, Proc: 27
[2010-08-26 07:28:44.278804] T [auth-glusterfs.c:176:auth_glusterfs_authenticate] rpc-service: Auth Info: pid: 25441, uid: 0, gid: 0, owner: 25441
[2010-08-26 07:28:44.278820] T [rpcsvc.c:718:rpcsvc_program_actor] rpc-service: Actor found: GlusterFS-3.1.0 - LOOKUP
[2010-08-26 07:28:44.278887] D [server-resolve.c:118:resolve_loc_touchup] : return value inode_path 1
[2010-08-26 07:28:44.278996] T [rpcsvc.c:1280:rpcsvc_submit_generic] rpc-service: Tx message: 516
[2010-08-26 07:28:44.279011] T [rpcsvc.c:1108:rpcsvc_record_build_header] rpc-service: Reply fraglen 540, payload: 516, rpc hdr: 24
[2010-08-26 07:28:44.280147] T [rpcsvc-auth.c:276:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_GLUSTERFS
[2010-08-26 07:28:44.280162] T [rpcsvc.c:880:rpcsvc_request_create] rpc-service: RPC XID: 1bd9b7, Ver: 2, Program: 1298437, ProgVers: 310, Proc: 20
[2010-08-26 07:28:44.280175] T [auth-glusterfs.c:176:auth_glusterfs_authenticate] rpc-service: Auth Info: pid: 25441, uid: 0, gid: 0, owner: 25441
[2010-08-26 07:28:44.280188] T [rpcsvc.c:718:rpcsvc_program_actor] rpc-service: Actor found: GlusterFS-3.1.0 - OPENDIR
[2010-08-26 07:28:44.280206] D [server-resolve.c:118:resolve_loc_touchup] : return value inode_path 1
[2010-08-26 07:28:44.280247] T [rpcsvc.c:1280:rpcsvc_submit_generic] rpc-service: Tx message: 24
[2010-08-26 07:28:44.280268] T [rpcsvc.c:1108:rpcsvc_record_build_header] rpc-service: Reply fraglen 48, payload: 24, rpc hdr: 24
[2010-08-26 07:30:31.22186] D [io-threads.c:107:iot_worker] /mnt: timeout, terminated. conf->curr_count=6
[2010-08-26 07:30:31.22258] D [io-threads.c:107:iot_worker] /mnt: timeout, terminated. conf->curr_count=5
[2010-08-26 07:30:31.22357] D [io-threads.c:107:iot_worker] /mnt: timeout, terminated. conf->curr_count=4
[2010-08-26 07:30:31.23174] D [io-threads.c:107:iot_worker] /mnt: timeout, terminated. conf->curr_count=3
[2010-08-26 07:30:31.23241] D [io-threads.c:107:iot_worker] /mnt: timeout, terminated. conf->curr_count=2
[2010-08-26 07:30:44.22572] D [io-threads.c:107:iot_worker] /mnt: timeout, terminated. conf->curr_count=1
Comment 1 Lakshmipathi G 2010-08-27 11:37:25 UTC 
updates - 4 replicated worked fine while testing with gnfs with qa3,qa6 and qa7 releases.with qa9, dd command works with local machine.
Comment 2 Lakshmipathi G 2010-08-28 04:54:13 UTC 
checked with 3.1.0qa9 with gnfs with 4 replicate ,it works.
Comment 3 Lakshmipathi G 2010-08-28 05:19:00 UTC 
while checking with 3.1.0qa9 without any performance translator,dd command works!
Comment 4 Lakshmipathi G 2010-08-28 06:44:03 UTC 
write-behind seems like causing this issue. dd worked fine without any translator -when write-behind is added ..it hangs.
Comment 5 Lakshmipathi G 2010-08-28 07:34:14 UTC 
with 3.1.0qa9 worked when writebehind flush-behind turned off. 
with 3.0.5, dd worked,with default writebehind options.
Comment 6 Lakshmipathi G 2010-09-01 05:45:09 UTC 
logs file can be found under /share/tickets/1448. 
creating 1mb file hangs . creating empty file "touch f1" works.
Comment 7 Lakshmipathi G 2010-09-03 08:23:35 UTC 
tested with 3.1.0qa14 ,started nfs using dvm commands, and mounted using nfs. even here dd hangs for 25mb file creation.
Comment 8 Lakshmipathi G 2010-09-16 03:18:14 UTC 
tested with qa24 ,dd hangs.
Comment 9 Lakshmipathi G 2010-09-17 04:46:46 UTC 
tested with 3.1.0qa25 ,its not yet fixed.
Comment 10 Lakshmipathi G 2010-09-20 06:58:09 UTC 
(In reply to comment #9)
> tested with 3.1.0qa25 ,its not yet fixed.

checked with 3.1.0qa26 with nfs and fuse mount dd worked.
Comment 11 Lakshmipathi G 2010-09-22 04:10:39 UTC 
tested again,working.after testing against qa28 and this will be moved verified.
Comment 12 Lakshmipathi G 2010-09-28 08:00:59 UTC 
works with qa34.