Bug 763969 (GLUSTER-2237)

Summary: GlusterFS Stripe Translator Cannot perform random reads
Product: [Community] GlusterFS Reporter: Idan Shinberg <idan>
Component: stripeAssignee: shishir gowda <sgowda>
Status: CLOSED WORKSFORME QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 3.1.1CC: gluster-bugs, nsathyan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: fuse
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Idan Shinberg 2010-12-19 17:08:18 UTC
While Using Io Zone To Test GlusterFS 3.1.1 Random Read on a striped volume , I encounter read errors.

OS - 
Client - Centos 5.2
Servers - Centos 5.5 , Centos 5.5 Xen

Iozone version : 3.291

Steps to reproduce :
--------------------
Use Iozone To create Files:

iozone -t 2 -w  -F io.1 io.2 -i 0 -s 5g -r 50k > 3.1.1.write.2.threads.stripe 2>&1

Then , Random - Read Them Using Iozone :
iozone -t 2 -w  -F io.1 io.2 -i 2 -s 5g -r 50k > 3.1.1.random.2.threads.stripe 2>&1

Creation Reports No Errors , Reading reports the following :

Error reading block at 0
ranread: Success

Error reading block at 0
ranread: Success

Tnks

Comment 1 Amar Tumballi 2010-12-29 14:03:47 UTC
Hi can you share the 'gluster volume info' output ? I tried with stripe of 3 bricks, and things worked without any errors for the same Iozone command. (it was not in xen server though).

Can you also share the glusterfs client log (located @ $prefix/var/log/glusterfs/<mountpoint>.log), which will help in cornering the issue.

Comment 2 Idan Shinberg 2010-12-30 09:47:02 UTC
(In reply to comment #1)
> Hi can you share the 'gluster volume info' output ? I tried with stripe of 3
> bricks, and things worked without any errors for the same Iozone command. (it
> was not in xen server though).
> 
> Can you also share the glusterfs client log (located @
> $prefix/var/log/glusterfs/<mountpoint>.log), which will help in cornering the
> issue.

Gluster Xen Server Is the Dom0 ( Physical Server ) not a virtual guest


Gluster Volume Info

Volume Name: replicate
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: x1:/ssd
Brick2: x2:/ssd
Options Reconfigured:
performance.cache-size: 1024MB
performance.write-behind-window-size: 1024MB
performance.io-thread-count: 64


Gluster Volume log on client

[2010-12-19 18:31:24.350653] I [client-handshake.c:993:select_server_supported_programs] strip-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-12-19 18:31:24.350927] I [client-handshake.c:829:client_setvolume_cbk] strip-client-0: Connected to 192.168.1.1:24010, attached to remote volume '/ssd'.
[2010-12-19 18:31:24.353692] I [client-handshake.c:993:select_server_supported_programs] strip-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-12-19 18:31:24.353989] I [client-handshake.c:829:client_setvolume_cbk] strip-client-1: Connected to 192.168.1.1:24010, attached to remote volume '/ssd'.
[2010-12-19 18:31:24.358840] I [fuse-bridge.c:2817:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.8
[2010-12-19 18:32:02.243793] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 106: /io.1 => -1 (File exists)
[2010-12-19 18:32:02.246767] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 112: /io.2 => -1 (File exists)
[2010-12-19 18:47:29.818022] E [stripe.c:2359:stripe_open_getxattr_cbk] strip-stripe-0: entry-count (1) != stripe-count (2)
[2010-12-19 18:47:29.818080] W [fuse-bridge.c:570:fuse_fd_cbk] glusterfs-fuse: 1021070: OPEN() /3.1.1.write.2.threads.stripe => -1 (Input/output error)
[2010-12-19 18:48:43.478620] E [stripe.c:2359:stripe_open_getxattr_cbk] strip-stripe-0: entry-count (1) != stripe-count (2)
[2010-12-19 18:48:43.478671] W [fuse-bridge.c:570:fuse_fd_cbk] glusterfs-fuse: 1021176: OPEN() /3.1.1.write.2.threads.stripe => -1 (Input/output error)
[2010-12-19 18:49:19.338832] E [stripe.c:2359:stripe_open_getxattr_cbk] strip-stripe-0: entry-count (1) != stripe-count (2)
[2010-12-19 18:49:19.338887] W [fuse-bridge.c:570:fuse_fd_cbk] glusterfs-fuse: 1021179: OPEN() /3.1.1.write.2.threads.stripe => -1 (Input/output error)
[2010-12-19 18:49:23.491022] I [glusterfsd-mgmt.c:349:mgmt_getspec_cbk] : No change in volfile, continuing
[2010-12-19 18:49:52.449812] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: unmounting /strip
[2010-12-19 18:49:52.450251] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd: shutting down
[2010-12-19 18:49:55.407654] W [io-stats.c:1644:init] strip: dangling volume. check volfile 
[2010-12-19 18:49:55.407697] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-12-19 18:49:55.407710] W [dict.c:1204:data_to_str] dict: @data=(nil)

Comment 3 Amar Tumballi 2010-12-30 11:36:09 UTC
The inconsistency in the information you sent is that, the 'volume info' shows that the volume type is 'replicate', but the client log you shared shows stripe specific logs. Can you confirm that this stripe volume also is created by 3.1.x version?

-Amar

Comment 4 Idan Shinberg 2010-12-30 11:59:45 UTC
Sorry About that , I simply moved on to other volume configurations in the meanwhile :-)

The Volume in question was infact a stripe volume created by glusterFS 3.1.1

Comment 5 Amar Tumballi 2011-03-04 08:20:18 UTC
Hi Idan,

Let us know if this issue is still 'critical' for you, and does it gets reproduced in 3.1.2 or later versions.

I am marking it as 'normal' for two things.

1. stripe being not recommended translator
2. we couldn't reproduce this issue.

Comment 6 shishir gowda 2011-06-01 06:40:49 UTC
I ran your tests on a stripe config, and it works fine for me on the latest git (master).
Please re-open the bug if it re-occurs.