Bug 854627 - [glusterfs-3.2.5qa2] dbench fails with 'bad fd' error on a volume with tcp,rdma transport type
[glusterfs-3.2.5qa2] dbench fails with 'bad fd' error on a volume with tcp,rd...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs-rdma (Show other bugs)
2.0
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Raghavendra G
shylesh
: Triaged
Depends On: GLUSTER-3749
Blocks: 849126 858450
  Show dependency treegraph
 
Reported: 2012-09-05 09:26 EDT by Vidya Sakar
Modified: 2015-05-13 13:18 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: GLUSTER-3749
Environment:
Last Closed: 2015-02-13 04:52:00 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Vidya Sakar 2012-09-05 09:26:03 EDT
+++ This bug was initially created as a clone of Bug #765481 +++

Created a 2 way distribute volume with tcp,rdma transport type. Mounted via tcp and started dbench. dbench failed with 'file descriptor in bad state' error. 

[1289] read failed on handle 10137 (File descriptor in bad state)
[1323] read failed on handle 10139 (File descriptor in bad state)
[1311] read failed on handle 10138 (File descriptor in bad state)
[1312] read failed on handle 10138 (File descriptor in bad state)
 100      1443     0.14 MB/sec  execute 243 sec  latency 144553.751 ms
[1290] read failed on handle 10137 (File descriptor in bad state)
[1312] read failed on handle 10138 (File descriptor in bad state)
[1311] read failed on handle 10138 (File descriptor in bad state)
[1289] read failed on handle 10137 (File descriptor in bad state)
[1323] read failed on handle 10139 (File descriptor in bad state)
[1313] read failed on handle 10138 (File descriptor in bad state)
[1352] read failed on handle 10139 (File descriptor in bad state)
[1309] read failed on handle 10138 (File descriptor in bad state)
[1349] read failed on handle 10139 (File descriptor in bad state)
[1313] read failed on handle 10138 (File descriptor in bad state)
[1312] read failed on handle 10138 (File descriptor in bad state)
[1301] read failed on handle 10138 (File descriptor in bad state)
[1314] read failed on handle 10138 (File descriptor in bad state)
[1324] read failed on handle 10139 (File descriptor in bad state)
[1323] read failed on handle 10139 (File descriptor in bad state)
Child failed with status 1
[1290] read failed on handle 10137 (File descriptor in bad state)
[root@client4 mnt]# [1290] read failed on handle 10137 (File descriptor in bad state)
[1324] read failed on handle 10139 (File descriptor in bad state)
[1312] read failed on handle 10138 (File descriptor in bad state)
[1313] read failed on handle 10138 (File descriptor in bad state)
[1291] read failed on handle 10137 (File descriptor in bad state)
[1313] read failed on handle 10138 (File descriptor in bad state)
[1312] read failed on handle 10138 (File descriptor in bad state)
[1290] read failed on handle 10137 (File descriptor in bad state)
[1324] read failed on handle 10139 (File descriptor in bad state)
[1314] read failed on handle 10138 (File descriptor in bad state)
[1353] read failed on handle 10139 (File descriptor in bad state)
[1310] read failed on handle 10138 (File descriptor in bad state)
[1350] read failed on handle 10139 (File descriptor in bad state)
[1314] read failed on handle 10138 (File descriptor in bad state)
[1313] read failed on handle 10138 (File descriptor in bad state)
[1302] read failed on handle 10138 (File descriptor in bad state)
[1315] read failed on handle 10138 (File descriptor in bad state)
[1331] read failed on handle 10139 (File descriptor in bad state)
[1325] read failed on handle 10139 (File descriptor in bad state)


I see lot of these errors in client logs.


[2011-10-21 00:20:02.968436] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.968551] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7869409): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.968571] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.968960] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7606383): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.968983] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.969165] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7867713): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.969185] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.969552] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7606485): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.969574] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.969806] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7606523): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.969827] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.969934] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7869569): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.969954] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.970083] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7606553): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.970103] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.970323] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7606625): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.970343] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.970756] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7606677): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.970778] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.971102] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7607013): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.971122] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.973831] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7867053): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.973855] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.973978] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-hosdu-client-1: sending release on fd
[2011-10-21 00:20:02.974206] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7866561): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.974231] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.974350] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7867271): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.974370] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.974704] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-hosdu-client-1: sending release on fd
[2011-10-21 00:20:02.975006] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7866809): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.975034] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.975090] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-hosdu-client-1: sending release on fd
[2011-10-21 00:20:02.975158] W [client3_1-fops.c:3662:client3_1_flush] 0-hosdu-client-1: (7869623): failed to get fd ctx. EBADFD
[2011-10-21 00:20:02.975183] W [client3_1-fops.c:3692:client3_1_flush] 0-hosdu-client-1: failed to send the fop: File descriptor in bad state
[2011-10-21 00:20:02.975683] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-hosdu-client-1: sending release on fd
[2011-10-21 00:20:02.976080] I [client3_1-fops.c:2364:client_fdctx_destroy] 0-hosdu-client-1: sending release on fd

I have attached the client log.

--- Additional comment from amarts@redhat.com on 2011-10-28 05:51:02 EDT ---

Even this is linked to the 'ping timeout' issue (described in bug 765486). by reducing the number of threads in dbench (from 50 to 20), the issues got fixed.

--- Additional comment from amarts@redhat.com on 2012-09-04 05:30:18 EDT ---

need to check if this happens with latest codebase.
Comment 4 Sudhir D 2013-07-30 01:46:24 EDT
removed from 2.1 as rdma is not slated for this release.
Comment 7 Vivek Agarwal 2014-06-17 08:19:44 EDT
Removing RDMA related bugs, as they are not in scope for  Denali

Note You need to log in before you can comment on or make changes to this bug.