Hide Forgot
Was running Iozone over a 4 distribute setup 256 4 63054 15414 2502 59520 57 55279 24 57463 47 10159 62729 31 50 256 8 121095 113321 73 108244 96 107698 42 108797 74 113980 16623 686 82 256 16 59549 215649 49 211747 57 183249 20 202531 89 179901 200626 1620 189 256 32 335956 280102 92 257801 28 280394 94 326530 53 271186 358041 2018 41 256 64 455522 337299 6068 428014 608075 5453 66 436115 254 495164 384962 47 2960 256 128 641541 460473 133 545951 77 502946 33 366762 37 847682 980843 29 63 256 256 455561 444379 146 418303 58 552915 21 764179 32 567627 711111 32 58 512 4 64926 17587 50 18849 109 17472 1863 17650 36 60398 17780 35 1619 512 8 118302 119457 39 20613 4072 178025 54 102256 37 191401 336 68 379 512 16 193860 195490 3815 22284 158 203255 34 23311 59 188929 39708 48 148 512 32 327782 259640 92 23408 169 250489 1038 44244 220 277356 71229 62 163 512 64 597506 451876 79 37052 86 24229 105 413570 133 423140 448336 41 172 512 128 551754 64095 54 592621 176 505928 27 535005 130 594657 579185 59 154 512 256 727392 577840 141 655664 139 619105 28 579185 61 821829 774583 68 185 512 512 451541 612487 78 612433 113 618357 34 774583 55 681757 231779 53 174 1024 4 26518 62756 48 20765 2936 576 27 1276 50 38888 20628 62 412 1024 8 35934 35326 416 206 218 4034 24 31795 367 55118 759 4425 783 1024 16 178304 190160 6532 31289 365 41093 34 19800 30 37383 188339 46 264 1024 32 338619 329473 109 331173 5840 318903 29 289102 35 320100 23677 46 189 1024 64 476080 83042 64 501735 250 471454 20 35447 240 467153 440999 59 400 1024 128 727224 63745 333 2700 210 614277 36 512256 272 598480 567627 146 427 1024 256 702303 655161 76 745226 333 57126 31 721634 164 785878 509452 112 4138 1024 512 723614 708601 69 789548 259 735632 29 632489 156 686327 779299 66 241 1024 1024 221355 552936 72 618376 211 216170 57 665799 55 Can not fdopen temp file: iozone.tmp 107 fdopen: Transport endpoint is not connected root@ip-10-202-199-95:/mnt/client1/ip-10-202-199-95# ls ls: cannot open directory .: Transport endpoint is not connected Logs: [2010-08-26 20:14:00.280162] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988764 [2010-08-26 20:14:00.280176] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988772 [2010-08-26 20:14:00.280191] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988784 [2010-08-26 20:14:00.280205] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988792 [2010-08-26 20:14:00.280219] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988800 [2010-08-26 20:14:00.280233] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988808 [2010-08-26 20:14:00.280248] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988816 [2010-08-26 20:14:00.280262] D [inode.c:1059:inode_path] statprefetch/inode: no dentry for non-root inode 162988824 [2010-08-26 20:14:00.280287] D [client3_1-fops.c:2260:protocol_client_reopendir] 10.201.207.96-/mnt/ext3/exports: attempting reopen on /ip-10-202-199-95/iozone_L1_0/iozone_L1_0_L2_0 [2010-08-26 20:14:00.280470] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (1) [2010-08-26 20:14:00.280500] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (2) [2010-08-26 20:14:00.280526] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (3) [2010-08-26 20:14:00.280551] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (4) [2010-08-26 20:14:00.280577] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (5) [2010-08-26 20:14:00.280602] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (6) [2010-08-26 20:14:00.280628] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (7) [2010-08-26 20:14:00.281328] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95 returned 0 (8) [2010-08-26 20:14:00.283439] D [client3_1-fops.c:2118:client3_1_reopen_cbk] 10.201.207.96-/mnt/ext3/exports: reopen on /ip-10-202-199-95/million/ddfile_1 returned 0 (9) [2010-08-26 20:14:00.283482] D [client3_1-fops.c:2184:client3_1_reopendir_cbk] 10.201.207.96-/mnt/ext3/exports: reopendir on /ip-10-202-199-95/iozone_L1_0/iozone_L1_0_L2_0 returned 0 (10) [2010-08-26 20:14:54.248660] D [rpc-clnt.c:1247:rpc_clnt_submit] rpc-clnt: cannot build rpc-record [2010-08-26 20:14:54.248709] E [rpc-clnt.c:1086:rpc_clnt_record_build_record] rpc-clnt: Failed to get iobuf [2010-08-26 20:14:54.248725] D [rpc-clnt.c:1163:rpc_clnt_record] rpc-clnt: cannot build rpc-record [2010-08-26 20:14:54.248740] D [rpc-clnt.c:1247:rpc_clnt_submit] rpc-clnt: cannot build rpc-record [2010-08-26 20:14:54.248763] E [rpc-clnt.c:1086:rpc_clnt_record_build_record] rpc-clnt: Failed to get iobuf [2010-08-26 20:14:54.248778] D [rpc-clnt.c:1163:rpc_clnt_record] rpc-clnt: cannot build rpc-record [2010-08-26 20:14:54.248792] D [rpc-clnt.c:1247:rpc_clnt_submit] rpc-clnt: cannot build rpc-record [2010-08-26 20:14:54.272613] E [write-behind.c:1778:__wb_copy_into_holder] writebehind: out of memory [2010-08-26 20:14:54.272654] E [fuse-bridge.c:3071:fuse_thread_proc] fuse: Out of memory [2010-08-26 20:14:54.282833] E [socket.c:1346:__socket_proto_state_machine] 10.201.207.96-/mnt/ext3/exports: unable to allocate IO buffer for peer 10.201.207.96:6970 [2010-08-26 20:14:54.282911] D [rpc-clnt.c:482:rpc_clnt_connection_cleanup] rpc-clnt: cleaning up state in transport object 0x9162950 [2010-08-26 20:14:54.282940] E [rpc-clnt.c:331:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2010-08-26 20:14:54.246420 [2010-08-26 20:14:54.282967] E [rpc-clnt.c:331:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2010-08-26 20:14:54.246543 [2010-08-26 20:14:54.282989] E [rpc-clnt.c:331:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2010-08-26 20:14:54.246557 [2010-08-26 20:14:54.283010] E [rpc-clnt.c:331:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2010-08-26 20:14:54.246564 [2010-08-26 20:14:54.283031] E [rpc-clnt.c:331:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2010-08-26 20:14:54.246571 [2010-08-26 20:14:54.283054] E [rpc-clnt.c:331:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2010-08-26 20:14:54.246578 [2010-08-26 20:16:24.979740] E [rpc-clnt.c:1086:rpc_clnt_record_build_record] rpc-clnt: Failed to get iobuf [2010-08-26 20:16:24.979756] D [rpc-clnt.c:1163:rpc_clnt_record] rpc-clnt: cannot build rpc-record [2010-08-26 20:16:24.979771] D [rpc-clnt.c:1247:rpc_clnt_submit] rpc-clnt: cannot build rpc-record [2010-08-26 20:16:24.979787] W [fuse-bridge.c:190:fuse_entry_cbk] glusterfs-fuse: 42263430: MKDIR() /ip-10-202-199-95/iozone_L1_0 => -1 (Transport endpoint is not connected)
out of memory might be caused by bug 763170. Hence marking this as duplicate of bug 763170 *** This bug has been marked as a duplicate of bug 1438 ***