Bug 1241807

Summary: Brick crashed after a complete node failure
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Bhaskarakiran <byarlaga>
Component: quotaAssignee: Vijaikumar Mallikarjuna <vmallika>
Status: CLOSED ERRATA QA Contact: Bhaskarakiran <byarlaga>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: annair, asrivast, divya, mselvaga, mzywusko, nsathyan, pkarampu, rhs-bugs, smohan, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.1-12 Doc Type: Bug Fix
Doc Text:
Previously, brick crashed during parallel rename and write operation on a quota enabled volume. With this fix, brick does not crash even if write and rename operations are performed in parallel.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-05 07:19:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1236672, 1244109, 1244116    
Bug Blocks: 1251815    
Attachments:
Description Flags
core file. none

Description Bhaskarakiran 2015-07-10 07:01:17 UTC
Created attachment 1050525 [details]
core file.

Description of problem:
======================

Glusterfsd crashed after a system hard reset. Created a distribute disperse volume and fuse mounted on the client. While the IO is in progress, hard rebooted the system. When it came up bricks failed to come up and there's a crash 

Seeing 4 crashes :

[root@transformers ~]# file /core.4319 
/core.4319: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/glusterfsd -s transformers --volfile-id newvol.transformers.rhs-brick'
[root@transformers ~]# file /core.4296 
/core.4296: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/glusterfsd -s transformers --volfile-id newvol.transformers.rhs-brick'
[root@transformers ~]# file /core.4277 
/core.4277: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/glusterfsd -s transformers --volfile-id newvol.transformers.rhs-brick'
[root@transformers ~]# file /core.4286 
/core.4286: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/glusterfsd -s transformers --volfile-id newvol.transformers.rhs-brick'
[root@transformers ~]# 


Backtrace :
===========

(gdb) bt
#0  0x00007f4387ded064 in mq_remove_contri ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#1  0x00007f4387dedb2c in mq_reduce_parent_size_task ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#2  0x00007f439bd151f2 in synctask_wrap ()
   from /usr/lib64/libglusterfs.so.0
#3  0x00007f439a67d8f0 in ?? () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
(gdb) t a a bt

Thread 34 (Thread 0x7f4327fff700 (LWP 5338)):
#0  0x00007f439a714f05 in _lxstat () from /lib64/libc.so.6
#1  0x00007f438e3c3c13 in posix_handle_hard ()
   from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#2  0x00007f438e3bfe64 in posix_gfid_set ()
   from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#3  0x00007f438e3af998 in posix_create ()
   from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#4  0x00007f439bcd6665 in default_create ()
   from /usr/lib64/libglusterfs.so.0
#5  0x00007f438d9678ed in ctr_create (frame=0x7f439988f5e0, 
    this=0x7f438800a900, loc=0x7f4399313a30, flags=2, mode=420, umask=0, 
    fd=0x7f43880befb8, xdata=0x7f43992830d4) at changetimerecorder.c:1239
#6  0x00007f438d2b09a1 in changelog_create (frame=0x7f439988f1d8, 
    this=0x7f438800d220, loc=0x7f4399313a30, flags=2, mode=420, umask=0, 
    fd=0x7f43880befb8, xdata=0x7f43992830d4) at changelog.c:1110
#7  0x00007f438ce82058 in br_stub_create (frame=0x7f439988bb6c, 
    this=0x7f438800f0c0, loc=0x7f4399313a30, flags=2, mode=420, umask=0, 
    fd=0x7f43880befb8, xdata=0x7f43992830d4) at bit-rot-stub.c:2098
#8  0x00007f438cc78ee1 in posix_acl_create ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/access-control.so
#9  0x00007f438ca61879 in pl_create (frame=0x7f439988b8bc, 
    this=0x7f43880118d0, loc=0x7f4399313a30, flags=2, mode=420, umask=0, 
    fd=0x7f43880befb8, xdata=0x7f43992830d4) at posix.c:1335
#10 0x00007f438c8491bb in up_create (frame=0x7f43998933b0, 
    this=0x7f4388012c50, loc=0x7f4399313a30, flags=2, mode=420, umask=0, 
    fd=0x7f43880befb8, params=0x7f43992830d4) at upcall.c:734
#11 0x00007f439bcda455 in default_create_resume ()
   from /usr/lib64/libglusterfs.so.0
#12 0x00007f439bcf44ca in call_resume_wind ()
   from /usr/lib64/libglusterfs.so.0
#13 0x00007f439bcf8580 in call_resume () from /usr/lib64/libglusterfs.so.0
#14 0x00007f438c63e541 in iot_worker (data=0x7f4388054210)
    at io-threads.c:215
#15 0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7f436f4fd700 (LWP 5212)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7f4374745700 (LWP 5014)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7f43761fc700 (LWP 4809)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438e3bf930 in ?? ()
   from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
---Type <return> to continue, or q <return> to quit---
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f4386504700 (LWP 4685)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f4391e0b700 (LWP 4284)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f439bd14cab in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x00007f439bd19ba0 in syncenv_processor ()
   from /usr/lib64/libglusterfs.so.0
#3  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f439140a700 (LWP 4285)):
#0  0x00007f439a7166c7 in unlink () from /lib64/libc.so.6
#1  0x00007f439bce81a4 in ?? () from /usr/lib64/libglusterfs.so.0
#2  0x00007f439bce8245 in gf_backtrace_save ()
   from /usr/lib64/libglusterfs.so.0
#3  0x00007f439bd14d9c in synctask_yield ()
   from /usr/lib64/libglusterfs.so.0
#4  0x00007f439bd1a776 in syncop_inodelk ()
   from /usr/lib64/libglusterfs.so.0
#5  0x00007f4387ded759 in mq_lock ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#6  0x00007f4387def83b in mq_start_quota_txn_v2 ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#7  0x00007f4387defd71 in mq_initiate_quota_task ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#8  0x00007f4387debb0e in mq_synctask ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#9  0x00007f4387debd9a in _mq_initiate_quota_txn ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#10 0x00007f4387deda10 in mq_reduce_parent_size_task ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#11 0x00007f439bd151f2 in synctask_wrap ()
   from /usr/lib64/libglusterfs.so.0
#12 0x00007f439a67d8f0 in ?? () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()

Thread 27 (Thread 0x7f438756e700 (LWP 4555)):
#0  0x00007f439a722f63 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f439bd317b1 in ?? () from /usr/lib64/libglusterfs.so.0
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f436e8fa700 (LWP 5260)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f439bd14cab in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x00007f439bd19ba0 in syncenv_processor ()
   from /usr/lib64/libglusterfs.so.0
#3  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#4  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f4374442700 (LWP 5026)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f436ddf8700 (LWP 5329)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f437413f700 (LWP 5177)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f4374341700 (LWP 5044)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f4374846700 (LWP 5013)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f4374543700 (LWP 5016)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f436f5fe700 (LWP 5211)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f4374644700 (LWP 5015)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f436f3fc700 (LWP 5258)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f436ffff700 (LWP 4810)):
#0  0x00007f439adbc63c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438e3bedbb in posix_fsyncer_pick ()
   from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#2  0x00007f438e3bf515 in posix_fsyncer ()
   from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#3  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f4374240700 (LWP 5045)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f4386605700 (LWP 4684)):
#0  0x00007f439adbc63c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c227e0b in index_worker (data=<value optimized out>)
    at index.c:71
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f43855fe700 (LWP 4753)):
#0  0x00007f439adbc63c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438d2ba373 in changelog_ev_connector (data=0x7f43880766b0)
    at changelog-ev-handle.c:193
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f43775fe700 (LWP 4756)):
#0  0x00007f439a71b3e3 in select () from /lib64/libc.so.6
#1  0x00007f438d2ba0ba in changelog_ev_dispatch (data=0x7f43880766b0)
    at changelog-ev-handle.c:335
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f4376bfd700 (LWP 4808)):
#0  0x00007f439a6e6aad in nanosleep () from /lib64/libc.so.6
#1  0x00007f439a6e6920 in sleep () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#2  0x00007f438e3c1d5c in ?? ()
   from /usr/lib64/glusterfs/3.7.1/xlator/storage/posix.so
#3  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f4377fff700 (LWP 4755)):
#0  0x00007f439a71b3e3 in select () from /lib64/libc.so.6
#1  0x00007f438d2ba0ba in changelog_ev_dispatch (data=0x7f43880766b0)
    at changelog-ev-handle.c:335
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f438efd1700 (LWP 4315)):
#0  0x00007f439a722f63 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f439bd317b1 in ?? () from /usr/lib64/libglusterfs.so.0
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f436dcf7700 (LWP 5339)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f4386403700 (LWP 4687)):
#0  0x00007f439adbc63c in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438ce836f3 in br_stub_signth (arg=0x7f4388069040)
    at bit-rot-stub.c:649
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f439280c700 (LWP 4283)):
#0  0x00007f439adc0535 in sigwait () from /lib64/libpthread.so.0
#1  0x00007f439c19702b in glusterfs_sigwaiter ()
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f439320d700 (LWP 4282)):
#0  0x00007f439adbffbd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f439bcef5ca in gf_timer_proc ()
   from /usr/lib64/libglusterfs.so.0
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f439c17b740 (LWP 4277)):
#0  0x00007f439adb92ad in pthread_join () from /lib64/libpthread.so.0
#1  0x00007f439bd3142d in ?? () from /usr/lib64/libglusterfs.so.0
#2  0x00007f439c198ef1 in main ()

Thread 3 (Thread 0x7f4384bfd700 (LWP 4754)):
#0  0x00007f439a71b3e3 in select () from /lib64/libc.so.6
#1  0x00007f438d2ba0ba in changelog_ev_dispatch (data=0x7f43880766b0)
    at changelog-ev-handle.c:335
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 2 (Thread 0x7f436def9700 (LWP 5326)):
#0  0x00007f439adbca0e in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f438c63e4a0 in iot_worker (data=0x7f4388054210)
    at io-threads.c:181
#2  0x00007f439adb8a51 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f439a72296d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f436f2fb700 (LWP 5259)):
#0  0x00007f4387ded064 in mq_remove_contri ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#1  0x00007f4387dedb2c in mq_reduce_parent_size_task ()
   from /usr/lib64/glusterfs/3.7.1/xlator/features/marker.so
#2  0x00007f439bd151f2 in synctask_wrap ()
   from /usr/lib64/libglusterfs.so.0
#3  0x00007f439a67d8f0 in ?? () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
(gdb) 
(gdb) 

Version-Release number of selected component (if applicable):
=============================================================

[root@transformers ~]# gluster --version
glusterfs 3.7.1 built on Jul  7 2015 21:34:11
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@transformers ~]# 

How reproducible:
=================
seen once

Steps to Reproduce:
===================
As in description

Actual results:


Expected results:


Additional info:
================
Attaching the core files.

Comment 2 Bhaskarakiran 2015-07-10 07:05:08 UTC
for the volume which has come up there are IO errors when tried to list from the client 

[root@rhs-client17 fuse]# ll 20k_heal/
ls: cannot open directory 20k_heal/: Input/output error
[root@rhs-client17 fuse]# ll 50k_heal/
ls: cannot open directory 50k_heal/: Input/output error
[root@rhs-client17 fuse]# df .
Filesystem          1K-blocks      Used  Available Use% Mounted on
transformers:/vol1 7743007744 145250304 7597757440   2% /mnt/fuse
[root@rhs-client17 fuse]# 

[root@transformers ~]# gluster v status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick transformers:/rhs/brick10/vol1-1      49205     0          Y       4386 
Brick interstellar:/rhs/brick10/vol1-2      49203     0          Y       18690
Brick transformers:/rhs/brick11/vol1-3      49206     0          Y       4373 
Brick interstellar:/rhs/brick11/vol1-4      49204     0          Y       18711
Brick transformers:/rhs/brick11/vol1-5      49207     0          Y       4374 
Brick interstellar:/rhs/brick12/vol1-6      49205     0          Y       18731
Snapshot Daemon on localhost                49208     0          Y       4453 
NFS Server on localhost                     2049      0          Y       4199 
Self-heal Daemon on localhost               N/A       N/A        Y       4204 
Quota Daemon on localhost                   N/A       N/A        Y       4241 
Snapshot Daemon on vertigo                  49168     0          Y       1329 
NFS Server on vertigo                       2049      0          Y       1337 
Self-heal Daemon on vertigo                 N/A       N/A        Y       1219 
Quota Daemon on vertigo                     N/A       N/A        Y       1386 
Snapshot Daemon on 10.70.34.68              49178     0          Y       17058
NFS Server on 10.70.34.68                   2049      0          Y       17083
Self-heal Daemon on 10.70.34.68             N/A       N/A        Y       16973
Quota Daemon on 10.70.34.68                 N/A       N/A        Y       17125
Snapshot Daemon on interstellar             49206     0          Y       18950
NFS Server on interstellar                  2049      0          Y       18964
Self-heal Daemon on interstellar            N/A       N/A        Y       18769
Quota Daemon on interstellar                N/A       N/A        Y       19031
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks

Comment 7 Vijaikumar Mallikarjuna 2015-08-14 06:20:39 UTC
Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/55061/
Upstream patch:  http://review.gluster.org/11706
Release 3.7 patch: http://review.gluster.org/#/c/11707/

Comment 8 Bhaskarakiran 2015-08-25 11:22:37 UTC
verified this on 3.7.1-12 build and didn't see the crash. Marking this as fixed.

Comment 10 errata-xmlrpc 2015-10-05 07:19:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html