Attachment 868205 Details for Bug 1019874 – Logfile file from similar crash during expand/rebalance.

Logfile file from similar crash during expand/rebalance.

samba-SF-Engr.log (text/plain), 81.47 KB, created by Jeff Byers on 2014-02-26 21:23:27 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Jeff Byers

Created: 2014-02-26 21:23:27 UTC

Size: 81.47 KB

patch

obsolete

>[2014-02-26 04:02:40.748155] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:02:40.748204] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:02:40.748230] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5431541: WRITE => -1 (No space left on device)
>[2014-02-26 04:02:40.829038] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-939/VMW2K8R2-108WRC3315.rvf
>[2014-02-26 04:02:50.929067] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:02:50.929104] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:02:50.929129] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5431663: WRITE => -1 (No space left on device)
>[2014-02-26 04:02:51.002702] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-94/VMW2K8R2-108WRC3316.rvf
>[2014-02-26 04:03:01.097701] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:03:01.097977] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:03:01.098015] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5431788: WRITE => -1 (No space left on device)
>[2014-02-26 04:03:01.173823] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-940/VMW2K8R2-108WRC3317.rvf
>[2014-02-26 04:03:11.245528] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:03:11.245610] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:03:11.245643] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5431872: WRITE => -1 (No space left on device)
>[2014-02-26 04:03:11.316091] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-941/VMW2K8R2-108WRC3318.rvf
>[2014-02-26 04:03:21.548680] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:03:21.548809] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:03:21.548845] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432025: WRITE => -1 (No space left on device)
>[2014-02-26 04:03:21.621913] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-942/VMW2K8R2-108WRC3319.rvf
>[2014-02-26 04:03:31.709649] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:03:31.709806] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:03:31.709862] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432109: WRITE => -1 (No space left on device)
>[2014-02-26 04:03:31.793124] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-943/VMW2K8R2-108WRC3320.rvf
>[2014-02-26 04:03:41.888736] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:03:41.888781] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:03:41.888815] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432250: WRITE => -1 (No space left on device)
>[2014-02-26 04:03:41.974687] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-944/VMW2K8R2-108WRC3321.rvf
>[2014-02-26 04:03:52.035705] W [dht-diskusage.c:232:dht_is_subvol_filled] 0-SF-Engr-dht: disk space on subvolume 'SF-Engr-replicate-0' is getting full (100.00 %), consider adding more nodes
>[2014-02-26 04:03:52.051633] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:03:52.051678] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:03:52.051712] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432333: WRITE => -1 (No space left on device)
>[2014-02-26 04:03:52.137716] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-945/VMW2K8R2-108WRC3322.rvf
>[2014-02-26 04:04:02.233716] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:04:02.233764] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:04:02.233797] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432479: WRITE => -1 (No space left on device)
>[2014-02-26 04:04:02.335198] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-946/VMW2K8R2-108WRC3323.rvf
>[2014-02-26 04:04:12.402663] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:04:12.402733] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:04:12.402766] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432567: WRITE => -1 (No space left on device)
>[2014-02-26 04:04:12.488231] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-947/VMW2K8R2-108WRC3324.rvf
>[2014-02-26 04:04:22.575823] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:04:22.576027] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:04:22.576067] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432704: WRITE => -1 (No space left on device)
>[2014-02-26 04:04:22.667687] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-948/VMW2K8R2-108WRC3325.rvf
>[2014-02-26 04:04:32.745028] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:04:32.745230] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:04:32.745269] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432815: WRITE => -1 (No space left on device)
>[2014-02-26 04:04:32.839396] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-949/VMW2K8R2-108WRC3326.rvf
>[2014-02-26 04:04:42.925851] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:04:42.926102] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:04:42.926143] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5432915: WRITE => -1 (No space left on device)
>[2014-02-26 04:04:43.018164] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-95/VMW2K8R2-108WRC3327.rvf
>[2014-02-26 04:04:53.098092] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:04:53.098229] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:04:53.098268] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433001: WRITE => -1 (No space left on device)
>[2014-02-26 04:04:53.196967] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-950/VMW2K8R2-108WRC3328.rvf
>[2014-02-26 04:05:03.639224] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:05:03.639298] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:05:03.639333] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433118: WRITE => -1 (No space left on device)
>[2014-02-26 04:05:03.718531] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-951/VMW2K8R2-108WRC3329.rvf
>[2014-02-26 04:05:13.793093] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:05:13.793302] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:05:13.793340] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433203: WRITE => -1 (No space left on device)
>[2014-02-26 04:05:14.377771] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-952/VMW2K8R2-108WRC3330.rvf
>[2014-02-26 04:05:25.001295] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:05:25.001450] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:05:25.001491] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433290: WRITE => -1 (No space left on device)
>[2014-02-26 04:05:25.096893] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-953/VMW2K8R2-108WRC3331.rvf
>[2014-02-26 04:05:35.172365] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:05:35.172564] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:05:35.172605] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433428: WRITE => -1 (No space left on device)
>[2014-02-26 04:05:35.243390] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-954/VMW2K8R2-108WRC3332.rvf
>[2014-02-26 04:05:45.301708] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:05:45.301800] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:05:45.301886] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433519: WRITE => -1 (No space left on device)
>[2014-02-26 04:05:45.380178] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-955/VMW2K8R2-108WRC3333.rvf
>[2014-02-26 04:05:56.875591] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:05:56.875758] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:05:56.875863] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433632: WRITE => -1 (No space left on device)
>[2014-02-26 04:05:56.974556] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-956/VMW2K8R2-108WRC3334.rvf
>[2014-02-26 04:06:07.055437] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:06:07.055542] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:06:07.055586] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433745: WRITE => -1 (No space left on device)
>[2014-02-26 04:06:07.131461] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-957/VMW2K8R2-108WRC3335.rvf
>[2014-02-26 04:06:17.218154] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:06:17.218345] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:06:17.218391] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433832: WRITE => -1 (No space left on device)
>[2014-02-26 04:06:17.309862] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-958/VMW2K8R2-108WRC3336.rvf
>[2014-02-26 04:06:27.389873] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:06:27.389922] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:06:27.389953] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5433948: WRITE => -1 (No space left on device)
>[2014-02-26 04:06:27.464667] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-959/VMW2K8R2-108WRC3337.rvf
>[2014-02-26 04:06:38.550404] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:06:38.550595] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:06:38.550640] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434088: WRITE => -1 (No space left on device)
>[2014-02-26 04:06:39.658051] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-96/VMW2K8R2-108WRC3338.rvf
>[2014-02-26 04:06:40.611148] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-96/VMW2K8R2-108WRC3338.rvf
>[2014-02-26 04:06:50.702026] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:06:50.702149] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:06:50.702193] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434192: WRITE => -1 (No space left on device)
>[2014-02-26 04:06:50.777846] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-960/VMW2K8R2-108WRC3339.rvf
>[2014-02-26 04:07:00.845409] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:07:00.845483] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:07:00.845522] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434295: WRITE => -1 (No space left on device)
>[2014-02-26 04:07:00.917159] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-961/VMW2K8R2-108WRC3340.rvf
>[2014-02-26 04:07:11.001012] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:07:11.001060] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:07:11.001088] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434380: WRITE => -1 (No space left on device)
>[2014-02-26 04:07:11.119670] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-962/VMW2K8R2-108WRC3341.rvf
>[2014-02-26 04:07:21.181520] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:07:21.181583] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:07:21.181618] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434463: WRITE => -1 (No space left on device)
>[2014-02-26 04:07:21.265435] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-963/VMW2K8R2-108WRC3342.rvf
>[2014-02-26 04:07:31.360706] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:07:31.360885] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:07:31.360919] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434551: WRITE => -1 (No space left on device)
>[2014-02-26 04:07:31.454462] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-964/VMW2K8R2-108WRC3343.rvf
>[2014-02-26 04:07:41.533615] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:07:41.533783] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:07:41.533829] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434659: WRITE => -1 (No space left on device)
>[2014-02-26 04:07:41.611514] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-965/VMW2K8R2-108WRC3344.rvf
>[2014-02-26 04:07:51.701846] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:07:51.702051] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:07:51.702095] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434785: WRITE => -1 (No space left on device)
>[2014-02-26 04:07:51.789860] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-966/VMW2K8R2-108WRC3345.rvf
>[2014-02-26 04:08:01.865631] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:08:01.865802] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:08:01.865833] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5434899: WRITE => -1 (No space left on device)
>[2014-02-26 04:08:01.949959] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-967/VMW2K8R2-108WRC3346.rvf
>[2014-02-26 04:08:12.039948] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:08:12.040124] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:08:12.040214] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435049: WRITE => -1 (No space left on device)
>[2014-02-26 04:08:12.132217] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-968/VMW2K8R2-108WRC3347.rvf
>[2014-02-26 04:08:22.802379] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:08:22.802449] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:08:22.802485] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435182: WRITE => -1 (No space left on device)
>[2014-02-26 04:08:23.892202] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-969/VMW2K8R2-108WRC3348.rvf
>[2014-02-26 04:08:24.944498] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-969/VMW2K8R2-108WRC3348.rvf
>[2014-02-26 04:08:24.974240] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-969/VMW2K8R2-108WRC3348.rvf
>[2014-02-26 04:08:35.083897] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:08:35.083947] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:08:35.083992] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435300: WRITE => -1 (No space left on device)
>[2014-02-26 04:08:35.178291] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-97/VMW2K8R2-108WRC3349.rvf
>[2014-02-26 04:08:45.245881] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:08:45.246042] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:08:45.246081] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435388: WRITE => -1 (No space left on device)
>[2014-02-26 04:08:45.315858] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-970/VMW2K8R2-108WRC3350.rvf
>[2014-02-26 04:08:55.386748] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:08:55.386863] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:08:55.386912] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435474: WRITE => -1 (No space left on device)
>[2014-02-26 04:08:55.467569] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-971/VMW2K8R2-108WRC3351.rvf
>[2014-02-26 04:09:06.338900] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:09:06.338953] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:09:06.338981] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435560: WRITE => -1 (No space left on device)
>[2014-02-26 04:09:07.489373] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-972/VMW2K8R2-108WRC3352.rvf
>[2014-02-26 04:09:08.573144] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-972/VMW2K8R2-108WRC3352.rvf
>[2014-02-26 04:09:08.599871] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-972/VMW2K8R2-108WRC3352.rvf
>[2014-02-26 04:09:18.678658] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:09:18.678814] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:09:18.678854] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435653: WRITE => -1 (No space left on device)
>[2014-02-26 04:09:18.756657] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-973/VMW2K8R2-108WRC3353.rvf
>[2014-02-26 04:09:28.837280] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:09:28.837479] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:09:28.837512] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435744: WRITE => -1 (No space left on device)
>[2014-02-26 04:09:28.910473] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-974/VMW2K8R2-108WRC3354.rvf
>[2014-02-26 04:09:38.985541] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:09:38.985785] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:09:38.985861] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435829: WRITE => -1 (No space left on device)
>[2014-02-26 04:09:39.061816] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-975/VMW2K8R2-108WRC3355.rvf
>[2014-02-26 04:09:50.979799] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:09:50.979883] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:09:50.979919] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5435922: WRITE => -1 (No space left on device)
>[2014-02-26 04:09:51.062727] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-976/VMW2K8R2-108WRC3356.rvf
>[2014-02-26 04:10:01.134433] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:10:01.134475] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:10:01.134504] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436009: WRITE => -1 (No space left on device)
>[2014-02-26 04:10:01.219623] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-977/VMW2K8R2-108WRC3357.rvf
>[2014-02-26 04:10:11.301941] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:10:11.301987] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:10:11.302021] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436097: WRITE => -1 (No space left on device)
>[2014-02-26 04:10:11.383053] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-978/VMW2K8R2-108WRC3358.rvf
>[2014-02-26 04:10:21.440860] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:10:21.440989] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:10:21.441041] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436183: WRITE => -1 (No space left on device)
>[2014-02-26 04:10:21.516187] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-979/VMW2K8R2-108WRC3359.rvf
>[2014-02-26 04:10:32.424176] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:10:32.424218] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:10:32.424246] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436268: WRITE => -1 (No space left on device)
>[2014-02-26 04:10:34.080242] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-98/VMW2K8R2-108WRC3360.rvf
>[2014-02-26 04:10:37.601984] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-98/VMW2K8R2-108WRC3360.rvf
>[2014-02-26 04:10:37.648127] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-98/VMW2K8R2-108WRC3360.rvf
>[2014-02-26 04:10:47.735617] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:10:47.735686] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:10:47.735724] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436363: WRITE => -1 (No space left on device)
>[2014-02-26 04:10:47.799066] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-980/VMW2K8R2-108WRC3361.rvf
>[2014-02-26 04:10:57.891398] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:10:57.891574] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:10:57.891606] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436448: WRITE => -1 (No space left on device)
>[2014-02-26 04:10:57.978667] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-981/VMW2K8R2-108WRC3362.rvf
>[2014-02-26 04:11:08.252351] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:11:08.252532] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:11:08.252569] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436535: WRITE => -1 (No space left on device)
>[2014-02-26 04:11:08.338768] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-982/VMW2K8R2-108WRC3363.rvf
>[2014-02-26 04:11:18.434066] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:11:18.434262] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:11:18.434319] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436624: WRITE => -1 (No space left on device)
>[2014-02-26 04:11:18.516067] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-983/VMW2K8R2-108WRC3364.rvf
>[2014-02-26 04:11:28.971757] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:11:28.972013] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:11:28.972052] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436713: WRITE => -1 (No space left on device)
>[2014-02-26 04:11:29.054561] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-984/VMW2K8R2-108WRC3365.rvf
>[2014-02-26 04:11:40.389384] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:11:40.389455] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:11:40.389483] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436800: WRITE => -1 (No space left on device)
>[2014-02-26 04:11:40.468777] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-985/VMW2K8R2-108WRC3366.rvf
>[2014-02-26 04:11:50.550552] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:11:50.550805] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:11:50.550846] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436885: WRITE => -1 (No space left on device)
>[2014-02-26 04:11:50.626889] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-986/VMW2K8R2-108WRC3367.rvf
>[2014-02-26 04:12:00.696700] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:12:00.696746] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:12:00.696779] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5436973: WRITE => -1 (No space left on device)
>[2014-02-26 04:12:00.788531] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-987/VMW2K8R2-108WRC3368.rvf
>[2014-02-26 04:12:11.068968] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:12:11.069058] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:12:11.069086] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437059: WRITE => -1 (No space left on device)
>[2014-02-26 04:12:11.148748] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-988/VMW2K8R2-108WRC3369.rvf
>[2014-02-26 04:12:21.217546] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:12:21.217644] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:12:21.217684] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437147: WRITE => -1 (No space left on device)
>[2014-02-26 04:12:21.293001] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-989/VMW2K8R2-108WRC3370.rvf
>[2014-02-26 04:12:31.380814] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:12:31.380857] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:12:31.380881] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437231: WRITE => -1 (No space left on device)
>[2014-02-26 04:12:31.452938] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-99/VMW2K8R2-108WRC3371.rvf
>[2014-02-26 04:12:44.702203] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:12:44.702275] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:12:44.702302] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437324: WRITE => -1 (No space left on device)
>[2014-02-26 04:12:44.807722] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-990/VMW2K8R2-108WRC3372.rvf
>[2014-02-26 04:12:55.399620] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:12:55.399803] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:12:55.399841] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437410: WRITE => -1 (No space left on device)
>[2014-02-26 04:12:55.482181] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-991/VMW2K8R2-108WRC3373.rvf
>[2014-02-26 04:13:05.567893] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:13:05.568113] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:13:05.568157] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437497: WRITE => -1 (No space left on device)
>[2014-02-26 04:13:05.641971] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-992/VMW2K8R2-108WRC3374.rvf
>[2014-02-26 04:13:15.736897] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:13:15.737074] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:13:15.737105] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437586: WRITE => -1 (No space left on device)
>[2014-02-26 04:13:15.825678] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-993/VMW2K8R2-108WRC3375.rvf
>[2014-02-26 04:13:25.869987] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:13:25.870207] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:13:25.870247] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437675: WRITE => -1 (No space left on device)
>[2014-02-26 04:13:25.941509] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-994/VMW2K8R2-108WRC3376.rvf
>[2014-02-26 04:13:36.050213] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:13:36.050293] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:13:36.050323] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437759: WRITE => -1 (No space left on device)
>[2014-02-26 04:13:36.133621] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-995/VMW2K8R2-108WRC3377.rvf
>[2014-02-26 04:13:47.672175] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:13:47.672418] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:13:47.672494] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437849: WRITE => -1 (No space left on device)
>[2014-02-26 04:13:48.808982] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-996/VMW2K8R2-108WRC3378.rvf
>[2014-02-26 04:13:50.023493] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-996/VMW2K8R2-108WRC3378.rvf
>[2014-02-26 04:13:50.062590] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-996/VMW2K8R2-108WRC3378.rvf
>[2014-02-26 04:14:00.140606] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-2: remote operation failed: No space left on device
>[2014-02-26 04:14:00.140846] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-3: remote operation failed: No space left on device
>[2014-02-26 04:14:00.140884] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5437942: WRITE => -1 (No space left on device)
>[2014-02-26 04:14:00.222397] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-1: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-997/VMW2K8R2-108WRC3379.rvf
>[2014-02-26 04:14:10.316611] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:14:10.316759] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:14:10.316794] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5438030: WRITE => -1 (No space left on device)
>[2014-02-26 04:14:10.392794] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-998/VMW2K8R2-108WRC3380.rvf
>[2014-02-26 04:14:20.456247] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-0: remote operation failed: No space left on device
>[2014-02-26 04:14:20.456453] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-SF-Engr-client-1: remote operation failed: No space left on device
>[2014-02-26 04:14:20.456498] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 5438117: WRITE => -1 (No space left on device)
>[2014-02-26 04:14:20.536532] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-999/VMW2K8R2-108WRC3381.rvf
>[2014-02-26 05:16:39.058276] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-100/VMW2K8R2-108WRC2379.rvf
>[2014-02-26 05:29:45.429641] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-100/VMW2K8R2-108WRC2379.rvf
>[2014-02-26 05:30:25.544696] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-100/VMW2K8R2-108WRC2379.rvf
>[2014-02-26 05:30:25.628999] I [afr-self-heal-data.c:655:afr_sh_data_fix] 0-SF-Engr-replicate-0: no active sinks for performing self-heal on file /cifs_share/VMW2K8R2-108-100/VMW2K8R2-108WRC2379.rvf
>[2014-02-26 16:59:09.305141] I [dht-layout.c:753:dht_layout_dir_mismatch] 0-SF-Engr-dht: subvol: SF-Engr-replicate-0; inode layout - 0 - 2147483646; disk layout - 0 - 1431655764
>[2014-02-26 16:59:09.305166] I [dht-common.c:623:dht_revalidate_cbk] 0-SF-Engr-dht: mismatching layouts for /cifs_share
>[2014-02-26 16:59:09.305224] I [dht-layout.c:753:dht_layout_dir_mismatch] 0-SF-Engr-dht: subvol: SF-Engr-replicate-1; inode layout - 2147483647 - 4294967295; disk layout - 2863311530 - 4294967295
>[2014-02-26 16:59:09.305237] I [dht-common.c:623:dht_revalidate_cbk] 0-SF-Engr-dht: mismatching layouts for /cifs_share
>[2014-02-26 16:59:09.305694] I [dht-layout.c:638:dht_layout_normalize] 0-SF-Engr-dht: found anomalies in /cifs_share. holes=1 overlaps=0
>[2014-02-26 16:59:11.074558] I [dht-layout.c:753:dht_layout_dir_mismatch] 0-SF-Engr-dht: subvol: SF-Engr-replicate-1; inode layout - 2147483647 - 4294967295; disk layout - 2863311530 - 4294967295
>[2014-02-26 16:59:11.074593] I [dht-common.c:623:dht_revalidate_cbk] 0-SF-Engr-dht: mismatching layouts for /
>[2014-02-26 16:59:11.074664] I [dht-layout.c:753:dht_layout_dir_mismatch] 0-SF-Engr-dht: subvol: SF-Engr-replicate-0; inode layout - 0 - 2147483646; disk layout - 0 - 1431655764
>[2014-02-26 16:59:11.074682] I [dht-common.c:623:dht_revalidate_cbk] 0-SF-Engr-dht: mismatching layouts for /
>[2014-02-26 16:59:11.075233] I [dht-layout.c:638:dht_layout_normalize] 0-SF-Engr-dht: found anomalies in /. holes=1 overlaps=0
>[2014-02-26 16:59:48.113589] I [fuse-bridge.c:4628:fuse_thread_proc] 0-fuse: unmounting /samba/SF-Engr
>[2014-02-26 16:59:48.114172] W [glusterfsd.c:1002:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x30008e8b6d] (-->/lib64/libpthread.so.0() [0x30010079d1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x40529d]))) 0-: received signum (15), shutting down
>[2014-02-26 16:59:48.114195] I [fuse-bridge.c:5260:fini] 0-fuse: Unmounting '/samba/SF-Engr'.
>[2014-02-26 16:59:52.631344] I [glusterfsd.c:1910:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.4.2 (/usr/local/sbin/glusterfs --acl --volfile=/var/lib/glusterd/vols/SF-Engr/SF-Engr-fuse.vol /samba/SF-Engr)
>[2014-02-26 16:59:52.640841] I [socket.c:3480:socket_init] 0-SF-Engr-client-5: SSL support is NOT enabled
>[2014-02-26 16:59:52.640878] I [socket.c:3495:socket_init] 0-SF-Engr-client-5: using system polling thread
>[2014-02-26 16:59:52.641552] I [socket.c:3480:socket_init] 0-SF-Engr-client-4: SSL support is NOT enabled
>[2014-02-26 16:59:52.641565] I [socket.c:3495:socket_init] 0-SF-Engr-client-4: using system polling thread
>[2014-02-26 16:59:52.642224] I [socket.c:3480:socket_init] 0-SF-Engr-client-3: SSL support is NOT enabled
>[2014-02-26 16:59:52.642236] I [socket.c:3495:socket_init] 0-SF-Engr-client-3: using system polling thread
>[2014-02-26 16:59:52.642889] I [socket.c:3480:socket_init] 0-SF-Engr-client-2: SSL support is NOT enabled
>[2014-02-26 16:59:52.642901] I [socket.c:3495:socket_init] 0-SF-Engr-client-2: using system polling thread
>[2014-02-26 16:59:52.643568] I [socket.c:3480:socket_init] 0-SF-Engr-client-1: SSL support is NOT enabled
>[2014-02-26 16:59:52.643582] I [socket.c:3495:socket_init] 0-SF-Engr-client-1: using system polling thread
>[2014-02-26 16:59:52.644239] I [socket.c:3480:socket_init] 0-SF-Engr-client-0: SSL support is NOT enabled
>[2014-02-26 16:59:52.644252] I [socket.c:3495:socket_init] 0-SF-Engr-client-0: using system polling thread
>[2014-02-26 16:59:52.644289] I [client.c:2154:notify] 0-SF-Engr-client-0: parent translators are ready, attempting connect on transport
>[2014-02-26 16:59:52.647509] I [client.c:2154:notify] 0-SF-Engr-client-1: parent translators are ready, attempting connect on transport
>[2014-02-26 16:59:52.650474] I [client.c:2154:notify] 0-SF-Engr-client-2: parent translators are ready, attempting connect on transport
>[2014-02-26 16:59:52.653392] I [client.c:2154:notify] 0-SF-Engr-client-3: parent translators are ready, attempting connect on transport
>[2014-02-26 16:59:52.656230] I [client.c:2154:notify] 0-SF-Engr-client-4: parent translators are ready, attempting connect on transport
>[2014-02-26 16:59:52.659050] I [client.c:2154:notify] 0-SF-Engr-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume SF-Engr-client-0
>  2:     type protocol/client
>  3:     option transport-type tcp
>  4:     option remote-subvolume /exports/nas-segment-0001/SF-Engr
>  5:     option remote-host 10.10.60.92
>  6: end-volume
>  7: 
>  8: volume SF-Engr-client-1
>  9:     type protocol/client
> 10:     option transport-type tcp
> 11:     option remote-subvolume /exports/nas-segment-0002/SF-Engr
> 12:     option remote-host 10.10.60.90
> 13: end-volume
> 14: 
> 15: volume SF-Engr-client-2
> 16:     type protocol/client
> 17:     option transport-type tcp
> 18:     option remote-subvolume /exports/nas-segment-0003/SF-Engr
> 19:     option remote-host 10.10.60.92
> 20: end-volume
> 21: 
> 22: volume SF-Engr-client-3
> 23:     type protocol/client
> 24:     option transport-type tcp
> 25:     option remote-subvolume /exports/nas-segment-0004/SF-Engr
> 26:     option remote-host 10.10.60.90
> 27: end-volume
> 28: 
> 29: volume SF-Engr-client-4
> 30:     type protocol/client
> 31:     option transport-type tcp
> 32:     option remote-subvolume /exports/nas-segment-0005/SF-Engr
> 33:     option remote-host 10.10.60.92
> 34: end-volume
> 35: 
> 36: volume SF-Engr-client-5
> 37:     type protocol/client
> 38:     option transport-type tcp
> 39:     option remote-subvolume /exports/nas-segment-0006/SF-Engr
> 40:     option remote-host 10.10.60.90
> 41: end-volume
> 42: 
> 43: volume SF-Engr-replicate-0
> 44:     type cluster/replicate
> 45:     subvolumes SF-Engr-client-0 SF-Engr-client-1
> 46: end-volume
> 47: 
> 48: volume SF-Engr-replicate-1
> 49:     type cluster/replicate
> 50:     subvolumes SF-Engr-client-2 SF-Engr-client-3
> 51: end-volume
> 52: 
> 53: volume SF-Engr-replicate-2
> 54:     type cluster/replicate
> 55:     subvolumes SF-Engr-client-4 SF-Engr-client-5
> 56: end-volume
> 57: 
> 58: volume SF-Engr-dht
> 59:     type cluster/distribute
> 60:     subvolumes SF-Engr-replicate-0 SF-Engr-replicate-1 SF-Engr-replicate-2
> 61: end-volume
> 62: 
> 63: volume SF-Engr-io-cache
> 64:     type performance/io-cache
> 65:     subvolumes SF-Engr-dht
> 66: end-volume
> 67: 
> 68: volume SF-Engr-quick-read
> 69:     type performance/quick-read
> 70:     subvolumes SF-Engr-io-cache
> 71: end-volume
> 72: 
> 73: volume SF-Engr-open-behind
> 74:     type performance/open-behind
> 75:     subvolumes SF-Engr-quick-read
> 76: end-volume
> 77: 
> 78: volume SF-Engr
> 79:     type debug/io-stats
> 80:     option count-fop-hits off
> 81:     option latency-measurement off
> 82:     subvolumes SF-Engr-open-behind
> 83: end-volume
>
>+------------------------------------------------------------------------------+
>[2014-02-26 16:59:52.662300] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-0: changing port to 49155 (from 0)
>[2014-02-26 16:59:52.662400] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-0: readv failed (No data available)
>[2014-02-26 16:59:52.665401] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-2: changing port to 49158 (from 0)
>[2014-02-26 16:59:52.665436] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-4: changing port to 49161 (from 0)
>[2014-02-26 16:59:52.665462] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-1: changing port to 49156 (from 0)
>[2014-02-26 16:59:52.665513] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-2: readv failed (No data available)
>[2014-02-26 16:59:52.668434] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-4: readv failed (No data available)
>[2014-02-26 16:59:52.671297] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-1: readv failed (No data available)
>[2014-02-26 16:59:52.674133] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-3: changing port to 49159 (from 0)
>[2014-02-26 16:59:52.674159] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-3: readv failed (No data available)
>[2014-02-26 16:59:52.676983] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-5: changing port to 49162 (from 0)
>[2014-02-26 16:59:52.677011] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 16:59:52.677124] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-5: readv failed (No data available)
>[2014-02-26 16:59:52.679999] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 16:59:52.680063] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-0: Connected to 10.10.60.92:49155, attached to remote volume '/exports/nas-segment-0001/SF-Engr'.
>[2014-02-26 16:59:52.680088] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 16:59:52.680142] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-0: Subvolume 'SF-Engr-client-0' came back up; going online.
>[2014-02-26 16:59:52.680170] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 16:59:52.680217] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-2: Connected to 10.10.60.92:49158, attached to remote volume '/exports/nas-segment-0003/SF-Engr'.
>[2014-02-26 16:59:52.680230] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 16:59:52.680267] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-1: Subvolume 'SF-Engr-client-2' came back up; going online.
>[2014-02-26 16:59:52.680288] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-0: Server lk version = 1
>[2014-02-26 16:59:52.680326] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 16:59:52.680363] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 16:59:52.680399] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-2: Server lk version = 1
>[2014-02-26 16:59:52.680439] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-4: Connected to 10.10.60.92:49161, attached to remote volume '/exports/nas-segment-0005/SF-Engr'.
>[2014-02-26 16:59:52.680457] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 16:59:52.680513] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-2: Subvolume 'SF-Engr-client-4' came back up; going online.
>[2014-02-26 16:59:52.680553] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 16:59:52.680597] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-1: Connected to 10.10.60.90:49156, attached to remote volume '/exports/nas-segment-0002/SF-Engr'.
>[2014-02-26 16:59:52.680609] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 16:59:52.680694] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-4: Server lk version = 1
>[2014-02-26 16:59:52.680730] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-3: Connected to 10.10.60.90:49159, attached to remote volume '/exports/nas-segment-0004/SF-Engr'.
>[2014-02-26 16:59:52.680741] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 16:59:52.680829] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-5: Connected to 10.10.60.90:49162, attached to remote volume '/exports/nas-segment-0006/SF-Engr'.
>[2014-02-26 16:59:52.680843] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 16:59:52.684991] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 0
>[2014-02-26 16:59:52.685088] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-5: Server lk version = 1
>[2014-02-26 16:59:52.685115] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-1: Server lk version = 1
>[2014-02-26 16:59:52.685149] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-3: Server lk version = 1
>[2014-02-26 16:59:52.685269] I [fuse-bridge.c:3724:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2014-02-26 16:59:52.685802] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-0: added root inode
>[2014-02-26 16:59:52.686149] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-0: selecting local read_child SF-Engr-client-0
>[2014-02-26 16:59:52.686273] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-2: added root inode
>[2014-02-26 16:59:52.686363] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-1: added root inode
>[2014-02-26 16:59:52.686563] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-2: selecting local read_child SF-Engr-client-4
>[2014-02-26 16:59:52.686589] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-1: selecting local read_child SF-Engr-client-2
>pending frames:
>frame : type(0) op(0)
>frame : type(0) op(0)
>frame : type(0) op(0)
>frame : type(1) op(LOOKUP)
>frame : type(0) op(0)
>
>patchset: git://git.gluster.com/glusterfs.git
>signal received: 11
>time of crash: 2014-02-26 17:44:44configuration details:
>argp 1
>backtrace 1
>dlfcn 1
>fdatasync 1
>libpthread 1
>llistxattr 1
>setfsid 1
>spinlock 1
>epoll.h 1
>xattr.h 1
>st_atim.tv_nsec 1
>package-string: glusterfs 3.4.2
>/lib64/libc.so.6[0x30008329a0]
>/lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x300100c380]
>/usr/local/lib/libglusterfs.so.0(fd_unref+0x36)[0x7f0dd8bb0ee6]
>/usr/local/lib/glusterfs/3.4.2/xlator/protocol/client.so(client_local_wipe+0x28)[0x7f0dd558bc18]
>/usr/local/lib/glusterfs/3.4.2/xlator/protocol/client.so(client3_3_finodelk_cbk+0x157)[0x7f0dd5597237]
>/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7f0dd896fa75]
>/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x11f)[0x7f0dd8970a4f]
>/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f0dd896c388]
>/usr/local/lib/glusterfs/3.4.2/rpc-transport/socket.so(+0x8da6)[0x7f0dd351eda6]
>/usr/local/lib/glusterfs/3.4.2/rpc-transport/socket.so(+0xa95d)[0x7f0dd352095d]
>/usr/local/lib/libglusterfs.so.0(+0x5cdf7)[0x7f0dd8bd4df7]
>/usr/local/sbin/glusterfs(main+0x5c6)[0x406706]
>/lib64/libc.so.6(__libc_start_main+0xfd)[0x300081ed1d]
>/usr/local/sbin/glusterfs[0x4045b9]
>---------
>[2014-02-26 18:35:28.538834] I [glusterfsd.c:1910:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.4.2 (/usr/local/sbin/glusterfs --acl --volfile=/var/lib/glusterd/vols/SF-Engr/SF-Engr-fuse.vol /samba/SF-Engr)
>[2014-02-26 18:35:28.557212] I [socket.c:3480:socket_init] 0-SF-Engr-client-5: SSL support is NOT enabled
>[2014-02-26 18:35:28.557247] I [socket.c:3495:socket_init] 0-SF-Engr-client-5: using system polling thread
>[2014-02-26 18:35:28.557838] I [socket.c:3480:socket_init] 0-SF-Engr-client-4: SSL support is NOT enabled
>[2014-02-26 18:35:28.557851] I [socket.c:3495:socket_init] 0-SF-Engr-client-4: using system polling thread
>[2014-02-26 18:35:28.558430] I [socket.c:3480:socket_init] 0-SF-Engr-client-3: SSL support is NOT enabled
>[2014-02-26 18:35:28.558442] I [socket.c:3495:socket_init] 0-SF-Engr-client-3: using system polling thread
>[2014-02-26 18:35:28.559031] I [socket.c:3480:socket_init] 0-SF-Engr-client-2: SSL support is NOT enabled
>[2014-02-26 18:35:28.559043] I [socket.c:3495:socket_init] 0-SF-Engr-client-2: using system polling thread
>[2014-02-26 18:35:28.559627] I [socket.c:3480:socket_init] 0-SF-Engr-client-1: SSL support is NOT enabled
>[2014-02-26 18:35:28.559638] I [socket.c:3495:socket_init] 0-SF-Engr-client-1: using system polling thread
>[2014-02-26 18:35:28.560229] I [socket.c:3480:socket_init] 0-SF-Engr-client-0: SSL support is NOT enabled
>[2014-02-26 18:35:28.560242] I [socket.c:3495:socket_init] 0-SF-Engr-client-0: using system polling thread
>[2014-02-26 18:35:28.560278] I [client.c:2154:notify] 0-SF-Engr-client-0: parent translators are ready, attempting connect on transport
>[2014-02-26 18:35:28.563424] I [client.c:2154:notify] 0-SF-Engr-client-1: parent translators are ready, attempting connect on transport
>[2014-02-26 18:35:28.566294] I [client.c:2154:notify] 0-SF-Engr-client-2: parent translators are ready, attempting connect on transport
>[2014-02-26 18:35:28.569108] I [client.c:2154:notify] 0-SF-Engr-client-3: parent translators are ready, attempting connect on transport
>[2014-02-26 18:35:28.571931] I [client.c:2154:notify] 0-SF-Engr-client-4: parent translators are ready, attempting connect on transport
>[2014-02-26 18:35:28.574747] I [client.c:2154:notify] 0-SF-Engr-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume SF-Engr-client-0
>  2:     type protocol/client
>  3:     option transport-type tcp
>  4:     option remote-subvolume /exports/nas-segment-0001/SF-Engr
>  5:     option remote-host 10.10.60.92
>  6: end-volume
>  7: 
>  8: volume SF-Engr-client-1
>  9:     type protocol/client
> 10:     option transport-type tcp
> 11:     option remote-subvolume /exports/nas-segment-0002/SF-Engr
> 12:     option remote-host 10.10.60.90
> 13: end-volume
> 14: 
> 15: volume SF-Engr-client-2
> 16:     type protocol/client
> 17:     option transport-type tcp
> 18:     option remote-subvolume /exports/nas-segment-0003/SF-Engr
> 19:     option remote-host 10.10.60.92
> 20: end-volume
> 21: 
> 22: volume SF-Engr-client-3
> 23:     type protocol/client
> 24:     option transport-type tcp
> 25:     option remote-subvolume /exports/nas-segment-0004/SF-Engr
> 26:     option remote-host 10.10.60.90
> 27: end-volume
> 28: 
> 29: volume SF-Engr-client-4
> 30:     type protocol/client
> 31:     option transport-type tcp
> 32:     option remote-subvolume /exports/nas-segment-0005/SF-Engr
> 33:     option remote-host 10.10.60.92
> 34: end-volume
> 35: 
> 36: volume SF-Engr-client-5
> 37:     type protocol/client
> 38:     option transport-type tcp
> 39:     option remote-subvolume /exports/nas-segment-0006/SF-Engr
> 40:     option remote-host 10.10.60.90
> 41: end-volume
> 42: 
> 43: volume SF-Engr-replicate-0
> 44:     type cluster/replicate
> 45:     subvolumes SF-Engr-client-0 SF-Engr-client-1
> 46: end-volume
> 47: 
> 48: volume SF-Engr-replicate-1
> 49:     type cluster/replicate
> 50:     subvolumes SF-Engr-client-2 SF-Engr-client-3
> 51: end-volume
> 52: 
> 53: volume SF-Engr-replicate-2
> 54:     type cluster/replicate
> 55:     subvolumes SF-Engr-client-4 SF-Engr-client-5
> 56: end-volume
> 57: 
> 58: volume SF-Engr-dht
> 59:     type cluster/distribute
> 60:     subvolumes SF-Engr-replicate-0 SF-Engr-replicate-1 SF-Engr-replicate-2
> 61: end-volume
> 62: 
> 63: volume SF-Engr-io-cache
> 64:     type performance/io-cache
> 65:     subvolumes SF-Engr-dht
> 66: end-volume
> 67: 
> 68: volume SF-Engr-quick-read
> 69:     type performance/quick-read
> 70:     subvolumes SF-Engr-io-cache
> 71: end-volume
> 72: 
> 73: volume SF-Engr-open-behind
> 74:     type performance/open-behind
> 75:     subvolumes SF-Engr-quick-read
> 76: end-volume
> 77: 
> 78: volume SF-Engr
> 79:     type debug/io-stats
> 80:     option count-fop-hits off
> 81:     option latency-measurement off
> 82:     subvolumes SF-Engr-open-behind
> 83: end-volume
>
>+------------------------------------------------------------------------------+
>[2014-02-26 18:35:28.577888] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-0: changing port to 49155 (from 0)
>[2014-02-26 18:35:28.577974] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-0: readv failed (No data available)
>[2014-02-26 18:35:28.580968] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-2: changing port to 49158 (from 0)
>[2014-02-26 18:35:28.581026] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-4: changing port to 49161 (from 0)
>[2014-02-26 18:35:28.581089] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-2: readv failed (No data available)
>[2014-02-26 18:35:28.583920] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-4: readv failed (No data available)
>[2014-02-26 18:35:28.586755] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-1: changing port to 49156 (from 0)
>[2014-02-26 18:35:28.586785] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 18:35:28.586841] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-3: changing port to 49159 (from 0)
>[2014-02-26 18:35:28.586876] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-5: changing port to 49162 (from 0)
>[2014-02-26 18:35:28.586954] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-1: readv failed (No data available)
>[2014-02-26 18:35:28.590584] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-3: readv failed (No data available)
>[2014-02-26 18:35:28.593403] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-5: readv failed (No data available)
>[2014-02-26 18:35:28.596224] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 18:35:28.596293] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-0: Connected to 10.10.60.92:49155, attached to remote volume '/exports/nas-segment-0001/SF-Engr'.
>[2014-02-26 18:35:28.596307] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 18:35:28.596353] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-0: Subvolume 'SF-Engr-client-0' came back up; going online.
>[2014-02-26 18:35:28.596380] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 18:35:28.596466] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-4: Connected to 10.10.60.92:49161, attached to remote volume '/exports/nas-segment-0005/SF-Engr'.
>[2014-02-26 18:35:28.596479] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 18:35:28.596524] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-2: Subvolume 'SF-Engr-client-4' came back up; going online.
>[2014-02-26 18:35:28.596545] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-0: Server lk version = 1
>[2014-02-26 18:35:28.596582] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-4: Server lk version = 1
>[2014-02-26 18:35:28.596618] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-2: Connected to 10.10.60.92:49158, attached to remote volume '/exports/nas-segment-0003/SF-Engr'.
>[2014-02-26 18:35:28.596629] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 18:35:28.596675] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-1: Subvolume 'SF-Engr-client-2' came back up; going online.
>[2014-02-26 18:35:28.596713] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 18:35:28.596751] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 18:35:28.596786] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 18:35:28.596820] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-2: Server lk version = 1
>[2014-02-26 18:35:28.597071] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-1: Connected to 10.10.60.90:49156, attached to remote volume '/exports/nas-segment-0002/SF-Engr'.
>[2014-02-26 18:35:28.597097] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 18:35:28.597265] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-5: Connected to 10.10.60.90:49162, attached to remote volume '/exports/nas-segment-0006/SF-Engr'.
>[2014-02-26 18:35:28.597288] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 18:35:28.597411] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-1: Server lk version = 1
>[2014-02-26 18:35:28.597448] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-3: Connected to 10.10.60.90:49159, attached to remote volume '/exports/nas-segment-0004/SF-Engr'.
>[2014-02-26 18:35:28.597471] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 18:35:28.601087] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 0
>[2014-02-26 18:35:28.601178] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-3: Server lk version = 1
>[2014-02-26 18:35:28.601201] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-5: Server lk version = 1
>[2014-02-26 18:35:28.601263] I [fuse-bridge.c:3724:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2014-02-26 18:35:28.601807] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-0: added root inode
>[2014-02-26 18:35:28.602223] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-0: selecting local read_child SF-Engr-client-0
>[2014-02-26 18:35:28.602331] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-1: added root inode
>[2014-02-26 18:35:28.602502] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-2: added root inode
>[2014-02-26 18:35:28.602561] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-1: selecting local read_child SF-Engr-client-2
>[2014-02-26 18:35:28.602698] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-2: selecting local read_child SF-Engr-client-4
>[2014-02-26 19:28:46.030839] I [fuse-bridge.c:4628:fuse_thread_proc] 0-fuse: unmounting /samba/SF-Engr
>[2014-02-26 19:28:46.031199] W [glusterfsd.c:1002:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x30008e8b6d] (-->/lib64/libpthread.so.0() [0x30010079d1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x40529d]))) 0-: received signum (15), shutting down
>[2014-02-26 19:28:46.031217] I [fuse-bridge.c:5260:fini] 0-fuse: Unmounting '/samba/SF-Engr'.
>[2014-02-26 19:39:09.239025] I [glusterfsd.c:1910:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.4.2 (/usr/local/sbin/glusterfs --acl --volfile=/var/lib/glusterd/vols/SF-Engr/SF-Engr-fuse.vol /samba/SF-Engr)
>[2014-02-26 19:39:09.248063] I [socket.c:3480:socket_init] 0-SF-Engr-client-5: SSL support is NOT enabled
>[2014-02-26 19:39:09.248097] I [socket.c:3495:socket_init] 0-SF-Engr-client-5: using system polling thread
>[2014-02-26 19:39:09.248686] I [socket.c:3480:socket_init] 0-SF-Engr-client-4: SSL support is NOT enabled
>[2014-02-26 19:39:09.248698] I [socket.c:3495:socket_init] 0-SF-Engr-client-4: using system polling thread
>[2014-02-26 19:39:09.249288] I [socket.c:3480:socket_init] 0-SF-Engr-client-3: SSL support is NOT enabled
>[2014-02-26 19:39:09.249302] I [socket.c:3495:socket_init] 0-SF-Engr-client-3: using system polling thread
>[2014-02-26 19:39:09.249886] I [socket.c:3480:socket_init] 0-SF-Engr-client-2: SSL support is NOT enabled
>[2014-02-26 19:39:09.249898] I [socket.c:3495:socket_init] 0-SF-Engr-client-2: using system polling thread
>[2014-02-26 19:39:09.250484] I [socket.c:3480:socket_init] 0-SF-Engr-client-1: SSL support is NOT enabled
>[2014-02-26 19:39:09.250495] I [socket.c:3495:socket_init] 0-SF-Engr-client-1: using system polling thread
>[2014-02-26 19:39:09.251080] I [socket.c:3480:socket_init] 0-SF-Engr-client-0: SSL support is NOT enabled
>[2014-02-26 19:39:09.251093] I [socket.c:3495:socket_init] 0-SF-Engr-client-0: using system polling thread
>[2014-02-26 19:39:09.251129] I [client.c:2154:notify] 0-SF-Engr-client-0: parent translators are ready, attempting connect on transport
>[2014-02-26 19:39:09.254327] I [client.c:2154:notify] 0-SF-Engr-client-1: parent translators are ready, attempting connect on transport
>[2014-02-26 19:39:09.257223] I [client.c:2154:notify] 0-SF-Engr-client-2: parent translators are ready, attempting connect on transport
>[2014-02-26 19:39:09.260079] I [client.c:2154:notify] 0-SF-Engr-client-3: parent translators are ready, attempting connect on transport
>[2014-02-26 19:39:09.262935] I [client.c:2154:notify] 0-SF-Engr-client-4: parent translators are ready, attempting connect on transport
>[2014-02-26 19:39:09.265783] I [client.c:2154:notify] 0-SF-Engr-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume SF-Engr-client-0
>  2:     type protocol/client
>  3:     option transport-type tcp
>  4:     option remote-subvolume /exports/nas-segment-0001/SF-Engr
>  5:     option remote-host 10.10.60.92
>  6: end-volume
>  7: 
>  8: volume SF-Engr-client-1
>  9:     type protocol/client
> 10:     option transport-type tcp
> 11:     option remote-subvolume /exports/nas-segment-0002/SF-Engr
> 12:     option remote-host 10.10.60.90
> 13: end-volume
> 14: 
> 15: volume SF-Engr-client-2
> 16:     type protocol/client
> 17:     option transport-type tcp
> 18:     option remote-subvolume /exports/nas-segment-0003/SF-Engr
> 19:     option remote-host 10.10.60.92
> 20: end-volume
> 21: 
> 22: volume SF-Engr-client-3
> 23:     type protocol/client
> 24:     option transport-type tcp
> 25:     option remote-subvolume /exports/nas-segment-0004/SF-Engr
> 26:     option remote-host 10.10.60.90
> 27: end-volume
> 28: 
> 29: volume SF-Engr-client-4
> 30:     type protocol/client
> 31:     option transport-type tcp
> 32:     option remote-subvolume /exports/nas-segment-0005/SF-Engr
> 33:     option remote-host 10.10.60.92
> 34: end-volume
> 35: 
> 36: volume SF-Engr-client-5
> 37:     type protocol/client
> 38:     option transport-type tcp
> 39:     option remote-subvolume /exports/nas-segment-0006/SF-Engr
> 40:     option remote-host 10.10.60.90
> 41: end-volume
> 42: 
> 43: volume SF-Engr-replicate-0
> 44:     type cluster/replicate
> 45:     subvolumes SF-Engr-client-0 SF-Engr-client-1
> 46: end-volume
> 47: 
> 48: volume SF-Engr-replicate-1
> 49:     type cluster/replicate
> 50:     subvolumes SF-Engr-client-2 SF-Engr-client-3
> 51: end-volume
> 52: 
> 53: volume SF-Engr-replicate-2
> 54:     type cluster/replicate
> 55:     subvolumes SF-Engr-client-4 SF-Engr-client-5
> 56: end-volume
> 57: 
> 58: volume SF-Engr-dht
> 59:     type cluster/distribute
> 60:     subvolumes SF-Engr-replicate-0 SF-Engr-replicate-1 SF-Engr-replicate-2
> 61: end-volume
> 62: 
> 63: volume SF-Engr-io-cache
> 64:     type performance/io-cache
> 65:     subvolumes SF-Engr-dht
> 66: end-volume
> 67: 
> 68: volume SF-Engr-quick-read
> 69:     type performance/quick-read
> 70:     subvolumes SF-Engr-io-cache
> 71: end-volume
> 72: 
> 73: volume SF-Engr-open-behind
> 74:     type performance/open-behind
> 75:     subvolumes SF-Engr-quick-read
> 76: end-volume
> 77: 
> 78: volume SF-Engr
> 79:     type debug/io-stats
> 80:     option count-fop-hits off
> 81:     option latency-measurement off
> 82:     subvolumes SF-Engr-open-behind
> 83: end-volume
>
>+------------------------------------------------------------------------------+
>[2014-02-26 19:39:09.269086] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-0: changing port to 49155 (from 0)
>[2014-02-26 19:39:09.269209] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-0: readv failed (No data available)
>[2014-02-26 19:39:09.272097] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-2: changing port to 49158 (from 0)
>[2014-02-26 19:39:09.272134] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-1: changing port to 49156 (from 0)
>[2014-02-26 19:39:09.272166] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-2: readv failed (No data available)
>[2014-02-26 19:39:09.275024] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-1: readv failed (No data available)
>[2014-02-26 19:39:09.277880] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-4: changing port to 49161 (from 0)
>[2014-02-26 19:39:09.277917] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-3: changing port to 49159 (from 0)
>[2014-02-26 19:39:09.277939] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-SF-Engr-client-5: changing port to 49162 (from 0)
>[2014-02-26 19:39:09.277998] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-4: readv failed (No data available)
>[2014-02-26 19:39:09.280856] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-3: readv failed (No data available)
>[2014-02-26 19:39:09.283689] W [socket.c:514:__socket_rwv] 0-SF-Engr-client-5: readv failed (No data available)
>[2014-02-26 19:39:09.286575] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 19:39:09.286682] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 19:39:09.286750] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 19:39:09.286795] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 19:39:09.286836] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 19:39:09.286872] I [client-handshake.c:1659:select_server_supported_programs] 0-SF-Engr-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-26 19:39:09.286922] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-0: Connected to 10.10.60.92:49155, attached to remote volume '/exports/nas-segment-0001/SF-Engr'.
>[2014-02-26 19:39:09.286936] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 19:39:09.286988] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-0: Subvolume 'SF-Engr-client-0' came back up; going online.
>[2014-02-26 19:39:09.287017] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-2: Connected to 10.10.60.92:49158, attached to remote volume '/exports/nas-segment-0003/SF-Engr'.
>[2014-02-26 19:39:09.287027] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 19:39:09.287070] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-1: Subvolume 'SF-Engr-client-2' came back up; going online.
>[2014-02-26 19:39:09.287093] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-0: Server lk version = 1
>[2014-02-26 19:39:09.287112] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-4: Connected to 10.10.60.92:49161, attached to remote volume '/exports/nas-segment-0005/SF-Engr'.
>[2014-02-26 19:39:09.287122] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 19:39:09.287162] I [afr-common.c:3698:afr_notify] 0-SF-Engr-replicate-2: Subvolume 'SF-Engr-client-4' came back up; going online.
>[2014-02-26 19:39:09.287186] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-3: Connected to 10.10.60.90:49159, attached to remote volume '/exports/nas-segment-0004/SF-Engr'.
>[2014-02-26 19:39:09.287196] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 19:39:09.287287] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-1: Connected to 10.10.60.90:49156, attached to remote volume '/exports/nas-segment-0002/SF-Engr'.
>[2014-02-26 19:39:09.287302] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 19:39:09.287412] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-4: Server lk version = 1
>[2014-02-26 19:39:09.287440] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-3: Server lk version = 1
>[2014-02-26 19:39:09.287466] I [client-handshake.c:1456:client_setvolume_cbk] 0-SF-Engr-client-5: Connected to 10.10.60.90:49162, attached to remote volume '/exports/nas-segment-0006/SF-Engr'.
>[2014-02-26 19:39:09.287476] I [client-handshake.c:1468:client_setvolume_cbk] 0-SF-Engr-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-26 19:39:09.290968] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 0
>[2014-02-26 19:39:09.291052] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-2: Server lk version = 1
>[2014-02-26 19:39:09.291102] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-5: Server lk version = 1
>[2014-02-26 19:39:09.291126] I [client-handshake.c:450:client_set_lk_version_cbk] 0-SF-Engr-client-1: Server lk version = 1
>[2014-02-26 19:39:09.291635] I [fuse-bridge.c:3724:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2014-02-26 19:39:09.292282] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-0: added root inode
>[2014-02-26 19:39:09.292608] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-0: selecting local read_child SF-Engr-client-0
>[2014-02-26 19:39:09.292733] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-1: added root inode
>[2014-02-26 19:39:09.292816] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-SF-Engr-replicate-2: added root inode
>[2014-02-26 19:39:09.293060] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-1: selecting local read_child SF-Engr-client-2
>[2014-02-26 19:39:09.293094] I [afr-common.c:2120:afr_discovery_cbk] 0-SF-Engr-replicate-2: selecting local read_child SF-Engr-client-4

Actions: View

Attachments on bug 1019874: 812961 | 868204 | 868205