Description of problem: Log messages like below snippet should go to ERROR or WARNING: ================================================= [2012-02-24 15:53:09.723929] I [dht-layout.c:192:dht_layout_search] 0-gl03storage03-dht: no subvolume for hash (value) = 2181660640 [2012-02-24 15:53:09.731151] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gl03storage03-replicate-1: split-brain possible, no source detected [2012-02-24 15:53:09.731177] I [dht-common.c:479:dht_revalidate_cbk] 0-gl03storage03-dht: subvolume gl03storage03-replicate-1 for /d/96/14 returned -1 (Input/output err or) [2012-02-24 15:53:09.731828] I [dht-layout.c:192:dht_layout_search] 0-gl03storage03-dht: no subvolume for hash (value) = 2825566214 [2012-02-24 15:53:09.738091] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gl03storage03-replicate-1: split-brain possible, no source detected [2012-02-24 15:53:09.738116] I [dht-common.c:479:dht_revalidate_cbk] 0-gl03storage03-dht: subvolume gl03storage03-replicate-1 for /d/97/2e returned -1 (Input/output err or) ==========================================
Some more: [2012-02-27 05:56:26.479592] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gl03storage03-replicate-0: split-brain possible, no source detected [2012-02-27 05:56:26.479706] I [dht-layout.c:581:dht_layout_normalize] 0-gl03storage03-dht: found anomalies in /d/a8/ed. holes=1 overlaps=0 [2012-02-27 05:56:26.479720] I [dht-selfheal.c:576:dht_selfheal_directory] 0-gl03storage03-dht: 1 subvolumes have unrecoverable errors [2012-02-27 05:56:27.366073] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gl03storage03-replicate-1: split-brain possible, no source detected [2012-02-27 05:56:27.366105] I [dht-layout.c:581:dht_layout_normalize] 0-gl03storage03-dht: found anomalies in /d/a8/c0. holes=1 overlaps=0 [2012-02-27 05:56:27.366118] I [dht-selfheal.c:576:dht_selfheal_directory] 0-gl03storage03-dht: 1 subvolumes have unrecoverable errors [2012-02-27 05:56:28.786603] I [afr-self-heal-common.c:705:afr_mark_sources] 0-gl03storage03-replicate-1: split-brain possible, no source detected [2012-02-27 05:56:28.786680] I [dht-layout.c:581:dht_layout_normalize] 0-gl03storage03-dht: found anomalies in /d/a8/de. holes=1 overlaps=0 [2012-02-27 05:56:28.786699] I [dht-selfheal.c:576:dht_selfheal_directory] 0-gl03storage03-dht: 1 subvolumes have unrecoverable errors [2012-02-27 05:56:28.816176] I [dht-layout.c:682:dht_layout_dir_mismatch] 0-gl03storage03-dht: subvol: gl03storage03-replicate-0; inode layout - 0 - 0; disk layout - 0 - 2147483646
CHANGE: http://review.gluster.com/2885 (cluster/distribute: fix log level of few messages) merged in master by Vijay Bellur (vijay)