Description of problem: ======================= after hard reboot of all server glusterd is not cming up on one node and peer info file on that node is trauncated to zero Version-Release number of selected component (if applicable): ============================================================= 0.803.gitf64666f.el6.x86_64 How reproducible: ================= havent tried but intemittent Steps to Reproduce: =================== 1. had cluster of 3 nodes and run few test. 2. unable to login to server so performed hard reboot of all servers 3. glusterd was not coming up on one node after reboot and peer info file was trunctaed to zero [root@rhs-client38 ~]# service glusterd status glusterd is stopped [root@rhs-client38 ~]# service glusterd start Starting glusterd: [FAILED] [root@rhs-client38 ~]# ls -l /var/lib/glusterd/peers/ total 4 -rw------- 1 root root 73 Mar 26 14:29 33fd732c-41c3-4fa5-a588-f7b352333724 -rw------- 1 root root 0 Mar 30 06:29 743cfed7-9578-4fae-8a93-47ef73be22ed [root@rhs-client38 ~]# tail -f /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2015-03-31 00:50:14.787013] I [glusterd.c:154:glusterd_uuid_init] 0-management: retrieved UUID: 9a7435a5-877e-45a4-a94e-4d7ef2ab9cbc [2015-03-31 00:50:14.787107] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2015-03-31 00:50:14.787288] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2015-03-31 00:50:14.787416] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2015-03-31 00:50:14.787551] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2015-03-31 00:50:14.787676] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2015-03-31 00:50:16.050763] E [xlator.c:426:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2015-03-31 00:50:16.050795] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed [2015-03-31 00:50:16.050807] E [graph.c:661:glusterfs_graph_activate] 0-graph: init failed [2015-03-31 00:50:16.051269] W [glusterfsd.c:1212:cleanup_and_exit] (--> 0-: received signum (0), shutting down log snippet :- Mar 30 06:37:30 rhs-client38 kernel: EXT4-fs (dm-0): orphan cleanup on readonly fs Mar 30 06:37:30 rhs-client38 kernel: ------------[ cut here ]------------ Mar 30 06:37:30 rhs-client38 kernel: WARNING: at fs/ext4/inode.c:3929 ext4_flush_unwritten_io+0x74/0x80 [ext4]() (Not tainted) Mar 30 06:37:30 rhs-client38 kernel: Hardware name: X9DRW-3LN4F+/X9DRW-3TF+ Mar 30 06:37:30 rhs-client38 kernel: Modules linked in: ext4 jbd2 mbcache sd_mod crc_t10dif isci libsas scsi_transport_sas ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Mar 30 06:37:30 rhs-client38 kernel: Pid: 862, comm: mount Not tainted 2.6.32-504.12.2.el6.x86_64 #1 Mar 30 06:37:30 rhs-client38 kernel: Call Trace: Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff81074df7>] ? warn_slowpath_common+0x87/0xc0 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff81074e4a>] ? warn_slowpath_null+0x1a/0x20 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00c8bb4>] ? ext4_flush_unwritten_io+0x74/0x80 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00f0937>] ? ext4_ext_truncate+0x37/0x1f0 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00cebf8>] ? ext4_truncate+0x4c8/0x6a0 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff81529445>] ? printk+0x41/0x44 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00dfed8>] ? ext4_msg+0x68/0x80 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00c61d3>] ? ext4_orphan_get+0xb3/0x1f0 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00e50d1>] ? ext4_fill_super+0x26e1/0x28f0 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff812969d4>] ? snprintf+0x34/0x40 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff8119156e>] ? get_sb_bdev+0x18e/0x1d0 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00e29f0>] ? ext4_fill_super+0x0/0x28f0 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffffa00de178>] ? ext4_get_sb+0x18/0x20 [ext4] Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff811909bb>] ? vfs_kern_mount+0x7b/0x1b0 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff81190b62>] ? do_kern_mount+0x52/0x130 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff811b270b>] ? do_mount+0x2fb/0x930 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff81145734>] ? strndup_user+0x64/0xc0 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff811b2dd0>] ? sys_mount+0x90/0xe0 Mar 30 06:37:30 rhs-client38 kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b Mar 30 06:37:30 rhs-client38 kernel: ---[ end trace a0b4d05ba5881b34 ]--- Mar 30 06:37:30 rhs-client38 kernel: EXT4-fs (dm-0): 6 orphan inodes deleted Mar 30 06:37:30 rhs-client38 kernel: EXT4-fs (dm-0): 1 truncate cleaned up Mar 30 06:37:30 rhs-client38 kernel: EXT4-fs (dm-0): recovery complete Mar 30 06:37:30 rhs-client38 kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: Mar 30 06:37:30 rhs-client38 kernel: dracut: Mounted root filesystem /dev/mapper/vg_rhsclient38-lv_root Mar 30 06:37:30 rhs-client38 kernel: SELinux: Disabled at runtime. Actual results: =============== - glusterd not staring - peer info file trucated