Created attachment 787227 [details] glusterfs logs Description of problem: If one out of two gluster hosts goes down, the filesystem's in the VM's still running on the host that is still up go into read-only mode. The only way to get them back to R/W mode is to get the failed brick back up and to reboot the VM's. The attached logs are from our testbed environment which is exactly identical to our production setup.
I see quite a bit of logs with split-brain like below: [2013-08-15 09:28:35.431034] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-gluster-replicate-0: background meta-data data self-heal failed on /images/eiswurf.qcow2 [2013-08-15 09:28:43.600072] E [afr-self-heal-data.c:764:afr_sh_data_fxattrop_fstat_done] 0-gluster-replicate-0: Unable to self-heal contents of '/images/eiswurf.qcow2' (possible split-brain). Please delete the file from all but the preferred subvolume. [2013-08-15 09:28:43.600620] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-gluster-replicate-0: background meta-data data self-heal failed on /images/eiswurf.qcow2 [2013-08-15 09:29:33.497170] E [afr-self-heal-data.c:764:afr_sh_data_fxattrop_fstat_done] 0-gluster-replicate-0: Unable to self-heal contents of '/images/eiswurf.qcow2' (possible split-brain). Please delete the file from all but the preferred subvolume. But you are saying only one of the brick is taken down and brought backup. I see a lot of permission denied errors on the bricks logs. What is the issue with permissions?
The version that this bug has been reported against, does not get any updates from the Gluster Community anymore. Please verify if this report is still valid against a current (3.4, 3.5 or 3.6) release and update the version, or close this bug. If there has been no update before 9 December 2014, this bug will get automatocally closed.