Bug 765585 (GLUSTER-3853)

I was hoping to back-end cyrus-imapd with glusterfs in order to make a scalable mail system.  Unfortunately, cyrus-imapd is unable to reliably start/function on top of a GlusterFS volume.

Environment overview:
Centos6 2.6.32-71.29.1.el6.x86_64
2 Virtual machines with 1024M RAM, 1vCPU on KVM
Each virtual machine has a 10GB VirtIO raw virtual disk that is used solely for Gluster.

Nodes are called "gluster1" and "gluster2", and there is a dedicated virtual network.  

All hosts/IPs are in /etc/hosts:
192.168.122.11 gluster1.forklocal gluster1
192.168.100.11 gluster1-storage.forklocal gluster1-storage
192.168.122.12 gluster2.forklocal gluster2
192.168.100.12 gluster2-storage.forklocal gluster2-storage

The 10GB disk is formatted with ext4 and mounted at /mnt/glustervol

The gluster volume, mailvol, was created with the following line:
gluster volume create mailvol replica 2 transport tcp gluster1-storage.forklocal:/mnt/glustervol gluster2-storage.forklocal:/mnt/glustervol

[root@gluster1 ~]# gluster volume info

Volume Name: mailvol
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: gluster1-storage.forklocal:/mnt/glustervol
Brick2: gluster2-storage.forklocal:/mnt/glustervol

The Gluster FS is then mounted locally on gluster1.forklocal into the /var/lib/imap folder, which is where Cyrus wants to do its business:
[root@gluster1 ~]# mount -t glusterfs gluster2-storage.forklocal:/mailvol /var/lib/imap
[root@gluster1 ~]# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/vda1 on /boot type ext4 (rw)
/dev/vdb1 on /mnt/glustervol type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
gluster2-storage.forklocal:/mailvol on /var/lib/imap type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)

Starting the cyrus-imapd service is where things get wacky.  I wish that I could provide some kind of real solid description of what goes wrong, but I can't.  I've tried to strace -f starting the cyrus process manually, but it doesn't tell me anything of interest.

* Sometimes cyrus-imapd will start cleanly.  
* If it does manage to start cleanly, sometimes one of the database services hangs or eats tons of CPU.
* If that doesn't happen, I then usually can't make a connection to cyrus with cyradm, the administration tool.
* Sometimes cyrus will complain bitterly about not being able to open files.
* One time I got some strange memory cache / out of memory / unable to write issues.
* Sometimes even if everything starts cleanly, but I'm unable to connect, it won't stop cleanly, and processes need to be killed.
* Sometimes some of the spawned processes can't be killed with -9 or SIGSEGV and the machine must be rebooted.

If I unmount the glusterfs from /var/lib/imap and start the service on the local disk, everything works perfectly.  If I then remount and try again, the issues reappear.

I am happy to provide any kind of output or other information that people are interested in seeing.  But, at this time, the best suggestion I can give is that cyrus is unusable on top of glusterfs in this particular configuration being that it will not reliably start or function.