Intermittent poorer read performance on 2-way replica mirror. In a GlusterFS configuration where CIFS is being offered via Samba, and GlusterFS is configured with 2- way replica mirrors for redundancy, we have been seeing intermittent poorer read performance. The write performance seems consistent. Sometimes the read performance is significantly better than other times. For example, with 32KB sequential reads, and 32 KB sequential writes, the poor results I was seeing were: Read: 5.5 MB/s, 175 IOPS, Avg Rsp 183 ms, Max Rsp 227 ms Write: 21.0 MB/s, 704 IOPS, Avg Rsp 46 ms, Max Rsp 104 ms better: Read: 29.5 MB/s, 903 IOPS, Avg Rsp 34 ms, Max Rsp 238 ms Write: 22.0 MB/s, 725 IOPS, Avg Rsp 44 ms, Max Rsp 106 ms This has been isolated to be due to the GlusterFS client on the system where the Samba session arrives choosing the remote GlusterFS brick to read from instead of the local one. The replica is in fully sync, so either node would do, but when the remote node is selected, there is an additional network hop, etc. GlusterFS actually seems to favor the remote node, as if the slower responder wins. Actions such as triggering a Gluster log rotation seems to change the current read node. I tried playing with the GlusterFS option 'read- subvolume', but this option looks to have been removed in version 3.1. GlusterFS should, when possible, favor the local storage instead of taking an unnecessary network hop.
Maybe this would be a good reason to get http://review.gluster.com/#change,3005 moving.
Fixed by http://review.gluster.org/#change,3005 (merged June 5) and further enhanced by http://review.gluster.org/#change,4234 (merged December 4) which re-enables NUFA to handle the same case for writes.