Red Hat Bugzilla – Bug 822253
Poor disk performance
Last modified: 2013-12-18 19:08:10 EST
we are currently running glusterfs 3.2.6 on a two node cluster that is hosting a replicated volume. Both storage servers have high end hardware: 22 disks in a hardware raid and dual sandy bridge quad core CPUs. There is 16TB of data stored on the volume, the average file size is 4mb. The default configuration was not changed. We are using TCP transport and XFS.
There are seven dedicated servers that are mounting the volume and make it available through nginx http server. There is lots of concurrency and gluster seems to have problems with this and is only capable to deliver 5-10% of the actual hardware ressources due to very inefficient disk usage.
Doing high concurrent reads directly on the underlying FS gives 10-15 times as much throughput compared to doing a localhost-mount and doing the reads on the gluster mount. Here is a quick hacked php script that should make it easy to reproduce this behaviour: http://pastebin.com/6bPj3GRt
The outcome is high-end hardware that is close to being overloaded at a few 100Mbit/s of throughput: http://pastebin.com/4BQk3CKZ (sdb is the 22disk hardware raid which is *only* hosting a single brick)
Such poor disk efficiency makes gluster nearly unusable and clearly a very bad choice cost wise.
If there is any additional info you need please let me know.
Gather some 'profile' information using the following approach,
- # gluster volume profile VOLNAME start
- Run workload
- # gluster volume profile VOLNAME info (Repeat this a few times while the workload is present on the volume)
- gluster volume profile VOLNAME stop (once the workload is complete)
This should shed more light on 'what' is happening.
please also try with the 3.3.0 version.
Philip, any update on this issue? did you happen to try newer version?
Philip, we will close the bug as INSUFFICIENT_DATA, please re-open with more data and would be great if you give new version a try before that.