Right now, we have no way to do automated tests of cluster-level functionality - e.g. probing, node failure detection, quorum. True cluster testing, including virtual machines or containers as well as multiple physical machines, would be both cumbersome and resource intensive. Instead, we can tweak glusterd so that multiple instances can run on a single machine, and a script can reach out to any of them (using the --remote-host option) to simulate various actions. This approach is already implemented in the following patch. http://review.gluster.org/#change,4556
CHANGE: http://review.gluster.org/4556 (glusterd: allow multiple instances of glusterd on one machine) merged in master by Anand Avati (avati)
CHANGE: http://review.gluster.org/4566 (tests/cluster.rc: support for virtual multi-server glusterd tests) merged in master by Vijay Bellur (vbellur)
REVIEW: http://review.gluster.org/4836 (tests/cluster.rc: support for virtual multi-server glusterd tests) posted (#1) for review on release-3.4 by Jeff Darcy (jdarcy)
COMMIT: http://review.gluster.org/4836 committed in release-3.4 by Anand Avati (avati) ------ commit 695d173b13f583de846720b66fc201bd84969330 Author: Jeff Darcy <jdarcy> Date: Mon Apr 15 10:32:56 2013 -0400 tests/cluster.rc: support for virtual multi-server glusterd tests Since http://review.gluster.org/4556 glusterd is capable of running many instances of itself on a single system. This patch exploits that feature and enhances the regression test framework to expose handy primitives so that test cases may be written to test glusterd in a cluster. Usage: 1. Include "$(dirname)/../cluster.rc" to get access to the extensions 2. Call launch_cluster $N where $N is the count of virtual servers Calling launch_cluster, starts $N glusterds which bind to $N different IPs and dynamically defines these primitives: - Variables $H1 .. $Hn assigned to hostnames of each "server". - Variables $CLI_1 .. $CLI_n assigned as commands to run CLI commands on the corresponding N'th server. - Variables $B1 .. $Bn assigned to the backend directories on each "server". - Function kill_glusterd, which accepts a parameter - index number of glusterd to be killed. - Variables $glusterd_1 .. $glusterd_n assigned to the command lines to restart the corresponding glusterd, if it was previously killed. The current set of primitives and functions were implemented with the goal of satisfying ./tests/bugs/bug-913555.t. The API will be made richer as we add more cluster test cases Change-Id: I6e79c58098ed0862cf75a0b56e4ce384ec2e4eb2 BUG: 913555 Original-author: Anand Avati <avati> Signed-off-by: Jeff Darcy <jdarcy> Reviewed-on: http://review.gluster.org/4836 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Krishnan Parthasarathi <kparthas>
REVIEW: http://review.gluster.org/4838 (glusterd: allow multiple instances of glusterd on one machine) posted (#1) for review on release-3.4 by Krishnan Parthasarathi (kparthas)
COMMIT: http://review.gluster.org/4838 committed in release-3.4 by Vijay Bellur (vbellur) ------ commit 29d7563416c0d94cf36d7e05493332aacebfa0e0 Author: Krishnan Parthasarathi <kparthas> Date: Tue Apr 16 10:36:31 2013 +0530 glusterd: allow multiple instances of glusterd on one machine This is needed to support automated testing of cluster-communication features such as probing and quorum. In order to use this, you need to do the following preparatory steps. * Copy /var/lib/glusterd to another directory for each virtual host * Ensure that each virtual host has a different UUID in its glusterd.info Now you can start each copy of glusterd with the following xlator-options. * management.transport.socket.bind-address=$ip_address * management.working-directory=$unique_working_directory You can use 127.x.y.z addresses for binding without needing to assign them to interfaces explicitly. Note that you must use addresses, not names, because of some stuff in the socket code that's not worth fixing just for this usage, but after that you can use names in /etc/hosts instead. At this point you can issue CLI commands to a specific glusterd using the --remote-host option. So far probe, volume create/start/stop, mount, and basic I/O all seem to work as expected with multiple instances. Change-Id: I1beabb44cff8763d2774bc208b2ffcda27c1a550 BUG: 913555 Original-author: Jeff Darcy <jdarcy> Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/4838 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>