Hide Forgot
Description of problem: For development purposes it would be very cool to have a way to set up a HA Cluster in Fedora in 10 minutes or less. Something like: * Install bare metal OS with KVM and hardware virt capabilities * yum install something * run a script that creates multiple guest images and configures them to run in libvirt with the cluster software preinstalled and cluster.conf preconfigured to 'just work' * This base config should assume a NAT network environment for simplicity * Should set up fence_virt/fence_virtd properly for guest fencing * It should also use a file backed shared block device between the guests and preconfigure clvmd and a GFS2 filesystem. * Maybe a 2nd block device as well to show HA-LVM * Maybe we set up a host based iSCSI target as well and expose that to the guests to provide a different storage type other than shared block device exposed from kvm This would allow cluster developers to rapidly set up a system and do development of things like resource agents, or work on pacemaker/rgmanager, etc.
The idea is very nice, but there are several issues in this approach. First of all a feature like this one can´t really be wrapped in a BZ for implementation/maintainance. This is a constantly on-going task as the script needs to be regularly tested across a matrix of Fedora releases and adapt as the distribution changes. Let´s use the BZ to sketch the specifications, capture them in a doc and close it there, otherwise it will hunt us forever :) Then I have a few questions: > * Install bare metal OS with KVM and hardware virt capabilities let´s limit the test/implementation matrix. Fedora supports 2 stable releases at any given time + rawhide that is constantly moving development tree. My suggestion would be that we only support the very latest stable release on the bare metal os, and either the very latest stable and rawhide in the guests. > yum install something Because fedora moves fast, the script might require quick constant updates, so we will need to do an evaluation phase íf it is worth shipping as part of a package (lots of overhead to push updates) or keep it separate. So my suggestion would be: wget -O something_script http://... bash ./something_script and the script will take care to do all the installation required here or there > * run a script that creates multiple guest images and configures them to run in libvirt with the cluster software preinstalled and cluster.conf preconfigured to 'just work' At this point, things get "complicated" :) and we need to limit the scope a bit. Are we expecting to run this script interactively? or drive it via configuration file? for example: 1) how many nodes should it create? 2) size of each node (ram/cpu/disk) 3) path to install ISO or what kind of install method? 4) size of the shared storage each of those bits involve a decent amount of work to be done "right". So either we require for example an install iso or some unlimited networking (to download some netinstall images and so on). (once the first image is installed, we can proceed via cloning, but for the first time, those issues still remain valid) > * This base config should assume a NAT network environment for simplicity libvirt default NAT rules did show some issues with cluster. Can we simply assume that the cluster is local to the machine and be done with it? aka the cluster cannot be visible from outside the host. that way we can create an ad-hoc bridge that will not conflict with anything else on the box. > * [SNIP] Other requirements are ok to me. I don´t see anything bad. We will need to define a bunch of resources to run by default (vip? http? nfs?), and default setup (v4, v6).
I would suggest keeping it really simple to start with: - KVM based virtual cluster of N nodes where N >= 3 - Using latest stable Fedora only - Set up networking on a private network - No communication outside the cluster (except maybe to physical host) - Whole cluster on one physical host - A single shared disk with a single CLVM volume - Fixed shared disk size of, say 5G - Working fencing set up - Autogenerated cluster.conf - No services running beyond Fedora defaults - Anything else I've missed set to a sensible default I'm imagining an interface along the lines of: build-cluster [-n <nodes>] name which will build a set of <nodes> nodes each of which is called name-nn where nn is its node number. If the number of nodes is not specified, it should probably default to 4 nodes. Once we have the basic script working, then we can worry about more advanced config issues, but to start with, lets keep it as simple as possible.
Right, I was assuming pretty much what Steve said... Except: - Explicitly no quorum disk - Use fence-virt for fencing - N >= 2 (default of 2) - Skip clvm ...? - Allow multiple shared disks - $NAME-nodeX (where $NAME is the cluster name; X is the node ID) It shouldn't be difficult, and I very much think this is a tool that should never, ever be used to create a production cluster - development purposes only.
(In reply to comment #1) > My suggestion would be that we only support the very latest stable release on > the bare metal os, and either the very latest stable and rawhide in the guests. Yep. > 3) path to install ISO or what kind of install method? Web install of current Fedora stable; allow people to change the URL if they have a local mirror. > > * This base config should assume a NAT network environment for simplicity > > libvirt default NAT rules did show some issues with cluster. Can we simply > assume that the cluster is local to the machine and be done with it? aka the > cluster cannot be visible from outside the host. I never had any trouble. I would prefer keeping the cluster only visible from the host and other cluster nodes. > We will need to define a bunch of resources to run by default (vip? http? > nfs?), and default setup (v4, v6). Maybe a pacemaker version of a simple VIP service.
> > 3) path to install ISO or what kind of install method? > > Web install of current Fedora stable; allow people to change the URL if they > have a local mirror. Path to a local ISO is a good idea as well. But other than URL or local ISO that should be it.
(In reply to comment #4) > (In reply to comment #1) > > > My suggestion would be that we only support the very latest stable release on > > the bare metal os, and either the very latest stable and rawhide in the guests. > > Yep. > > > 3) path to install ISO or what kind of install method? > > Web install of current Fedora stable; allow people to change the URL if they > have a local mirror. No need to, with current mirroring schema from Fedora, it's enough we use default URLS and local mirrors are automatically used. Or we simply clone them from the host "as preferred". > > > > * This base config should assume a NAT network environment for simplicity > > > > libvirt default NAT rules did show some issues with cluster. Can we simply > > assume that the cluster is local to the machine and be done with it? aka the > > cluster cannot be visible from outside the host. > > I never had any trouble. I would prefer keeping the cluster only visible from > the host and other cluster nodes. Ok perfect, then I'll just recheck in my setup to make sure it is working. Probably it's just my weird setup :) > > > We will need to define a bunch of resources to run by default (vip? http? > > nfs?), and default setup (v4, v6). > > Maybe a pacemaker version of a simple VIP service. Yes we can make it: cbox [-n num_of_nodes] [-t pacemaker|cman] clustername and then later on add some virt config options (disk size/ram/vcpus/ethernets). I am sure more will come to mind as we work on it. Let's start with super basic and extend as we go via meaning of hooks. I have some skeleton we can use for this already. Tomorrow I'll get kitchensink/ going where we can dump all our snippets and then we start gluing them together as needed.
I started a git tree here, for now to collect different snippets around. https://github.com/fabbione/cbox
Just an update: it is now possible to run cbox, define clustername and number of nodes, and it will do: set the host create the guests create shared storage setup a plain gfs2 partition (10G) on the shared storage setup a cluster vg (10G) on the shared storage create a 5G lv on the clustered vg and make it gfs2 create cluster.conf setup fencing (using fence_virt on the host and fence_xvm emulation in the guest) start the cluster. mount the fs´es. It is still very basic and mostly hardcoded, but it is already usable. *WARNING* handle with care *WARNING*
framework is out there for public consumption.