Hide Forgot
Description of problem: The improved support for container in upstream pacemaker is needed for OSP12 in order to manage galera and rabbit as a containerized service.
Pacemaker is adding a new syntax to better support Docker containers. There will be a new resource type (in addition to primitive, group, etc.) to define a container grouping, consisting of Docker settings, network settings, storage settings, and the service being containerized. The exact syntax is not yet finalized, but the components of a container grouping will be: * as with any resource type, a resource name and optional resource description (for the container grouping as a whole) * Docker settings: ** "image" (a Docker image identifier such as "beekhof:http") ** optional "replicas" (the desired number of container instances of this image, defaulting to 1) ** optional "replicas-per-host" (default 1) ** optional "masters" (integer) ** optional "pcmk-remote-bin" (file path to pacemaker_remote inside the container, defaulting to usual location) ** optional "options" (arbitrary Docker command-line options, e.g. "--log-driver=journald") * Optional network settings: ** "ip-range-start" (the first IPv4 address available to assign to containers; by default, no IPs will be assigned) ** "host-network" ** "host-netmask" ** "docker-network" ** zero or more "port-mapping" entries: each has an ID plus either an integer "port" or a "range" (starting port - ending port e.g. 1100-1199) * Optional storage settings: ** zero or more "storage-mapping" entries: each has an ID plus: *** either "source-dir" or "source-dir-root" (directory path on the host) *** "target-dir" (directory path inside the container to use as mount point) *** "options" (arbitrary mount options, e.g. "rw") * A single primitive resource for the containerized service We are keeping the option open for the (distant) future to support container technologies other than Docker. The idea is that the Docker settings part would be swapped out with something else (Rocket, OCI, LXC, etc.) but the network, storage, and primitive would hopefully stay the same. The name of the container grouping type is not yet decided -- possibly container-grouping, bundle, or launcher. The names of the individual settings will likely mostly stay as above, but a few may change. Not all of the settings may be implemented in time for 7.4. We will need pcs commands to add, remove, and edit container groupings.
QA: A full test procedure will be documented here once the new syntax is finalized, but the basic process will be: * install docker on all nodes * configure a docker image with apache and pacemaker remote on all nodes * configure pacemaker using new syntax here * test a variety of actions such as failing a container, migrating a container, disabling a container, putting a node running a container into standby, etc.
upstream tutorial for pacemaker: http://wiki.clusterlabs.org/wiki/Bundle_Walk-Through
Note that implementation for "docker" engine of a bundle has a hardcoded reference to ocf:heartbeat:docker resource agent, so it might make sense to check it's available prior to inner processing of pcs bundle create CONTAINER container docker ...
(In reply to Jan Pokorný from comment #9) > Note that implementation for "docker" engine of a bundle has a hardcoded > reference to ocf:heartbeat:docker resource agent, so it might make sense > to check it's available prior to inner processing of > > pcs bundle create CONTAINER container docker ... Should this be checked on all nodes or just the node where the command is run? When adding a node to a cluster, should pcs check docker agent is installed on the node? Should pcs check if docker is installed? If it runs? What if the user wants to edit the CIB outside a cluster node? Should pcs check on cib push? These checks are not done for any resource / stonith agents and it works - pacemaker reports an error when a resource cannot run and the user reacts to that. For now we are not going to implement these checks. If this becomes a problem, we will deal with it.
Created attachment 1276372 [details] proposed fix + tests summary of changes: * new commands: * bundle create * create a new bundle with no encapsulated resources * bundle update * change an existing bundle * updated commands - working with bundles: * resource create * new keyword "bundle" allows to create a resource inside a specified bundle * resource delete * allows deleting bundles and resources encapsulated in bundles * config show, resource show * displays bundles * resource clear, resource ban * resource restart * bz1447910 * resource cleanup * bz1447916 * resource failcount * works with inner bundle resources * bundles may be referenced in constraints * updated commands - not working with bundles: * these exit with an error when requested to operate on bundles * resource enable, resource disable * resource manage, resource unmanage * resource op add, resource op remove * resource meta * resource group add, resource group remove, resource ungroup * also moving a resource from a bundle to a group is not allowed * resource clone, resource master, resource unclone * also cloning a resource in a bundle is not allowed * resource utilization * resource debug-start, resource debug-stop, resource debug-promote, resource debug-demote, resource debug-monitor * bz1447918 * resource move * does not work for master and clone resource either * web ui * works when bundles are present in a cluster * no support for bundles in web ui
Re [comment 10]: Main problem is that docker agent is not present prior to resource-agents 3.9.6. So the suggested check would prevent issues new pcs vs. old resource-agents (ditto IPaddr2, but that was added substantially earlier). Btw. I think it's dearly unfortunate to allow the container type not being a required input while at the same type, the container options themselves are relative to this particular and now possibly omitted type. So anytime the preference for the implicit/defautl container type changes, there's an imminent breakage in the scripts relying on default being stable in time. And who can predict the future and whether the current default will be as relevant at some distant point in time? (For that reason, I will always emit the container type in clufter for the purpose of the reverse extraction of CIB into respective reinstating pcs commands.)
Created attachment 1285326 [details] proposed fix enable/disable + tests
After fix - enable and disable works: [root@rh73-node1:~]# rpm -q pcs pcs-0.9.158-4.el7.x86_64 [root@rh73-node1:~]# pcs resource Docker container set: http-bundle [pcmktest:http] http-bundle-0 (192.168.122.250) (ocf::pacemaker:Stateful): Started rh73-node2 http-bundle-1 (192.168.122.251) (ocf::pacemaker:Stateful): Started rh73-node1 [root@rh73-node1:~]# pcs resource disable http-bundle --wait resource 'http-bundle' is not running on any node [root@rh73-node1:~]# pcs resource Docker container set: http-bundle [pcmktest:http] http-bundle-0 (192.168.122.250) (ocf::pacemaker:Stateful): Stopped (disabled) http-bundle-1 (192.168.122.251) (ocf::pacemaker:Stateful): Stopped (disabled) [root@rh73-node1:~]# pcs resource enable http-bundle --wait resource 'http-bundle' is running on nodes 'http-bundle-0', 'http-bundle-1', 'rh73-node1', 'rh73-node2' [root@rh73-node1:~]# pcs resource Docker container set: http-bundle [pcmktest:http] http-bundle-0 (192.168.122.250) (ocf::pacemaker:Stateful): Started rh73-node2 http-bundle-1 (192.168.122.251) (ocf::pacemaker:Stateful): Started rh73-node1
Created attachment 1287024 [details] proposed fix container type + tests container type is now mandatory (see comment 13)
After fix: [root@rh73-node1:~]# pcs resource bundle create http-bundle container image=pcmktest:http Error: '' is not a valid container type value, use docker [root@rh73-node1:~]# echo $? 1 [root@rh73-node1:~]# pcs resource bundle create http-bundle container docker image=pcmktest:http [root@rh73-node1:~]# echo $? 0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1958