Description of problem: Cumin-web will not meet medium scale requirements as outlined in the PRD, evidenced by scripts run with a web-testing framework. Scale can be met at present by running multiple cumin instances with 5 users per instance. Changes are required in /etc/init.d/cumin, /usr/bin/cumin, /usr/bin/cumin-web, and /usr/bin/cumin-data to allow users to easily deploy multiple cumin-web instances. Additionally, a simple /etc/sysconfig/cumin file needs to be added to control startup up multiple instances. Documentation changes are also needed. Things that can be tested: 1) cumin-web and cumin-data each take a "--section" argument which specify the section in /etc/cumin/cumin.conf from which they take their configuration. Default behavior for each can be achieved as follows: a) /usr/bin/cumin-web b) /usr/bin/cumin-web --section=web c) /usr/bin/cumin-data d /usr/bin/cumin-data --section=data 2) /usr/bin/cumin-master starts instances of cumin-web and cumin-data. It passes a "--section" argument (described above) to each instance. cumin-master itself takes "--webs" and "--datas" options. The value for each option is a comma separated list of section names in /etc/cumin/cumin.conf. Each section name implies another instance of cumin-web or cumin-data that will use the named section for its configuration. Default behavior, a single cumin-web and cumin-data configured from the [web] and [data] sections, can be achieved with the following invocations: a) /usr/bin/cumin-master b) /usr/bin/cumin-master --webs=web --datas=data c) /usr/bin/cumin-master --webs=web d) /usr/bin/cumin-master --datas=data 3) Options for cumin-master are specified in /etc/sysconfig/cumin in the CUMIN_MULTI_OPTIONS variable. This file is read by /etc/init.d/cumin and the value of CUMIN_MULTI_OPTIONS is passed to cumin-master. Default behavior should be in effect if: a) /etc/sysconfig/cumin is absent b) /etc/sysconfig/cumin is empty or leaves CUMIN_MULTI_OPTIONS unset c) /etc/sysconfig/cumin sets CUMIN_MULTI_OPTIONS="--webs=web --data=data" d) /etc/sysconfig/cumin sets CUMIN_MULTI_OPTIONS="--webs=web" e) /etc/sysconfig/cumin sets CUMIN_MULTI_OPTIONS="--data=data" 4) If cumin-web or cumin-data is given a "--section" option which names a section that does not exist in /etc/cumin/cumin.conf, it should exit with an exception. 5) If cumin-master is given bad options, initd should indicate that the service failed to start. For example, in /etc/sysconfig/cumin set CUMIN_MULTI_OPTIONS as follows: a) CUMIN_MULTI_OPTIONS="--llamas=web" # bad, no such option b) CUMIN_MULTI_OPTIONS="--webs=web web0" # bad, not comma separated 6) The typical use case for this is to set up multiple cumin-web instances on different ports. This can be done by: a) add new sections to /etc/cumin/cumin.conf, for example [web1] port: 45674 [web2] port: 45675 b) set /etc/sysconfig/cumin to use those sections: CUMIN_MULTI_OPTIONS="--webs=web,web1,web2" c) start the service and look for multiple cumin-webs with ps, attach to each port from a browser (in this case 45672, 45674, and 45675) 7) Stopping cumin-master should stop multiple instances 8) As with the current /usr/cumin/bin, cumin-master should start any children that it notices have stopped operating. This is the same with multiple instances. If multiple instances of cumin-web are started, and a single instance is killed off, that instance should be restarted with the same options within 30 seconds. 9) If a cumin-web or cumin-data instance fails to start from cumin-master because it references a section that does not exist, cumin-master will continue to try to start it every 30 seconds. Editing /etc/cumin/cumin.conf and adding the missing section while leaving cumin-master running will allow that instance to start correctly at the next 30 second boundary.
Additionally, internal code changes around configuration affect the following programs: /usr/bin/cumin-admin /usr/bin/cumin-bench /usr/bin/cumin-smoketest These changes are transparent to the end user -- the programs continue to reference the [web] and [data] sections in cumin.conf, but they do it a little differently.
Point 5) from the bug's Description is not met in cumin-0.1.4475-1.el5: ------------------------------------------------------------------------ # echo CUMIN_OPTIONS="-nonsense" > /etc/sysconfig/cumin # service cumin restart Stopping cumin: [ OK ] Starting cumin: [ OK ] # usage: cumin [options] cumin: error: no such option: -n # echo CUMIN_OPTIONS="--webs=nonsense" > /etc/sysconfig/cumin # service cumin restart Stopping cumin: [FAILED] Starting cumin: [ OK ] # Traceback (most recent call last): File "/usr/bin/cumin-web", line 51, in ? main() File "/usr/bin/cumin-web", line 14, in main config, values, opts, args = get_configuration(CuminWebOptionParser(), CuminWebConfig) File "/usr/share/cumin/python/cumin/config.py", line 179, in get_configuration raise Exception("Section ["+opts.section+"] not present in configuration files.") Exception: Section [nonsense] not present in configuration files. # echo CUMIN_OPTIONS="--webs=web,web1,web2,web3" > /etc/sysconfig/cumin # service cumin restart Stopping cumin: [ OK ] Starting cumin: [ OK ] ------------------------------------------------------------------------ Otherwise it seems to be working, I am trying the loadbalancing with NGINX, using following configuration (package nginx-0.8.53-1.el5 from EPEL, not required) which I am pasting here for reference: ------------------------------------------------------------------------ http { upstream cumin { ip_hash; server 127.0.0.1:45672; server 127.0.0.1:45674; server 127.0.0.1:45675; server 127.0.0.1:45676; } server { listen 80; server_name servername.domain; location / { proxy_pass http://cumin; } } } ------------------------------------------------------------------------
On point 5: Note, previous packages have the same problem, I discovered. I thought this was a regression, but it's not. If /usr/bin/cumin fails for some reason in 1.3, /sbin/service will still say "OK". I think it's related to the fact that /usr/bin/cumin is spawned from init.d, and it is the return result from the spawn that is checked. At this point, I don't think it's worth trying to fix. On the load balancing: nice, looks simple. I wonder if we want to include this in the docs, as an unsupported solution? Do we do such things?
> On the load balancing: nice, looks simple. I wonder if we want to include > this in the docs, as an unsupported solution? Do we do such things? Note, Matt suggested this kind of content would be well-served by a knowledge base article.
OK, verified, spawning a new bug out of comment #5. Verified also on cumin-0.1.4478-1.el5
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause At present, a single instance of cumin-web cannot meet performance goals for medium scale deployments under peak usage. Change Multiple instances of cumin-web may be run on different ports to partition the work. To make this accesible for end users, support was added for multiple sections in the cumin configuration file and the service startup script was modified to allow creation of multiple instances. The new configuration options are covered in the MRG Management Console Installation Guide. Result MRG Management Console installations may easily be configured to run multiple instances of cumin-web on different ports of the same host. Users may select an instance manually by specifying a particular port number in the URL, or alternatively users may choose to set up a load balancing proxy server for cumin-web at a single URL.
Verified with cumin-0.1.4494-1.el5 as well
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,8 +1 @@ -Cause +Running multiple instances of cumin-web on different ports of the same host in order to distribute the workload and increase scalability is now supported. In order to achieve this goal, the cumin configuration file was altered to allow for multiple sections, and the service startup script was modified to allow the creation of multiple instances. Instances can be manually selected by specifying a particular port number in the URL; alternatively, a load-balancing proxy server for cumin-web can be used. These new configuration options are covered in the MRG Management Console Installation Guide.- At present, a single instance of cumin-web cannot meet performance goals for medium scale deployments under peak usage. - -Change - Multiple instances of cumin-web may be run on different ports to partition the work. To make this accesible for end users, support was added for multiple sections in the cumin configuration file and the service startup script was modified to allow creation of multiple instances. The new configuration options are covered in the MRG Management Console Installation Guide. - -Result - MRG Management Console installations may easily be configured to run multiple instances of cumin-web on different ports of the same host. Users may select an instance manually by specifying a particular port number in the URL, or alternatively users may choose to set up a load balancing proxy server for cumin-web at a single URL.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-0217.html