Bug 1012143

Summary: Conductor should be able to start multiple workers
Product: Red Hat OpenStack Reporter: Dan Smith <dasmith>
Component: openstack-novaAssignee: Xavier Queralt <xqueralt>
Status: CLOSED ERRATA QA Contact: Gabriel Szasz <gszasz>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.0CC: ajeain, apevec, breeler, dallan, hateya, ndipanov, sputhenp, xqueralt, yeylon
Target Milestone: z3Keywords: ZStream
Target Release: 3.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-2013.1.4-1.el6ost Doc Type: Enhancement
Doc Text:
A new configuration option, called "workers", has been added to the conductor section in nova.conf, which specifies the number of workers the conductor will start. This was added to enhance performance on big clusters, where having just one conductor may cause performance issues due to only one service managing all the requests to the database.
Story Points: ---
Clone Of:
: 1012148 (view as bug list) Environment:
Last Closed: 2013-11-18 15:16:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dan Smith 2013-09-25 20:04:36 UTC
Description of problem:

Any reasonable deployment needs more than one conductor process and grizzly did not support this natively. Right now, this is only workable if someone runs a conductor on several machines or starts multiple processes manually. We should have an /etc/sysconfig/nova-conductor config point to specify how many daemons to start so that it's easy to start multiple daemons on a single system.

Version-Release number of selected component (if applicable):

openstack-nova-conductor-2013.1.3-3.el6ost

How reproducible:

Very.

Steps to Reproduce:
1. Deploy conductor on one machine
2. Deploy compute on many machines
3. Notice failures and poor performance

Actual results:

Everything funnels through a single conductor.

Expected results:

Multiple conductors should be started to spread out the load.

Additional info:

This should be pretty easy to do in the script, starting multiple workers, with log files and pid files like /var/log/nova/conductor-$i.log, etc.

Comment 7 Gabriel Szasz 2013-11-04 16:03:28 UTC
VERIFICATION:

1. Deploy conductor node and at least one compute node
2. Alter workers param of the [conductor] section of /etc/nova/nova.conf to
   workers=2
3. Restart the openstack-nova-conductor service
4. Check out the /var/log/nova/conductor.log

2013-11-04 10:57:49.387 28450 INFO nova.service [-] Starting 2 workers
2013-11-04 10:57:49.388 28450 INFO nova.service [-] Started child 28462
2013-11-04 10:57:49.390 28450 INFO nova.service [-] Started child 28463
2013-11-04 10:57:49.393 28462 AUDIT nova.service [-] Starting conductor node (version 2013.1.4-1.el6ost)
2013-11-04 10:57:49.394 28463 AUDIT nova.service [-] Starting conductor node (version 2013.1.4-1.el6ost)
2013-11-04 10:57:49.658 28463 INFO nova.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.16.96.136:5672
2013-11-04 10:57:49.658 28462 INFO nova.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.16.96.136:5672
2013-11-04 10:57:50.024 INFO nova.openstack.common.rpc.impl_qpid [req-54827805-a4f8-44a5-87a6-d646d68ffe58 None None] Connected to AMQP server on 10.16.96.136:5672
2013-11-04 10:57:52.975 INFO nova.openstack.common.rpc.impl_qpid [req-7bead73e-970d-4b41-b594-a61e3f23540c None None] Connected to AMQP server on 10.16.96.136:5672

5. The conductor node is indeed running 2 workers as set in nova.conf

Comment 8 errata-xmlrpc 2013-11-18 15:16:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1510.html

Comment 9 Dave Allan 2013-11-25 22:18:56 UTC
*** Bug 1012012 has been marked as a duplicate of this bug. ***

Comment 10 Dan Smith 2013-12-05 18:57:58 UTC
*** Bug 1025352 has been marked as a duplicate of this bug. ***