Description of problem: As of this morning I'm seeing failure's when Nova network tries to run nova-dhcpbridge: 2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command. 2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro 2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Exit code: 11 2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Stdout: '' 2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Stderr: '2013-02-27 10:27:25 CRITICAL nova [req-485ea824-780f-4a95-b960-52c6c8ff41f8 None None] (OperationalError) (1045, "Access denied for user \'nova\'@\'localhost\' (using password: YES)") None None\n\ndnsmasq: lease-init script returned exit code 1\n' 2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp 2013-02-27 10:27:25 8033 ERROR nova.openstack.common.rpc.common [-] Returning exception Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro Exit code: 11 Stdout: '' Stderr: '2013-02-27 10:27:25 CRITICAL nova [req-485ea824-780f-4a95-b960-52c6c8ff41f8 None None] (OperationalError) (1045, "Access denied for user \'nova\'@\'localhost\' (using password: YES)") None None\n\ndnsmasq: lease-init script returned exit code 1\n' to caller ---- Running the same command manually yeilds this: -bash-4.1# sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro Traceback (most recent call last): File "/usr/bin/nova-dhcpbridge", line 130, in <module> main() File "/usr/bin/nova-dhcpbridge", line 101, in main default_config_files=jsonutils.loads(flagfile)) File "/usr/lib/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 134, in loads return json.loads(s) File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode obj, end = self._scanner.iterscan(s, **kw).next() File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan rval, next_pos = action(m, context) File "/usr/lib64/python2.6/json/decoder.py", line 219, in JSONArray raise ValueError(errmsg("Expecting object", s, end)) ValueError: Expecting object: line 1 column 1 (char 1) ---- The new nova.conf format (used by PackStack now?) seems to be to blame? Version-Release number of selected component (if applicable): -bash-4.1# rpm -qa | grep openstack openstack-keystone-2012.2.3-4.el6ost.noarch openstack-cinder-2012.2.3-4.el6ost.noarch openstack-nova-api-2012.2.3-2.el6ost.noarch openstack-nova-novncproxy-0.4-2.el6.noarch python-django-openstack-auth-1.0.6-2.el6ost.noarch openstack-dashboard-2012.2.3-5.el6ost.noarch openstack-swift-1.7.4-7.el6ost.noarch openstack-swift-object-1.7.4-7.el6ost.noarch openstack-glance-2012.2.3-2.el6ost.noarch openstack-nova-common-2012.2.3-2.el6ost.noarch openstack-selinux-0.1.2-5.el6ost.noarch openstack-nova-cert-2012.2.3-2.el6ost.noarch openstack-packstack-2012.2.3-0.1.dev454.el6ost.noarch openstack-nova-compute-2012.2.3-2.el6ost.noarch openstack-nova-network-2012.2.3-2.el6ost.noarch openstack-nova-volume-2012.2.3-2.el6ost.noarch openstack-nova-2012.2.3-2.el6ost.noarch openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch openstack-swift-account-1.7.4-7.el6ost.noarch openstack-swift-proxy-1.7.4-7.el6ost.noarch openstack-nova-scheduler-2012.2.3-2.el6ost.noarch openstack-nova-console-2012.2.3-2.el6ost.noarch openstack-nova-objectstore-2012.2.3-2.el6ost.noarch openstack-swift-container-1.7.4-7.el6ost.noarch openstack-utils-2012.2-6.1.el6ost.noarch Steps to Reproduce: 1. Run packstack installation. 2. Try to boot an instance via Nova.
The root cause here appears to just be that nova-dist.conf needs to be updated to include: dhcpbridge_flagfile=['/usr/share/nova/nova-dist.conf', '/etc/nova/nova.conf'] so that nova-dhcpbridge ends up reading both config files. The error when running it manually is just a shell quoting issue AFAICT. The original error from nova was failing to connect to the database, which is due to nova-dhcpbridge not reading /etc/nova/nova.conf.
Note that my previous comment about what should go in nova-dist.conf was wrong. It's dhcpbridge_flagfile=/usr/share/nova/nova-dist.conf dhcpbridge_flagfile=/etc/nova/nova.conf
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0657.html