Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 916241 - nova-dhcpbridge fails to startup
nova-dhcpbridge fails to startup
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
2.1
Unspecified Unspecified
urgent Severity urgent
: snapshot4
: 2.1
Assigned To: Nikola Dipanov
Martina Kollarova
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-02-27 10:50 EST by Dan Prince
Modified: 2016-04-26 12:13 EDT (History)
9 users (show)

See Also:
Fixed In Version: openstack-nova-2012.2.3-3.el6ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-21 14:17:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0657 normal SHIPPED_LIVE Moderate: openstack-nova security, bug fix, and enhancement update 2013-03-21 18:12:14 EDT

  None (edit)
Description Dan Prince 2013-02-27 10:50:16 EST
Description of problem:

As of this morning I'm seeing failure's when Nova network tries to run nova-dhcpbridge:

2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Command: sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Exit code: 11
2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Stdout: ''
2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp Stderr: '2013-02-27 10:27:25 CRITICAL nova [req-485ea824-780f-4a95-b960-52c6c8ff41f8 None None] (OperationalError) (1045, "Access denied for user \'nova\'@\'localhost\' (using password: YES)") None None\n\ndnsmasq: lease-init script returned exit code 1\n'
2013-02-27 10:27:25 8033 TRACE nova.openstack.common.rpc.amqp
2013-02-27 10:27:25 8033 ERROR nova.openstack.common.rpc.common [-] Returning exception Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
Exit code: 11
Stdout: ''
Stderr: '2013-02-27 10:27:25 CRITICAL nova [req-485ea824-780f-4a95-b960-52c6c8ff41f8 None None] (OperationalError) (1045, "Access denied for user \'nova\'@\'localhost\' (using password: YES)") None None\n\ndnsmasq: lease-init script returned exit code 1\n' to caller


----

Running the same command manually yeilds this:


-bash-4.1# sudo nova-rootwrap /etc/nova/rootwrap.conf FLAGFILE=["/usr/share/nova/nova-dist.conf"] NETWORK_ID=1 dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.0.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
Traceback (most recent call last):
  File "/usr/bin/nova-dhcpbridge", line 130, in <module>
    main()
  File "/usr/bin/nova-dhcpbridge", line 101, in main
    default_config_files=jsonutils.loads(flagfile))
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 134, in loads
    return json.loads(s)
  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.6/json/decoder.py", line 336, in raw_decode
    obj, end = self._scanner.iterscan(s, **kw).next()
  File "/usr/lib64/python2.6/json/scanner.py", line 55, in iterscan
    rval, next_pos = action(m, context)
  File "/usr/lib64/python2.6/json/decoder.py", line 219, in JSONArray
    raise ValueError(errmsg("Expecting object", s, end))
ValueError: Expecting object: line 1 column 1 (char 1)

----

The new nova.conf format (used by PackStack now?) seems to be to blame? 

Version-Release number of selected component (if applicable):

-bash-4.1# rpm -qa | grep openstack
openstack-keystone-2012.2.3-4.el6ost.noarch
openstack-cinder-2012.2.3-4.el6ost.noarch
openstack-nova-api-2012.2.3-2.el6ost.noarch
openstack-nova-novncproxy-0.4-2.el6.noarch
python-django-openstack-auth-1.0.6-2.el6ost.noarch
openstack-dashboard-2012.2.3-5.el6ost.noarch
openstack-swift-1.7.4-7.el6ost.noarch
openstack-swift-object-1.7.4-7.el6ost.noarch
openstack-glance-2012.2.3-2.el6ost.noarch
openstack-nova-common-2012.2.3-2.el6ost.noarch
openstack-selinux-0.1.2-5.el6ost.noarch
openstack-nova-cert-2012.2.3-2.el6ost.noarch
openstack-packstack-2012.2.3-0.1.dev454.el6ost.noarch
openstack-nova-compute-2012.2.3-2.el6ost.noarch
openstack-nova-network-2012.2.3-2.el6ost.noarch
openstack-nova-volume-2012.2.3-2.el6ost.noarch
openstack-nova-2012.2.3-2.el6ost.noarch
openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
openstack-swift-account-1.7.4-7.el6ost.noarch
openstack-swift-proxy-1.7.4-7.el6ost.noarch
openstack-nova-scheduler-2012.2.3-2.el6ost.noarch
openstack-nova-console-2012.2.3-2.el6ost.noarch
openstack-nova-objectstore-2012.2.3-2.el6ost.noarch
openstack-swift-container-1.7.4-7.el6ost.noarch
openstack-utils-2012.2-6.1.el6ost.noarch


Steps to Reproduce:
1. Run packstack installation.
2. Try to boot an instance via Nova.
Comment 1 Russell Bryant 2013-02-27 11:39:34 EST
The root cause here appears to just be that nova-dist.conf needs to be updated to include:

    dhcpbridge_flagfile=['/usr/share/nova/nova-dist.conf', '/etc/nova/nova.conf']

so that nova-dhcpbridge ends up reading both config files.

The error when running it manually is just a shell quoting issue AFAICT.  The original error from nova was failing to connect to the database, which is due to nova-dhcpbridge not reading /etc/nova/nova.conf.
Comment 4 Russell Bryant 2013-02-27 14:14:59 EST
Note that my previous comment about what should go in nova-dist.conf was wrong.  It's

     dhcpbridge_flagfile=/usr/share/nova/nova-dist.conf
     dhcpbridge_flagfile=/etc/nova/nova.conf
Comment 8 errata-xmlrpc 2013-03-21 14:17:12 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0657.html

Note You need to log in before you can comment on or make changes to this bug.