Bug 1043964 - openstack-foreman-installer: 'neutron-db-manage upgrade' failed and neutron services remained inactive after deployment of client as neutron-controller.
Summary: openstack-foreman-installer: 'neutron-db-manage upgrade' failed and neutron s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-foreman-installer
Version: 4.0
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: z1
: 4.0
Assignee: Jason Guiditta
QA Contact: Ami Jeain
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-17 14:12 UTC by Omri Hochman
Modified: 2014-02-04 17:18 UTC (History)
9 users (show)

Fixed In Version: openstack-foreman-installer-1.0.2-1.el6ost
Doc Type: Rebase: Bug Fixes and Enhancements
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-23 14:20:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
messages.log (193.31 KB, text/plain)
2013-12-22 17:34 UTC, Omri Hochman
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:0046 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform 4 Bug Fix and Enhancement Advisory 2014-01-23 00:51:59 UTC

Description Omri Hochman 2013-12-17 14:12:30 UTC
openstack-foreman-installer: 'neutron-db-manage upgrade' failed and neutron services remained inactive after deployment of client as neutron-controller. 


Environment (Puddle 2013-12-16.1):
-----------------------------------
openstack-foreman-installer-1.0.0-1.el6ost.noarch
puppet-server-3.2.4-3.el6_5.noarch
puppet-3.2.4-3.el6_5.noarch
packstack-modules-puppet-2013.2.1-0.20.dev936.el6ost.noarch
ruby-1.8.7.352-13.el6.x86_64
rubygems-1.8.16-1.el6.noarch
openstack-neutron-2013.2-16.el6ost.noarch


Steps:
-------
- Attempt to deploy foreman-client as neutron controller (hostGroup)

Results: 
--------
- After deployment finished neutron services remained inactive.
- Error in /var/log/messages indicate there was a problem to perform :' neutron-db-manage upgrade  ;


neutron services ('openstack-status' view):  
------------------------------------------

== neutron services ==                                                                                                                                                                        

neutron-server:                         inactive  (disabled on boot)                                                                                                                          

neutron-dhcp-agent:                     inactive  (disabled on boot)

neutron-l3-agent:                       inactive  (disabled on boot)

neutron-metadata-agent:                 inactive  (disabled on boot)

neutron-lbaas-agent:                    inactive  (disabled on boot)

neutron-openvswitch-agent:              inactive  (disabled on boot)

-------------------------------------------------------------------------


[root@puma01 ~]# chkconfig  | grep neutron
neutron-dhcp-agent      0:off   1:off   2:off   3:off   4:off   5:off   6:off
neutron-l3-agent        0:off   1:off   2:off   3:off   4:off   5:off   6:off
neutron-lbaas-agent     0:off   1:off   2:off   3:off   4:off   5:off   6:off
neutron-metadata-agent  0:off   1:off   2:off   3:off   4:off   5:off   6:off
neutron-openvswitch-agent       0:off   1:off   2:off   3:off   4:off   5:off   6:off
neutron-server  0:off   1:off   2:on    3:on    4:on    5:on    6:off
----------------------------------------------------------------------------


less /var/log/messages : 
------------------------
Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/auth_uri]) Scheduling refresh of Exec[glance-manage db_sync]
Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/auth_uri]) Scheduling refresh of Service[glance-api]
Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_api_config[filter:authtoken/auth_uri]/ensure) created
Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_api_config[filter:authtoken/auth_uri]) Scheduling refresh of Service[neutron-server]
Dec 17 10:46:54 puma01 abrt: detected unhandled Python exception in '/usr/bin/neutron-db-manage'
Dec 17 10:46:54 puma01 abrtd: New client connected
Dec 17 10:46:54 puma01 abrtd: Directory 'pyhook-2013-12-17-10:46:54-16842' creation detected
Dec 17 10:46:54 puma01 abrt-server[16850]: Saved Python crash dump of pid 16842 to /var/spool/abrt/pyhook-2013-12-17-10:46:54-16842
Dec 17 10:46:54 puma01 abrtd: Package 'openstack-neutron' isn't signed with proper key
Dec 17 10:46:54 puma01 abrtd: 'post-create' on '/var/spool/abrt/pyhook-2013-12-17-10:46:54-16842' exited with 1
Dec 17 10:46:54 puma01 abrtd: Deleting problem directory '/var/spool/abrt/pyhook-2013-12-17-10:46:54-16842'
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) No handlers could be found for logger "neutron.common.legac
y"
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) Traceback (most recent call last):
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/bin/neutron-db-manage", line 10, in <module>
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     sys.exit(main())
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/neutron/db/migrati
on/cli.py", line 143, in main
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     CONF.command.func(config, CONF.command.name)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/neutron/db/migrati
on/cli.py", line 80, in do_upgrade_downgrade
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     do_alembic_command(config, cmd, revision, sql=CONF.comm
and.sql)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/neutron/db/migrati
on/cli.py", line 59, in do_alembic_command
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     getattr(alembic_command, cmd)(config, *args, **kwargs)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/alembic/command.py
", line 124, in upgrade
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     script.run_env()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/alembic/script.py", line 191, in run_env
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     util.load_python_file(self.dir, 'env.py')
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/alembic/util.py", line 186, in load_python_file
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     module = imp.load_source(module_id, path, open(path, 'rb'))
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 105, in <module>
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     run_migrations_online()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 80, in run_migrations_online
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     connection = engine.connect()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2472, in connect
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return self._connection_cls(self, **kwargs)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     self.__connection = connection or engine.raw_connection()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2558, in raw_connection
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return self.pool.unique_connection()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 183, in unique_connection
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return _ConnectionFairy(self).checkout()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 387, in __init__
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     rec = self._connection_record = pool._do_get()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 802, in _do_get
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return self._create_connection()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 188, in _create_connection
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return _ConnectionRecord(self)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 270, in __init__
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     self.connection = self.__connect()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     connection = self.__pool._creator()
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return dialect.connect(*cargs, **cparams)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return self.dbapi.connect(*cargs, **cparams)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     return Connection(*args, **kwargs)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns)     super(Connection, self).__init__(*args, **kwargs2)
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) sqlalchemy.exc.OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '10.35.160.11' (111)") None None
Dec 17 10:46:54 puma01 puppet-agent[16303]: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head returned 1 instead of one of [0]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) change from notrun to 0 failed: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head returned 1 instead of one of [0]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_protocol]/ensure) created
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_protocol]) Scheduling refresh of Exec[glance-manage db_sync]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_protocol]) Scheduling refresh of Service[glance-registry]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/admin_user]/ensure) created
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/admin_user]) Scheduling refresh of Exec[glance-manage db_sync]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/admin_user]) Scheduling refresh of Service[glance-api]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/max_retries]/ensure) created
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/max_retries]) Scheduling refresh of Service[neutron-server]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/File[/etc/glance/glance-registry-paste.ini]/ensure) created
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/File[/etc/glance/glance-registry-paste.ini]) Scheduling refresh of Service[glance-registry]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/retry_interval]/ensure) created
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/retry_interval]) Scheduling refresh of Service[neutron-server]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_uri]/ensure) created
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_uri]) Scheduling refresh of Exec[glance-manage db_sync]
Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_uri]) Scheduling refresh of Service[glance-registry]

Comment 1 Omri Hochman 2013-12-17 14:13:26 UTC
After the installation : I was able to start neutron services manually.

Comment 2 Jason Guiditta 2013-12-17 14:51:34 UTC
We think there is an ordering dependency between setting up the database and running this stamp command.  I would be interested to see if the same thing happens with packstack, since we mimic what they did (perhaps there is a chunk we missed?).  Anyway, this used to be an outright error.  If you would, on your next test run, could you try running puppet agaent a second time instead of manually starting the neutron services?  If that results in services running, I think we can call this a known issue, and work on a 0day fix for the ordering.  Also note that this will likely be fixed in the puppet-neutron code, so this is a temporary work around anyway, as I mention in the related BZs, startin ghere:

https://bugzilla.redhat.com/show_bug.cgi?id=1042933


Please let me know the results of your test, and we'll proceed from there.  I am almost ready to run these teste here myself with the latest puddle.

Comment 3 Mike Orazi 2013-12-17 17:18:06 UTC
Jirka Stransky replicated the issues and confirmed that re-running seems to resolve the issues, but I'd like to get confirmation from Omri as well so we have confirmation in more than one environment.

Comment 4 Jason Guiditta 2013-12-17 18:37:44 UTC
OK, I have just posted a new patch that @cwolfe is reviewing:

https://github.com/redhat-openstack/astapor/pull/91

I ran it successfully against the latest on a fresh in stall, and it passed twice in this scenario.

Commit message:

The 'neutron-db-manage upgrade' command was failing on first runs of puppet agent due to a race condition where the database was not necessarily created before this command was run. The order matters, so we have now specified this
more precisely.

Comment 5 Jason Guiditta 2013-12-20 00:28:16 UTC
This is merged, ready for next release.

Comment 6 Omri Hochman 2013-12-20 15:02:05 UTC
The issue constantly reproduced on my environment - last try with openstack-foreman-installer-1.0.0-1.el6ost.noarch (puddle 2013-12-19.1). 

Attempted to run puppet agent the second time as a workaround: (Neutron services remained inactive.) 
-------------------------------------------------------------

[root@puma01 ~]# openstack-status 
== Nova services ==
openstack-nova-api:                     active
openstack-nova-cert:                    active
openstack-nova-compute:                 dead      (disabled on boot)
openstack-nova-network:                 dead      (disabled on boot)
openstack-nova-scheduler:               active
openstack-nova-conductor:               active
== Glance services ==
openstack-glance-api:                   active
openstack-glance-registry:              active
== Keystone service ==
openstack-keystone:                     active
== Horizon service ==
openstack-dashboard:                    500
== neutron services ==
neutron-server:                         active
neutron-dhcp-agent:                     inactive  (disabled on boot)
neutron-l3-agent:                       inactive  (disabled on boot)
neutron-metadata-agent:                 inactive  (disabled on boot)
neutron-lbaas-agent:                    inactive  (disabled on boot)
neutron-openvswitch-agent:              inactive  (disabled on boot)
== Swift services ==
openstack-swift-proxy:                  dead
openstack-swift-account:                dead      (disabled on boot)
openstack-swift-container:              dead      (disabled on boot)
openstack-swift-object:                 dead      (disabled on boot)
== Cinder services ==
openstack-cinder-api:                   active
openstack-cinder-scheduler:             active
openstack-cinder-volume:                inactive  (disabled on boot)
== Ceilometer services ==
openstack-ceilometer-api:               active
openstack-ceilometer-central:           active
openstack-ceilometer-compute:           dead      (disabled on boot)
openstack-ceilometer-collector:         active
== Heat services ==
openstack-heat-api:                     active
openstack-heat-api-cfn:                 inactive  (disabled on boot)
openstack-heat-api-cloudwatch:          inactive  (disabled on boot)
openstack-heat-engine:                  active
== Support services ==
mysqld:                                 active
openvswitch:                            dead
messagebus:                             active
qpidd:                                  active
memcached:                              active
== Keystone users ==
Warning keystonerc not sourced

[root@puma01 ~]# puppet agent -t -v
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ipa_client_configured.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hamysql_active_node.rb
Info: Loading facts in /var/lib/puppet/lib/facter/network.rb
Info: Loading facts in /var/lib/puppet/lib/facter/netns_support.rb
Info: Caching catalog for puma01.scl.lab.tlv.redhat.com
Info: Applying configuration version '1387534464'
Notice: /Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns: executed successfully
Notice: /File[/etc/httpd/conf/httpd.conf]/content: 
--- /etc/httpd/conf/httpd.conf  2013-12-20 16:45:15.152953403 +0200
+++ /tmp/puppet-file20131220-27250-1szxzp3-0    2013-12-20 16:56:08.271567023 +0200
@@ -131,7 +131,7 @@
 # prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
 #
 #Listen 12.34.56.78:80
-Listen 0.0.0.0:80
+Listen 80
 
 #
 # Load config files from the config directory "/etc/httpd/conf.d".

Info: FileBucket got a duplicate file {md5}716b40d1de69187bba4a36821f699591
Info: /File[/etc/httpd/conf/httpd.conf]: Filebucketed /etc/httpd/conf/httpd.conf to puppet with sum 716b40d1de69187bba4a36821f699591
Notice: /File[/etc/httpd/conf/httpd.conf]/content: content changed '{md5}716b40d1de69187bba4a36821f699591' to '{md5}ec999e9a079a8fad9e374a8e01df47f4'
Info: /File[/etc/httpd/conf/httpd.conf]: Scheduling refresh of Service[httpd]
Notice: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]/value: value changed 'f9cd7bb734ef998296b5e89cc9d32baf' to '%ENCRYPTION_KEY%'
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api]
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-engine]
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api-cloudwatch]
Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api-cfn]
Notice: /Stage[main]/Swift::Proxy/Service[swift-proxy]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Swift::Proxy/Service[swift-proxy]: Unscheduling refresh on Service[swift-proxy]
Notice: /Stage[main]/Horizon/File_line[httpd_listen_on_bind_address_80]/ensure: created
Info: /Stage[main]/Horizon/File_line[httpd_listen_on_bind_address_80]: Scheduling refresh of Service[httpd]
Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Heat::Api/Service[heat-api]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Heat::Engine/Exec[heat-encryption-key-replacement]/returns: executed successfully
Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Apache/Service[httpd]: Triggered 'refresh' from 2 events
Notice: Finished catalog run in 46.43 seconds
You have new mail in /var/spool/mail/root


[root@puma01 ~]# openstack-status

== Horizon service ==
openstack-dashboard:                    500
== neutron services ==
neutron-server:                         active
neutron-dhcp-agent:                     inactive  (disabled on boot)
neutron-l3-agent:                       inactive  (disabled on boot)
neutron-metadata-agent:                 inactive  (disabled on boot)
neutron-lbaas-agent:                    inactive  (disabled on boot)
neutron-openvswitch-agent:              inactive  (disabled on boot)

Comment 7 Omri Hochman 2013-12-22 17:27:28 UTC
I just had this problem with openstack-compute (with nova-network) : It seems that this issue is relevant not only to neutron host groups. 

/var/log/messages (attached full file) 
-----------------------------------------
Dec 22 18:48:08 puma02 puppet-agent[16089]: (/Stage[main]/Quickstack::Nova_network::Compute/Nova::Generic_service[metadata-api]/Service[nova-metadata-api]/ensure) ensure changed 'stopped' to
 'running'
Dec 22 18:48:10 puma02 puppet-agent[16089]: Execution of '/usr/bin/nova-manage network create novanetwork 10.0.0.0/24 1 255' returned 1: 2013-12-22 18:48:10.450 16849 INFO nova.network.drive
r [-] Loading network driver 'nova.network.linux_net'
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 CRITICAL nova [req-2e4f944a-dfc2-4f6b-87b7-a2702ecd8b72 None None] (OperationalError) (1130, "Host 'puma02.scl.lab.t
lv.redhat.com' is not allowed to connect to this MySQL server") None None
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova Traceback (most recent call last):
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/bin/nova-manage", line 10, in <module>
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     sys.exit(main())
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1377, in main
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     ret = fn(*fn_args, **fn_kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 530, in create
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     net_manager.create_networks(context.get_admin_context(), **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1073, in create_networks
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return self._do_create_networks(context, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1113, in _do_create_networks
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     nets = self.db.network_get_all(context)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 920, in network_get_all
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return IMPL.network_get_all(context, project_only)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 126, in wrapper
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return f(*args, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2526, in network_get_all
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     project_only=project_only).all()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 190, in model_query
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     session = kwargs.get('session') or get_session()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 423, in get_session
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     engine = get_engine(sqlite_fk=sqlite_fk, slave_engine=slave_session)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 561, in get_engine
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     sqlite_fk=sqlite_fk)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 675, in create_engine
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     engine.connect()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2472, in connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return self._connection_cls(self, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     self.__connection = connection or engine.raw_connection()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2558, in raw_connection
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return self.pool.unique_connection()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 183, in unique_connection
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return _ConnectionFairy(self).checkout()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 387, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     rec = self._connection_record = pool._do_get()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 741, in _do_get
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     con = self._create_connection()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 188, in _create_connection
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return _ConnectionRecord(self)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 270, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     self.connection = self.__connect()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     connection = self.__pool._creator()
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return dialect.connect(*cargs, **cparams)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return self.dbapi.connect(*cargs, **cparams)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     return Connection(*args, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova     super(Connection, self).__init__(*args, **kwargs2)
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova OperationalError: (OperationalError) (1130, "Host 'puma02.scl.lab.tlv.redhat.com' is not allowed to connect to this MySQL server") None None
Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova 
Dec 22 18:48:10 puma02 puppet-agent[16089]: Command failed, please check log for more info
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) change from absent to present failed: Execution of '/usr/bin/nova-manage network create novanetwork 10.0.0.0/24 1 255' returned 1: 2013-12-22 18:48:10.450 16849 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 CRITICAL nova [req-2e4f944a-dfc2-4f6b-87b7-a2702ecd8b72 None None] (OperationalError) (1130, "Host 'puma02.scl.lab.tlv.redhat.com' is not allowed to connect to this MySQL server") None None
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova Traceback (most recent call last):
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/bin/nova-manage", line 10, in <module>
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     sys.exit(main())
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1377, in main
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     ret = fn(*fn_args, **fn_kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 530, in create
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     net_manager.create_networks(context.get_admin_context(), **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1073, in create_networks
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return self._do_create_networks(context, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1113, in _do_create_networks
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     nets = self.db.network_get_all(context)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 920, in network_get_all
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return IMPL.network_get_all(context, project_only)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 126, in wrapper
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return f(*args, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2526, in network_get_all
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     project_only=project_only).all()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 190, in model_query
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     session = kwargs.get('session') or get_session()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 423, in get_session
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     engine = get_engine(sqlite_fk=sqlite_fk, slave_engine=slave_session)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 561, in get_engine
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     sqlite_fk=sqlite_fk)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 675, in create_engine
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     engine.connect()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2472, in connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return self._connection_cls(self, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     self.__connection = connection or engine.raw_connection()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2558, in raw_connection
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return self.pool.unique_connection()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 183, in unique_connection
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return _ConnectionFairy(self).checkout()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 387, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     rec = self._connection_record = pool._do_get()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 741, in _do_get
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     con = self._create_connection()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 188, in _create_connection
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return _ConnectionRecord(self)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 270, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     self.connection = self.__connect()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     connection = self.__pool._creator()
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return dialect.connect(*cargs, **cparams)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return self.dbapi.connect(*cargs, **cparams)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     return Connection(*args, **kwargs)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova     super(Connection, self).__init__(*args, **kwargs2)
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova OperationalError: (OperationalError) (1130, "Host 'puma02.scl.lab.tlv.redhat.com' is not allowed to connect to this MySQL server") None None
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova 
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) Command failed, please check log for more info
Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Ceilometer::Agent::Compute/File_line_after[nova-notification-driver-ceilometer]/ensure) created
Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Compute/Nova::Generic_service[compute]/Service[nova-compute]/ensure) ensure changed 'stopped' to 'running'
Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Generic_service[network]/Service[nova-network]/ensure) ensure changed 'stopped' to 'running'
Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Ceilometer::Agent::Auth/Ceilometer_config[DEFAULT/os_auth_region]/ensure) created
Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Ceilometer::Agent::Compute/Service[ceilometer-agent-compute]/ensure) ensure changed 'stopped' to 'running'
Dec 22 18:48:11 puma02 puppet-agent[16089]: Finished catalog run in 403.88 seconds
Dec 22 18:51:52 puma02 polkitd[16973]: started daemon version 0.96 using authority implementation `local' version `0.96'
Dec 22 18:53:31 puma02 puppet-agent[17044]: Retrieving plugin

Comment 8 Omri Hochman 2013-12-22 17:34:00 UTC
Created attachment 840379 [details]
messages.log

from the'compute with nova-network' machine.

Comment 9 Crag Wolfe 2014-01-02 17:35:18 UTC
Two things:

1. Please try the latest, openstack-foreman-installer-1.0.1-1.el6ost.noarch.rpm as this bug should have been fixed in the last minor bump revision.

2. The paste two comments above was from the compute node-- for this bug I'd like to stay focused on what was originally reported, a specific error on the neutron server.  If the errors reported above on the compute node are still an issue with the latest puddle, perhaps that should be a separate BZ.

Thanks,
--Crag

Comment 11 Crag Wolfe 2014-01-07 02:58:54 UTC
Clarification: the "'neutron-db-manage upgrade' failed" aspect of this bug is what is currently fixed.  However, the extra "neutron-dhcp-agent, neutron-l3-agent" services remain as inactive and turned off.  There is not a quick fix for this-- perhaps could be moved as a separate bug.

Comment 12 Omri Hochman 2014-01-07 08:41:56 UTC
(In reply to Crag Wolfe from comment #11)
> Clarification: the "'neutron-db-manage upgrade' failed" aspect of this bug
> is what is currently fixed.  However, the extra "neutron-dhcp-agent,
> neutron-l3-agent" services remain as inactive and turned off.  There is not
> a quick fix for this-- perhaps could be moved as a separate bug.


As Crag wrote: It seems that 'neutron-db-manage upgrade' aspect is already fixed with openstack-foreman-installer-1.0.1-1.el6ost.noarch. 

I've opened a new bug for the inactive services: 
https://bugzilla.redhat.com/show_bug.cgi?id=1049211

Comment 14 Omri Hochman 2014-01-12 13:22:23 UTC
Verified with: openstack-foreman-installer-1.0.1-1.el6ost.noarch.

Comment 19 Lon Hohberger 2014-02-04 17:18:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2014-0046.html


Note You need to log in before you can comment on or make changes to this bug.