Hide Forgot
openstack-foreman-installer: 'neutron-db-manage upgrade' failed and neutron services remained inactive after deployment of client as neutron-controller. Environment (Puddle 2013-12-16.1): ----------------------------------- openstack-foreman-installer-1.0.0-1.el6ost.noarch puppet-server-3.2.4-3.el6_5.noarch puppet-3.2.4-3.el6_5.noarch packstack-modules-puppet-2013.2.1-0.20.dev936.el6ost.noarch ruby-1.8.7.352-13.el6.x86_64 rubygems-1.8.16-1.el6.noarch openstack-neutron-2013.2-16.el6ost.noarch Steps: ------- - Attempt to deploy foreman-client as neutron controller (hostGroup) Results: -------- - After deployment finished neutron services remained inactive. - Error in /var/log/messages indicate there was a problem to perform :' neutron-db-manage upgrade ; neutron services ('openstack-status' view): ------------------------------------------ == neutron services == neutron-server: inactive (disabled on boot) neutron-dhcp-agent: inactive (disabled on boot) neutron-l3-agent: inactive (disabled on boot) neutron-metadata-agent: inactive (disabled on boot) neutron-lbaas-agent: inactive (disabled on boot) neutron-openvswitch-agent: inactive (disabled on boot) ------------------------------------------------------------------------- [root@puma01 ~]# chkconfig | grep neutron neutron-dhcp-agent 0:off 1:off 2:off 3:off 4:off 5:off 6:off neutron-l3-agent 0:off 1:off 2:off 3:off 4:off 5:off 6:off neutron-lbaas-agent 0:off 1:off 2:off 3:off 4:off 5:off 6:off neutron-metadata-agent 0:off 1:off 2:off 3:off 4:off 5:off 6:off neutron-openvswitch-agent 0:off 1:off 2:off 3:off 4:off 5:off 6:off neutron-server 0:off 1:off 2:on 3:on 4:on 5:on 6:off ---------------------------------------------------------------------------- less /var/log/messages : ------------------------ Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/auth_uri]) Scheduling refresh of Exec[glance-manage db_sync] Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/auth_uri]) Scheduling refresh of Service[glance-api] Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_api_config[filter:authtoken/auth_uri]/ensure) created Dec 17 10:46:53 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_api_config[filter:authtoken/auth_uri]) Scheduling refresh of Service[neutron-server] Dec 17 10:46:54 puma01 abrt: detected unhandled Python exception in '/usr/bin/neutron-db-manage' Dec 17 10:46:54 puma01 abrtd: New client connected Dec 17 10:46:54 puma01 abrtd: Directory 'pyhook-2013-12-17-10:46:54-16842' creation detected Dec 17 10:46:54 puma01 abrt-server[16850]: Saved Python crash dump of pid 16842 to /var/spool/abrt/pyhook-2013-12-17-10:46:54-16842 Dec 17 10:46:54 puma01 abrtd: Package 'openstack-neutron' isn't signed with proper key Dec 17 10:46:54 puma01 abrtd: 'post-create' on '/var/spool/abrt/pyhook-2013-12-17-10:46:54-16842' exited with 1 Dec 17 10:46:54 puma01 abrtd: Deleting problem directory '/var/spool/abrt/pyhook-2013-12-17-10:46:54-16842' Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) No handlers could be found for logger "neutron.common.legac y" Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) Traceback (most recent call last): Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/bin/neutron-db-manage", line 10, in <module> Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) sys.exit(main()) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/neutron/db/migrati on/cli.py", line 143, in main Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) CONF.command.func(config, CONF.command.name) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/neutron/db/migrati on/cli.py", line 80, in do_upgrade_downgrade Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) do_alembic_command(config, cmd, revision, sql=CONF.comm and.sql) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/neutron/db/migrati on/cli.py", line 59, in do_alembic_command Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) getattr(alembic_command, cmd)(config, *args, **kwargs) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/alembic/command.py ", line 124, in upgrade Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) script.run_env() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/alembic/script.py", line 191, in run_env Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) util.load_python_file(self.dir, 'env.py') Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/alembic/util.py", line 186, in load_python_file Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) module = imp.load_source(module_id, path, open(path, 'rb')) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 105, in <module> Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) run_migrations_online() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 80, in run_migrations_online Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) connection = engine.connect() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2472, in connect Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return self._connection_cls(self, **kwargs) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__ Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) self.__connection = connection or engine.raw_connection() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2558, in raw_connection Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return self.pool.unique_connection() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 183, in unique_connection Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return _ConnectionFairy(self).checkout() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 387, in __init__ Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) rec = self._connection_record = pool._do_get() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 802, in _do_get Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return self._create_connection() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 188, in _create_connection Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return _ConnectionRecord(self) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 270, in __init__ Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) self.connection = self.__connect() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) connection = self.__pool._creator() Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return dialect.connect(*cargs, **cparams) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return self.dbapi.connect(*cargs, **cparams) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) return Connection(*args, **kwargs) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__ Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) super(Connection, self).__init__(*args, **kwargs2) Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) sqlalchemy.exc.OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '10.35.160.11' (111)") None None Dec 17 10:46:54 puma01 puppet-agent[16303]: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head returned 1 instead of one of [0] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns) change from notrun to 0 failed: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head returned 1 instead of one of [0] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_protocol]/ensure) created Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_protocol]) Scheduling refresh of Exec[glance-manage db_sync] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_protocol]) Scheduling refresh of Service[glance-registry] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/admin_user]/ensure) created Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/admin_user]) Scheduling refresh of Exec[glance-manage db_sync] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Api/Glance_api_config[keystone_authtoken/admin_user]) Scheduling refresh of Service[glance-api] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/max_retries]/ensure) created Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/max_retries]) Scheduling refresh of Service[neutron-server] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/File[/etc/glance/glance-registry-paste.ini]/ensure) created Dec 17 10:46:54 puma01 puppet-agent[16303]: (/File[/etc/glance/glance-registry-paste.ini]) Scheduling refresh of Service[glance-registry] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/retry_interval]/ensure) created Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Neutron::Server/Neutron_config[database/retry_interval]) Scheduling refresh of Service[neutron-server] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_uri]/ensure) created Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_uri]) Scheduling refresh of Exec[glance-manage db_sync] Dec 17 10:46:54 puma01 puppet-agent[16303]: (/Stage[main]/Glance::Registry/Glance_registry_config[keystone_authtoken/auth_uri]) Scheduling refresh of Service[glance-registry]
After the installation : I was able to start neutron services manually.
We think there is an ordering dependency between setting up the database and running this stamp command. I would be interested to see if the same thing happens with packstack, since we mimic what they did (perhaps there is a chunk we missed?). Anyway, this used to be an outright error. If you would, on your next test run, could you try running puppet agaent a second time instead of manually starting the neutron services? If that results in services running, I think we can call this a known issue, and work on a 0day fix for the ordering. Also note that this will likely be fixed in the puppet-neutron code, so this is a temporary work around anyway, as I mention in the related BZs, startin ghere: https://bugzilla.redhat.com/show_bug.cgi?id=1042933 Please let me know the results of your test, and we'll proceed from there. I am almost ready to run these teste here myself with the latest puddle.
Jirka Stransky replicated the issues and confirmed that re-running seems to resolve the issues, but I'd like to get confirmation from Omri as well so we have confirmation in more than one environment.
OK, I have just posted a new patch that @cwolfe is reviewing: https://github.com/redhat-openstack/astapor/pull/91 I ran it successfully against the latest on a fresh in stall, and it passed twice in this scenario. Commit message: The 'neutron-db-manage upgrade' command was failing on first runs of puppet agent due to a race condition where the database was not necessarily created before this command was run. The order matters, so we have now specified this more precisely.
This is merged, ready for next release.
The issue constantly reproduced on my environment - last try with openstack-foreman-installer-1.0.0-1.el6ost.noarch (puddle 2013-12-19.1). Attempted to run puppet agent the second time as a workaround: (Neutron services remained inactive.) ------------------------------------------------------------- [root@puma01 ~]# openstack-status == Nova services == openstack-nova-api: active openstack-nova-cert: active openstack-nova-compute: dead (disabled on boot) openstack-nova-network: dead (disabled on boot) openstack-nova-scheduler: active openstack-nova-conductor: active == Glance services == openstack-glance-api: active openstack-glance-registry: active == Keystone service == openstack-keystone: active == Horizon service == openstack-dashboard: 500 == neutron services == neutron-server: active neutron-dhcp-agent: inactive (disabled on boot) neutron-l3-agent: inactive (disabled on boot) neutron-metadata-agent: inactive (disabled on boot) neutron-lbaas-agent: inactive (disabled on boot) neutron-openvswitch-agent: inactive (disabled on boot) == Swift services == openstack-swift-proxy: dead openstack-swift-account: dead (disabled on boot) openstack-swift-container: dead (disabled on boot) openstack-swift-object: dead (disabled on boot) == Cinder services == openstack-cinder-api: active openstack-cinder-scheduler: active openstack-cinder-volume: inactive (disabled on boot) == Ceilometer services == openstack-ceilometer-api: active openstack-ceilometer-central: active openstack-ceilometer-compute: dead (disabled on boot) openstack-ceilometer-collector: active == Heat services == openstack-heat-api: active openstack-heat-api-cfn: inactive (disabled on boot) openstack-heat-api-cloudwatch: inactive (disabled on boot) openstack-heat-engine: active == Support services == mysqld: active openvswitch: dead messagebus: active qpidd: active memcached: active == Keystone users == Warning keystonerc not sourced [root@puma01 ~]# puppet agent -t -v Info: Retrieving plugin Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/ipa_client_configured.rb Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/hamysql_active_node.rb Info: Loading facts in /var/lib/puppet/lib/facter/network.rb Info: Loading facts in /var/lib/puppet/lib/facter/netns_support.rb Info: Caching catalog for puma01.scl.lab.tlv.redhat.com Info: Applying configuration version '1387534464' Notice: /Stage[main]/Quickstack::Neutron::Controller/Exec[neutron-db-manage upgrade]/returns: executed successfully Notice: /File[/etc/httpd/conf/httpd.conf]/content: --- /etc/httpd/conf/httpd.conf 2013-12-20 16:45:15.152953403 +0200 +++ /tmp/puppet-file20131220-27250-1szxzp3-0 2013-12-20 16:56:08.271567023 +0200 @@ -131,7 +131,7 @@ # prevent Apache from glomming onto all bound IP addresses (0.0.0.0) # #Listen 12.34.56.78:80 -Listen 0.0.0.0:80 +Listen 80 # # Load config files from the config directory "/etc/httpd/conf.d". Info: FileBucket got a duplicate file {md5}716b40d1de69187bba4a36821f699591 Info: /File[/etc/httpd/conf/httpd.conf]: Filebucketed /etc/httpd/conf/httpd.conf to puppet with sum 716b40d1de69187bba4a36821f699591 Notice: /File[/etc/httpd/conf/httpd.conf]/content: content changed '{md5}716b40d1de69187bba4a36821f699591' to '{md5}ec999e9a079a8fad9e374a8e01df47f4' Info: /File[/etc/httpd/conf/httpd.conf]: Scheduling refresh of Service[httpd] Notice: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]/value: value changed 'f9cd7bb734ef998296b5e89cc9d32baf' to '%ENCRYPTION_KEY%' Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api] Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-engine] Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api-cloudwatch] Info: /Stage[main]/Heat::Engine/Heat_config[DEFAULT/auth_encryption_key]: Scheduling refresh of Service[heat-api-cfn] Notice: /Stage[main]/Swift::Proxy/Service[swift-proxy]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Swift::Proxy/Service[swift-proxy]: Unscheduling refresh on Service[swift-proxy] Notice: /Stage[main]/Horizon/File_line[httpd_listen_on_bind_address_80]/ensure: created Info: /Stage[main]/Horizon/File_line[httpd_listen_on_bind_address_80]: Scheduling refresh of Service[httpd] Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Heat::Api/Service[heat-api]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Heat::Engine/Exec[heat-encryption-key-replacement]/returns: executed successfully Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Apache/Service[httpd]: Triggered 'refresh' from 2 events Notice: Finished catalog run in 46.43 seconds You have new mail in /var/spool/mail/root [root@puma01 ~]# openstack-status == Horizon service == openstack-dashboard: 500 == neutron services == neutron-server: active neutron-dhcp-agent: inactive (disabled on boot) neutron-l3-agent: inactive (disabled on boot) neutron-metadata-agent: inactive (disabled on boot) neutron-lbaas-agent: inactive (disabled on boot) neutron-openvswitch-agent: inactive (disabled on boot)
I just had this problem with openstack-compute (with nova-network) : It seems that this issue is relevant not only to neutron host groups. /var/log/messages (attached full file) ----------------------------------------- Dec 22 18:48:08 puma02 puppet-agent[16089]: (/Stage[main]/Quickstack::Nova_network::Compute/Nova::Generic_service[metadata-api]/Service[nova-metadata-api]/ensure) ensure changed 'stopped' to 'running' Dec 22 18:48:10 puma02 puppet-agent[16089]: Execution of '/usr/bin/nova-manage network create novanetwork 10.0.0.0/24 1 255' returned 1: 2013-12-22 18:48:10.450 16849 INFO nova.network.drive r [-] Loading network driver 'nova.network.linux_net' Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 CRITICAL nova [req-2e4f944a-dfc2-4f6b-87b7-a2702ecd8b72 None None] (OperationalError) (1130, "Host 'puma02.scl.lab.t lv.redhat.com' is not allowed to connect to this MySQL server") None None Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova Traceback (most recent call last): Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/bin/nova-manage", line 10, in <module> Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova sys.exit(main()) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1377, in main Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova ret = fn(*fn_args, **fn_kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 530, in create Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova net_manager.create_networks(context.get_admin_context(), **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1073, in create_networks Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return self._do_create_networks(context, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1113, in _do_create_networks Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova nets = self.db.network_get_all(context) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 920, in network_get_all Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return IMPL.network_get_all(context, project_only) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 126, in wrapper Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return f(*args, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2526, in network_get_all Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova project_only=project_only).all() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 190, in model_query Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova session = kwargs.get('session') or get_session() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 423, in get_session Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova engine = get_engine(sqlite_fk=sqlite_fk, slave_engine=slave_session) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 561, in get_engine Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova sqlite_fk=sqlite_fk) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 675, in create_engine Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova engine.connect() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2472, in connect Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return self._connection_cls(self, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova self.__connection = connection or engine.raw_connection() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2558, in raw_connection Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return self.pool.unique_connection() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 183, in unique_connection Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return _ConnectionFairy(self).checkout() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 387, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova rec = self._connection_record = pool._do_get() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 741, in _do_get Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova con = self._create_connection() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 188, in _create_connection Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return _ConnectionRecord(self) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 270, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova self.connection = self.__connect() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova connection = self.__pool._creator() Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return dialect.connect(*cargs, **cparams) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return self.dbapi.connect(*cargs, **cparams) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova return Connection(*args, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova super(Connection, self).__init__(*args, **kwargs2) Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova OperationalError: (OperationalError) (1130, "Host 'puma02.scl.lab.tlv.redhat.com' is not allowed to connect to this MySQL server") None None Dec 22 18:48:10 puma02 puppet-agent[16089]: 2013-12-22 18:48:10.852 16849 TRACE nova Dec 22 18:48:10 puma02 puppet-agent[16089]: Command failed, please check log for more info Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) change from absent to present failed: Execution of '/usr/bin/nova-manage network create novanetwork 10.0.0.0/24 1 255' returned 1: 2013-12-22 18:48:10.450 16849 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net' Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 CRITICAL nova [req-2e4f944a-dfc2-4f6b-87b7-a2702ecd8b72 None None] (OperationalError) (1130, "Host 'puma02.scl.lab.tlv.redhat.com' is not allowed to connect to this MySQL server") None None Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova Traceback (most recent call last): Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/bin/nova-manage", line 10, in <module> Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova sys.exit(main()) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 1377, in main Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova ret = fn(*fn_args, **fn_kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/cmd/manage.py", line 530, in create Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova net_manager.create_networks(context.get_admin_context(), **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1073, in create_networks Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return self._do_create_networks(context, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1113, in _do_create_networks Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova nets = self.db.network_get_all(context) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 920, in network_get_all Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return IMPL.network_get_all(context, project_only) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 126, in wrapper Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return f(*args, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2526, in network_get_all Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova project_only=project_only).all() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 190, in model_query Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova session = kwargs.get('session') or get_session() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 423, in get_session Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova engine = get_engine(sqlite_fk=sqlite_fk, slave_engine=slave_session) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 561, in get_engine Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova sqlite_fk=sqlite_fk) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 675, in create_engine Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova engine.connect() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2472, in connect Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return self._connection_cls(self, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 878, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova self.__connection = connection or engine.raw_connection() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2558, in raw_connection Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return self.pool.unique_connection() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 183, in unique_connection Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return _ConnectionFairy(self).checkout() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 387, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova rec = self._connection_record = pool._do_get() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 741, in _do_get Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova con = self._create_connection() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 188, in _create_connection Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return _ConnectionRecord(self) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 270, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova self.connection = self.__connect() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/pool.py", line 330, in __connect Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova connection = self.__pool._creator() Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 80, in connect Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return dialect.connect(*cargs, **cparams) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 281, in connect Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return self.dbapi.connect(*cargs, **cparams) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova return Connection(*args, **kwargs) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__ Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova super(Connection, self).__init__(*args, **kwargs2) Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova OperationalError: (OperationalError) (1130, "Host 'puma02.scl.lab.tlv.redhat.com' is not allowed to connect to this MySQL server") None None Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) 2013-12-22 18:48:10.852 16849 TRACE nova Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure) Command failed, please check log for more info Dec 22 18:48:10 puma02 puppet-agent[16089]: (/Stage[main]/Ceilometer::Agent::Compute/File_line_after[nova-notification-driver-ceilometer]/ensure) created Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Compute/Nova::Generic_service[compute]/Service[nova-compute]/ensure) ensure changed 'stopped' to 'running' Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Nova::Network/Nova::Generic_service[network]/Service[nova-network]/ensure) ensure changed 'stopped' to 'running' Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Ceilometer::Agent::Auth/Ceilometer_config[DEFAULT/os_auth_region]/ensure) created Dec 22 18:48:11 puma02 puppet-agent[16089]: (/Stage[main]/Ceilometer::Agent::Compute/Service[ceilometer-agent-compute]/ensure) ensure changed 'stopped' to 'running' Dec 22 18:48:11 puma02 puppet-agent[16089]: Finished catalog run in 403.88 seconds Dec 22 18:51:52 puma02 polkitd[16973]: started daemon version 0.96 using authority implementation `local' version `0.96' Dec 22 18:53:31 puma02 puppet-agent[17044]: Retrieving plugin
Created attachment 840379 [details] messages.log from the'compute with nova-network' machine.
Two things: 1. Please try the latest, openstack-foreman-installer-1.0.1-1.el6ost.noarch.rpm as this bug should have been fixed in the last minor bump revision. 2. The paste two comments above was from the compute node-- for this bug I'd like to stay focused on what was originally reported, a specific error on the neutron server. If the errors reported above on the compute node are still an issue with the latest puddle, perhaps that should be a separate BZ. Thanks, --Crag
Clarification: the "'neutron-db-manage upgrade' failed" aspect of this bug is what is currently fixed. However, the extra "neutron-dhcp-agent, neutron-l3-agent" services remain as inactive and turned off. There is not a quick fix for this-- perhaps could be moved as a separate bug.
(In reply to Crag Wolfe from comment #11) > Clarification: the "'neutron-db-manage upgrade' failed" aspect of this bug > is what is currently fixed. However, the extra "neutron-dhcp-agent, > neutron-l3-agent" services remain as inactive and turned off. There is not > a quick fix for this-- perhaps could be moved as a separate bug. As Crag wrote: It seems that 'neutron-db-manage upgrade' aspect is already fixed with openstack-foreman-installer-1.0.1-1.el6ost.noarch. I've opened a new bug for the inactive services: https://bugzilla.redhat.com/show_bug.cgi?id=1049211
Verified with: openstack-foreman-installer-1.0.1-1.el6ost.noarch.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2014-0046.html