| Summary: | unable to deploy overcloud | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | bigswitch <rhosp-bugs-internal> | ||||||
| Component: | rhosp-director | Assignee: | Dan Sneddon <dsneddon> | ||||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Arik Chernetsky <achernet> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 8.0 (Liberty) | CC: | dbecker, jcoufal, mburns, morazi, plancast, rhel-osp-director-maint, sbaker, shardy, srevivo | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 10.0 (Newton) | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2016-10-04 19:16:42 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Attachments: |
|
||||||||
Created attachment 1137446 [details]
sos report of controller node
It seems pretty clear from a casual glance at the logs you pasted that RabbitMQ is dead. What makes you think it's a Heat bug? I suspect this is a misconfiguration in your network environment files. Can you attached those to the bug? This bug did not make the OSP 8.0 release. It is being deferred to OSP 10. |
Created attachment 1137445 [details] sosreport of compute node Description of problem: With rhosp 8 beta 8, I am not able to deploy overcloud. The deployment is with one controller and one compute node. Heat Stack create failed. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 883, in take_action self._deploy_tripleo_heat_templates(stack, parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 349, in _deploy_tripleo_heat_templates environments, parsed_args.timeout) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 273, in _heat_deploy raise Exception("Heat Stack create failed.") Exception: Heat Stack create failed. clean_up DeployOvercloud: Heat Stack create failed. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 112, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 255, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 883, in take_action self._deploy_tripleo_heat_templates(stack, parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 349, in _deploy_tripleo_heat_templates environments, parsed_args.timeout) File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 273, in _heat_deploy raise Exception("Heat Stack create failed.") Exception: Heat Stack create failed. END return value: 1 6:19 [0]: SIGNAL_COMPLETE Unknown 2016-03-17 06:46:20 [0]: SIGNAL_COMPLETE Unknown 2016-03-17 06:46:20 [ControllerDeployment]: SIGNAL_ [stack@c5220-01 ~]$ heat resource-list -n 5 overcloud | grep -i failed | ComputeNodesPostDeployment | 34b1f20c-1740-4a22-a00d-ebc72cecc5e3 | OS::TripleO::ComputePostDeployment | CREATE_FAILED | 2016-03-17T06:32:07 | overcloud | | ControllerNodesPostDeployment | 0f36c180-796f-4d2c-a51d-40537751cd8f | OS::TripleO::ControllerPostDeployment | CREATE_FAILED | 2016-03-17T06:32:07 | overcloud | | ComputePuppetDeployment | 20b24c33-7e34-4fbb-80ec-8fa3ea2621d5 | OS::Heat::StructuredDeployments | CREATE_FAILED | 2016-03-17T06:42:26 | overcloud-ComputeNodesPostDeployment-vv6i2k42c3zy | | ControllerOvercloudServicesDeployment_Step4 | 2c3d5671-a7b6-4981-8911-d0825561c15d | OS::Heat::StructuredDeployments | CREATE_FAILED | 2016-03-17T06:42:55 | overcloud-ControllerNodesPostDeployment-spqefhzczq3v | | 0 | 454d979a-704e-41da-93e1-35e631c7b3b8 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-03-17T06:46:55 | overcloud-ControllerNodesPostDeployment-spqefhzczq3v-ControllerOvercloudServicesDeployment_Step4-gsp67fjwdlw5 | 4762:Mar 17 06:46:10 overcloud-novacompute-0.localdomain nova-compute[18264]: 2016-03-17 06:46:10.459 18264 ERROR oslo.messaging._drivers.impl_rabbit [req-49d23488-c534-48c8-85be-98416bffc8eb - - - - -] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 4763:Mar 17 06:46:11 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:46:11.475 18231 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 4764:Mar 17 06:46:11 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:46:11.475 18231 ERROR oslo.messaging._drivers.impl_rabbit [req-b91c7937-a281-4899-ab41-490b5f30d073 - - - - -] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 4765:Mar 17 06:46:12 overcloud-novacompute-0.localdomain nova-compute[18264]: 2016-03-17 06:46:12.467 18264 ERROR oslo.messaging._drivers.impl_rabbit [req-49d23488-c534-48c8-85be-98416bffc8eb - - - - -] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 4787:Mar 17 06:46:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:46:13.686 18231 ERROR neutron.agent.linux.ovsdb_monitor [req-b91c7937-a281-4899-ab41-490b5f30d073 - - - - -] Interface monitor is not active 4802:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state! 4803:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 4804:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 336, in _report_state 4805:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.use_call) 4806:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 86, in report_state 4807:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs) 4808:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call 4809:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry) 4810:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send 4811:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry) 4812:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send 4813:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry) 4814:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 420, in _send 4815:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout) 4816:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 318, in wait 4817:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout) 4818:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 223, in get 4819:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id) 4820:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 50d8a658cd1446e7bab74b752c06a008 10046:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Ceilometer::Agent::Notification/Ceilometer_config[notification/ack_on_event_error]/ensure: created 10599:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Swift::Proxy::Catch_errors/Concat::Fragment[swift_catch_errors]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/24_swift_catch_errors]/ensure: defined content as '{md5}e07f0e5b125db7d6c8b4724c1648bcd5' 10696:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 CRITICAL keystone [-] OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'keystone'@'172.16.2.6' (using password: YES)") 10697:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone Traceback (most recent call last): 10698:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module> 10699:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone sys.exit(main()) 10700:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/manage.py", line 46, in main 10701:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone cli.main(argv=sys.argv, config_files=config_files) 10702:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 685, in main 10703:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone CONF.command.cmd_class.main() 10704:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 77, in main 10705:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone migration_helpers.sync_database_to_version(extension, version) 10706:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 194, in sync_database_to_version 10707:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone _sync_common_repo(version) 10708:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 145, in _sync_common_repo 10709:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone engine = sql.get_engine() 10710:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 188, in get_engine 10711:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return _get_engine_facade().get_engine() 10712:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 176, in _get_engine_facade 10713:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone _engine_facade = db_session.EngineFacade.from_config(CONF) 10714:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1015, in from_config 10715:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone expire_on_commit=expire_on_commit, _conf=conf) 10716:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__ 10717:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone slave_connection=slave_connection) 10718:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start 10719:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone engine_args, maker_args) 10720:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection 10721:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone sql_connection=sql_connection, **engine_kwargs) 10722:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine 10723:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone test_conn = _test_connection(engine, max_retries, retry_interval) 10724:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 326, in _test_connection 10725:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return engine.connect() 10726:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect 10727:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return self._connection_cls(self, **kwargs) 10728:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__ 10729:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone if connection is not None else engine.raw_connection() 10730:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection 10731:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone self.pool.unique_connection, _connection) 10732:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect 10733:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone e, dialect, self) 10734:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection 10735:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone util.raise_from_cause(newraise, exc_info) 10736:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 10737:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone reraise(type(exception), exception, tb=exc_tb) 10738:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 10739:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return fn() 10740:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection 10741:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return _ConnectionFairy._checkout(self) 10742:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout 10743:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone fairy = _ConnectionRecord.checkout(pool) 10744:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout 10745:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone rec = pool._do_get() 10746:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get 10747:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone self._dec_overflow() 10748:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 10749:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone compat.reraise(exc_type, exc_value, exc_tb) 10750:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get 10751:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return self._create_connection() 10752:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection 10753:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return _ConnectionRecord(self) 10754:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__ 10755:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone self.connection = self.__connect() 10756:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect 10757:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone connection = self.__pool._invoke_creator(self) 10758:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect 10759:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return dialect.connect(*cargs, **cparams) 10760:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect 10761:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return self.dbapi.connect(*cargs, **cparams) 10762:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 88, in Connect 10763:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone return Connection(*args, **kwargs) 10764:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 657, in __init__ 10765:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone self.connect() 10766:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 851, in connect 10767:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone self._request_authentication() 10768:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1034, in _request_authentication 10769:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone auth_packet = self._read_packet() 10770:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet 10771:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone packet.check_error() 10772:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error 10773:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone err.raise_mysql_exception(self._data) 10774:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception 10775:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone _check_mysql_exception(errinfo) 10776:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 112, in _check_mysql_exception 10777:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone raise errorclass(errno, errorvalue) 10778:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'keystone'@'172.16.2.6' (using password: YES)") Version-Release number of selected component (if applicable): rhosp 8 beta 8 How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: