Bug 1318748 - unable to deploy overcloud
Summary: unable to deploy overcloud
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 10.0 (Newton)
Assignee: Dan Sneddon
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-17 16:35 UTC by bigswitch
Modified: 2016-10-04 19:16 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-04 19:16:42 UTC
Target Upstream Version:


Attachments (Terms of Use)
sosreport of compute node (6.33 MB, application/x-xz)
2016-03-17 16:35 UTC, bigswitch
no flags Details
sos report of controller node (6.95 MB, application/x-xz)
2016-03-17 16:36 UTC, bigswitch
no flags Details

Description bigswitch 2016-03-17 16:35:46 UTC
Created attachment 1137445 [details]
sosreport of compute node

Description of problem:
With rhosp 8 beta 8, I am not able to deploy overcloud. The deployment is with one controller and one compute node. 


Heat Stack create failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 883, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 349, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 273, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.
clean_up DeployOvercloud: Heat Stack create failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 112, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 255, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 883, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 349, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 273, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.

END return value: 1
6:19 [0]: SIGNAL_COMPLETE  Unknown
2016-03-17 06:46:20 [0]: SIGNAL_COMPLETE  Unknown
2016-03-17 06:46:20 [ControllerDeployment]: SIGNAL_


[stack@c5220-01 ~]$ heat resource-list -n 5 overcloud | grep -i failed
| ComputeNodesPostDeployment                   | 34b1f20c-1740-4a22-a00d-ebc72cecc5e3          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED      | 2016-03-17T06:32:07 | overcloud                                                                                                                                       |
| ControllerNodesPostDeployment                | 0f36c180-796f-4d2c-a51d-40537751cd8f          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED      | 2016-03-17T06:32:07 | overcloud                                                                                                                                       |
| ComputePuppetDeployment                      | 20b24c33-7e34-4fbb-80ec-8fa3ea2621d5          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2016-03-17T06:42:26 | overcloud-ComputeNodesPostDeployment-vv6i2k42c3zy                                                                                               |
| ControllerOvercloudServicesDeployment_Step4  | 2c3d5671-a7b6-4981-8911-d0825561c15d          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2016-03-17T06:42:55 | overcloud-ControllerNodesPostDeployment-spqefhzczq3v                                                                                            |
| 0                                            | 454d979a-704e-41da-93e1-35e631c7b3b8          | OS::Heat::StructuredDeployment                    | CREATE_FAILED      | 2016-03-17T06:46:55 | overcloud-ControllerNodesPostDeployment-spqefhzczq3v-ControllerOvercloudServicesDeployment_Step4-gsp67fjwdlw5                                   |


4762:Mar 17 06:46:10 overcloud-novacompute-0.localdomain nova-compute[18264]: 2016-03-17 06:46:10.459 18264 ERROR oslo.messaging._drivers.impl_rabbit [req-49d23488-c534-48c8-85be-98416bffc8eb - - - - -] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
4763:Mar 17 06:46:11 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:46:11.475 18231 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
4764:Mar 17 06:46:11 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:46:11.475 18231 ERROR oslo.messaging._drivers.impl_rabbit [req-b91c7937-a281-4899-ab41-490b5f30d073 - - - - -] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
4765:Mar 17 06:46:12 overcloud-novacompute-0.localdomain nova-compute[18264]: 2016-03-17 06:46:12.467 18264 ERROR oslo.messaging._drivers.impl_rabbit [req-49d23488-c534-48c8-85be-98416bffc8eb - - - - -] AMQP server on 172.16.2.6:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
4787:Mar 17 06:46:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:46:13.686 18231 ERROR neutron.agent.linux.ovsdb_monitor [req-b91c7937-a281-4899-ab41-490b5f30d073 - - - - -] Interface monitor is not active
4802:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!
4803:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
4804:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 336, in _report_state
4805:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.use_call)
4806:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 86, in report_state
4807:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(context, 'report_state', **kwargs)
4808:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
4809:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
4810:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
4811:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     timeout=timeout, retry=retry)
4812:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
4813:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
4814:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 420, in _send
4815:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = self._waiter.wait(msg_id, timeout)
4816:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 318, in wait
4817:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
4818:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 223, in get
4819:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
4820:Mar 17 06:47:13 overcloud-novacompute-0.localdomain neutron-openvswitch-agent[18231]: 2016-03-17 06:47:13.552 18231 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 50d8a658cd1446e7bab74b752c06a008


10046:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Ceilometer::Agent::Notification/Ceilometer_config[notification/ack_on_event_error]/ensure: created
10599:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Swift::Proxy::Catch_errors/Concat::Fragment[swift_catch_errors]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/24_swift_catch_errors]/ensure: defined content as '{md5}e07f0e5b125db7d6c8b4724c1648bcd5'
10696:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 CRITICAL keystone [-] OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'keystone'@'172.16.2.6' (using password: YES)")
10697:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone Traceback (most recent call last):
10698:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/bin/keystone-manage", line 10, in <module>
10699:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     sys.exit(main())
10700:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/manage.py", line 46, in main
10701:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     cli.main(argv=sys.argv, config_files=config_files)
10702:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 685, in main
10703:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     CONF.command.cmd_class.main()
10704:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 77, in main
10705:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     migration_helpers.sync_database_to_version(extension, version)
10706:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 194, in sync_database_to_version
10707:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     _sync_common_repo(version)
10708:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/migration_helpers.py", line 145, in _sync_common_repo
10709:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     engine = sql.get_engine()
10710:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 188, in get_engine
10711:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return _get_engine_facade().get_engine()
10712:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 176, in _get_engine_facade
10713:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     _engine_facade = db_session.EngineFacade.from_config(CONF)
10714:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1015, in from_config
10715:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     expire_on_commit=expire_on_commit, _conf=conf)
10716:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__
10717:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     slave_connection=slave_connection)
10718:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start
10719:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     engine_args, maker_args)
10720:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection
10721:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     sql_connection=sql_connection, **engine_kwargs)
10722:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine
10723:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     test_conn = _test_connection(engine, max_retries, retry_interval)
10724:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 326, in _test_connection
10725:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return engine.connect()
10726:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect
10727:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return self._connection_cls(self, **kwargs)
10728:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__
10729:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     if connection is not None else engine.raw_connection()
10730:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
10731:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     self.pool.unique_connection, _connection)
10732:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
10733:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     e, dialect, self)
10734:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
10735:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     util.raise_from_cause(newraise, exc_info)
10736:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
10737:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     reraise(type(exception), exception, tb=exc_tb)
10738:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
10739:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return fn()
10740:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection
10741:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return _ConnectionFairy._checkout(self)
10742:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
10743:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     fairy = _ConnectionRecord.checkout(pool)
10744:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
10745:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     rec = pool._do_get()
10746:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get
10747:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     self._dec_overflow()
10748:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
10749:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     compat.reraise(exc_type, exc_value, exc_tb)
10750:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get
10751:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return self._create_connection()
10752:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
10753:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return _ConnectionRecord(self)
10754:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
10755:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     self.connection = self.__connect()
10756:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
10757:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     connection = self.__pool._invoke_creator(self)
10758:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
10759:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return dialect.connect(*cargs, **cparams)
10760:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
10761:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return self.dbapi.connect(*cargs, **cparams)
10762:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 88, in Connect
10763:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     return Connection(*args, **kwargs)
10764:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 657, in __init__
10765:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     self.connect()
10766:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 851, in connect
10767:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     self._request_authentication()
10768:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1034, in _request_authentication
10769:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     auth_packet = self._read_packet()
10770:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
10771:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     packet.check_error()
10772:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
10773:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     err.raise_mysql_exception(self._data)
10774:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
10775:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     _check_mysql_exception(errinfo)
10776:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 112, in _check_mysql_exception
10777:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone     raise errorclass(errno, errorvalue)
10778:Mar 17 06:50:18 overcloud-controller-0.localdomain os-collect-config[7326]: Notice: /Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]/returns: 2016-03-17 06:49:06.015 2044 ERROR keystone OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'keystone'@'172.16.2.6' (using password: YES)")


Version-Release number of selected component (if applicable):
rhosp 8 beta 8

How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 bigswitch 2016-03-17 16:36:45 UTC
Created attachment 1137446 [details]
sos report of controller node

Comment 3 Zane Bitter 2016-03-18 01:27:56 UTC
It seems pretty clear from a casual glance at the logs you pasted that RabbitMQ is dead. What makes you think it's a Heat bug?

Comment 4 Mike Burns 2016-03-22 17:28:43 UTC
I suspect this is a misconfiguration in your network environment files.  Can you attached those to the bug?

Comment 5 Mike Burns 2016-04-07 21:14:44 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.


Note You need to log in before you can comment on or make changes to this bug.