1464247 – MySQL server has gone away error reported when issuing an ironic command after the setup has been idle for a while

Bug 1464247 - MySQL server has gone away error reported when issuing an ironic command after the setup has been idle for a while

Summary: MySQL server has gone away error reported when issuing an ironic command afte...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-zaqar
Sub Component:
Version:	12.0 (Pike)
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	12.0 (Pike)
Assignee:	Thomas Hervé
QA Contact:	Ronnie Rasouli
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-06-22 19:24 UTC by Dan Yasny
Modified:	2018-02-05 19:07 UTC (History)
CC List:	17 users (show)
Fixed In Version:	openstack-zaqar-5.0.1-0.20170905222047.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-12-13 21:33:29 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1691951	None	None	None	2017-09-28 10:46:05 UTC
OpenStack gerrit	473612	None	MERGED	Use oslo_db for create_engine	2021-02-04 23:02:41 UTC
Red Hat Product Errata	RHEA-2017:3462	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-16 01:43:25 UTC

Description Dan Yasny 2017-06-22 19:24:56 UTC

Description of problem:
This has already happened several times, but seems to be hard to reproduce consistently.

A setup left alone for a long time (overnight tipycally) will fail to respond to an ironic related command producing a MySQL error.

A second attempt at the same command usually works.

Simple API queries (e.g. ironic node-list) also work.

Examples:
(openstack) baremetal introspection bulk start
This command is deprecated. Please use "openstack overcloud node introspect" to introspect manageable nodes instead.
Setting nodes for introspection to manageable...
Starting introspection of manageable nodes
Waiting for introspection to finish...
{u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'d97b577e-3812-4998-9afb-4710cd33bd37\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'd97b577e-3812-4998-9afb-4710cd33bd37'}, u'api': u'v2', u'headers': {u'Client-ID': u'e8c19088-723a-4621-b945-719155daa3bd', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}


 openstack baremetal import --json /home/stack/instackenv.json
This command is deprecated. Please use "openstack overcloud node import" to register nodes instead.
{u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'8f6d7d3087824450ade39c5854e7b7d1\', \'name\': u\'52cd9312-e11e-49f8-bb75-07859bacdd58\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'52cd9312-e11e-49f8-bb75-07859bacdd58'}, u'api': u'v2', u'headers': {u'Client-ID': u'88d661b5-c0d1-4887-85ea-21871ad0d836', u'X-Project-ID': u'8f6d7d3087824450ade39c5854e7b7d1'}}}

Version-Release number of selected component (if applicable):

puppet-ironic-11.2.0-0.20170611220351.0fa87ae.el7ost.noarch
python-ironic-lib-2.7.0-0.20170331023537.8d899bd.el7ost.noarch
python-ironicclient-1.13.0-0.20170512184144.7cceefe.el7ost.noarch
python-ironic-inspector-client-1.12.0-0.20170331023022.045fd9e.el7ost.noarch
python-PyMySQL-0.7.9-2.el7ost.noarch
openstack-nova-compute-16.0.0-0.20170609171025.el7ost.noarch
openstack-ironic-conductor-8.0.1-0.20170610002959.adf3461.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tempest-16.0.1-0.20170612053404.c0e31f1.el7ost.noarch
openstack-tripleo-common-containers-7.1.1-0.20170612015630.414f94b.el7ost.noarch
openstack-tripleo-ui-7.1.1-0.20170609131448.202695f.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170609171025.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170611233361.el7ost.noarch
openstack-swift-object-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-swift-proxy-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170612043222.a289f55.el7ost.noarch
python-openstackclient-3.11.0-0.20170505192416.el7ost.noarch
openstack-mistral-api-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-selinux-0.8.7-1.el7ost.noarch
openstack-keystone-12.0.0-0.20170602202335.8628235.el7ost.noarch
openstack-tripleo-image-elements-7.0.0-0.20170607161959.401d861.el7ost.noarch
puppet-openstack_extras-11.2.0-0.20170609110653.d9f8351.el7ost.noarch
openstack-neutron-common-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-ironic-inspector-5.1.1-0.20170608115120.a46e708.el7ost.noarch
python-openstacksdk-0.9.16-0.20170608090447.bfc88e7.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170605222006.25e08b3.el7ost.noarch
puppet-openstacklib-11.2.0-0.20170609110906.a99d5dc.el7ost.noarch
openstack-tripleo-common-7.1.1-0.20170612015630.414f94b.el7ost.noarch
openstack-swift-account-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-heat-common-9.0.0-0.20170612043222.a289f55.el7ost.noarch
openstack-ironic-api-8.0.1-0.20170610002959.adf3461.el7ost.noarch
openstack-mistral-executor-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-tripleo-validations-7.1.1-0.20170609083517.e120452.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170609171025.el7ost.noarch
openstack-glance-15.0.0-0.20170609191014.61046c8.el7ost.noarch
openstack-nova-common-16.0.0-0.20170609171025.el7ost.noarch
openstack-heat-api-9.0.0-0.20170612043222.a289f55.el7ost.noarch
python-openstack-mistral-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-mistral-common-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-zaqar-5.0.0-0.20170609114413.51299ba.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170609171025.el7ost.noarch
openstack-swift-container-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-neutron-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170612043222.a289f55.el7ost.noarch
openstack-ironic-common-8.0.1-0.20170610002959.adf3461.el7ost.noarch
openstack-mistral-engine-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-nova-api-16.0.0-0.20170609171025.el7ost.noarch
mariadb-5.5.56-2.el7.x86_64
mariadb-server-5.5.56-2.el7.x86_64
mariadb-libs-5.5.56-2.el7.x86_64

How reproducible:
hard to reproduce since it take a very long idle period

Steps to Reproduce:
1. install undercloud
2. leave it for a long time (~12 hours seems to work)
3. try to run one of the above commands (there may be others)

Actual results:
see the failures above

Expected results:
should always work

Additional info:
leaving a setup idle now so I can rerun this with --debug tomorrow

Comment 1 Dan Yasny 2017-06-23 15:27:52 UTC

(undercloud) [stack@undercloud-0 ~]$ openstack --debug baremetal introspection bulk start
START with options: [u'--debug', u'baremetal', u'introspection', u'bulk', u'start']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', auth_type='password', auth_url='http://192.168.24.1:5000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, old_profile=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.29', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', roles='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
orchestration API version 1, cmd group openstack.orchestration.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
key_manager API version 1, cmd group openstack.key_manager.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
baremetal API version 1.29, cmd group openstack.baremetal.v1
messaging API version 2, cmd group openstack.messaging.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
dns API version 2, cmd group openstack.dns.v2
data_processing API version 1.1, cmd group openstack.data_processing.v1
database API version 1, cmd group openstack.database.v1
container_infra API version 1, cmd group openstack.container_infra.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
command: baremetal introspection bulk start -> tripleoclient.v1.baremetal.StartBaremetalIntrospectionBulk (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.24.1:5000/', 'password': '***', 'project_domain_name': 'Default'}
Get auth_ref
REQ: curl -g -i -X GET http://192.168.24.1:5000/ -H "Accept: application/json" -H "User-Agent: osc-lib/1.6.0 keystoneauth1/2.20.0 python-requests/2.11.1 CPython/2.7.5"
Starting new HTTP connection (1): 192.168.24.1
"GET / HTTP/1.1" 300 598
RESP: [300] Date: Fri, 23 Jun 2017 15:23:49 GMT Server: Apache Vary: X-Auth-Token Content-Length: 598 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.24.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.24.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Making authentication request to http://192.168.24.1:5000/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 8174
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "ccda3744034848d2a0da0ec5eadcd821", "name": "admin"}], "expires_at": "2017-06-23T19:23:49.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "55f170434da2433d8d5c498f51efcc11", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.24.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11cece1679ad4d78acb58c22c914f58b"}, {"url": "http://192.168.24.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba56a2fb27a74c62b14625e96e325786"}, {"url": "http://192.168.24.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "d9d1d844f69b4f33aa3db31a9b6fbda0"}], "type": "messaging", "id": "5d5df786da294c3497330070dce18792", "name": "zaqar"}, {"endpoints": [{"url": "http://192.168.24.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4894e60793ec4a45a479205750b6d547"}, {"url": "http://192.168.24.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "979b44ec318540b994c0c532320ffe85"}, {"url": "http://192.168.24.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "c9142a68eb704c3eb3e6a1113746063c"}], "type": "placement", "id": "6c8bb1d3d3ef42b6ac85c4bc71412e94", "name": "placement"}, {"endpoints": [{"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8eb5261e2d55476e83de82c22ee62600"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a6446773bef34ac286856107b69da81f"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef43b528dc5e411a867b8b4a0ffa23be"}], "type": "cloudformation", "id": "76432ddf4a904711b1d8205d9f55c118", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "12375bd744494f41be2bd864bac4c5d5"}, {"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a66d122477c64c02b70a413276452bed"}, {"url": "http://192.168.24.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f013ed60d58f48b2a2fe5c15df91fa08"}], "type": "object-store", "id": "7da80e7fc8b44dc4b302dbf3bd4cf12c", "name": "swift"}, {"endpoints": [{"url": "http://192.168.24.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "16e55c12241f4ac18e896db4e1a61649"}, {"url": "http://192.168.24.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2e96fdd236eb40d0ae96603bd5792420"}, {"url": "http://192.168.24.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "93753cd9c4df4705bc9d1400918434e8"}], "type": "network", "id": "9bb9c0cd038241228a0a5c450343cbbd", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.24.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4348e9dbbcc14972b6dc1e1bd71d873c"}, {"url": "http://192.168.24.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59659611cdcc4a1a9a04205e8c99acfd"}, {"url": "http://192.168.24.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "59942a334d8541daa79b0ad68b6c186c"}], "type": "workflowv2", "id": "af9164fadde348588d7b5cd0e9ac30ad", "name": "mistral"}, {"endpoints": [{"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "927f45b99634430e8d5da5645313d838"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bd4266a54f9d456da85c11ca6dd24674"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ed92f2fce02442cbb26d18c09b6413ec"}], "type": "orchestration", "id": "c5040fa43d7d409bacb81c808b5d3d3e", "name": "heat"}, {"endpoints": [{"url": "http://192.168.24.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "37763251b4724be5bc2d16ac45caadc0"}, {"url": "http://192.168.24.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "481cd150fbbd485aa76d113b5a3749bf"}, {"url": "http://192.168.24.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "485ca1edc1ff4ec2b596048f7ce2b758"}], "type": "image", "id": "dbf7da3ec4fa4d2fb80f6605ced454e1", "name": "glance"}, {"endpoints": [{"url": "http://192.168.24.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "03f1a42237aa4e3b94153ddf4ece48a7"}, {"url": "http://192.168.24.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "41a07725ebdb453ea5cc189d747aec08"}, {"url": "http://192.168.24.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "699a2598556b46fca25071b98aa0d519"}], "type": "identity", "id": "dbfd7492733148368f6658cb8b86ee98", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.24.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "062a1f6f2a8843fdabdcfdef99a05625"}, {"url": "http://192.168.24.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0c8588622e86486b99ba8aa11213e138"}, {"url": "http://192.168.24.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9a915ec5999c43028e244722a1aab7a4"}], "type": "baremetal-introspection", "id": "e5493fc1f9914421802bdac54c9657fd", "name": "ironic-inspector"}, {"endpoints": [{"url": "ws://192.168.24.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "34366ac2b1f64f2398a5f38bef7b507d"}, {"url": "ws://192.168.24.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4d9ec45cd18749caa1f4d32929bae046"}, {"url": "ws://192.168.24.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6a061669d91d42fe9db38bf18c7e22fa"}], "type": "messaging-websocket", "id": "e5ec63923f0b4784aeb79c52d0f23a58", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.24.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09631c5799b64490af6f5a48c3d43232"}, {"url": "http://192.168.24.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0f4171a2e74f424898c7fba209b1c2e8"}, {"url": "http://192.168.24.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a1dfc6fab96842889cb7cb6a6b443ba4"}], "type": "baremetal", "id": "ef6aaf16eaf140dcb901309f9548282b", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.24.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1b2cf765c34c4d5494f571113321dfd2"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2abb7a435bf0448cb2e215f7685eb05a"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "35dcc2a70587405e99d9c34cf5134ec8"}], "type": "compute", "id": "fdba5af5b3ee45fab371773c1c17352f", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "289972c402df4bb39da6ed2135e8f56d"}, "audit_ids": ["jkhxv3EZSo6rg2K5rvT9Vg"], "issued_at": "2017-06-23T15:23:49.000000Z"}}
run(Namespace(run_validations=False))
take_action(Namespace(run_validations=False))
This command is deprecated. Please use "openstack overcloud node introspect" to introspect manageable nodes instead.
Instantiating baremetal client: <class 'ironicclient.v1.client.Client'>
Baremetal API version: 1.9
Setting nodes for introspection to manageable...
Moving available/enroll nodes to manageable state.
Making authentication request to http://192.168.24.1:5000/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 8174
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "ccda3744034848d2a0da0ec5eadcd821", "name": "admin"}], "expires_at": "2017-06-23T19:23:49.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "55f170434da2433d8d5c498f51efcc11", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.24.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11cece1679ad4d78acb58c22c914f58b"}, {"url": "http://192.168.24.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba56a2fb27a74c62b14625e96e325786"}, {"url": "http://192.168.24.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "d9d1d844f69b4f33aa3db31a9b6fbda0"}], "type": "messaging", "id": "5d5df786da294c3497330070dce18792", "name": "zaqar"}, {"endpoints": [{"url": "http://192.168.24.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4894e60793ec4a45a479205750b6d547"}, {"url": "http://192.168.24.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "979b44ec318540b994c0c532320ffe85"}, {"url": "http://192.168.24.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "c9142a68eb704c3eb3e6a1113746063c"}], "type": "placement", "id": "6c8bb1d3d3ef42b6ac85c4bc71412e94", "name": "placement"}, {"endpoints": [{"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8eb5261e2d55476e83de82c22ee62600"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a6446773bef34ac286856107b69da81f"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef43b528dc5e411a867b8b4a0ffa23be"}], "type": "cloudformation", "id": "76432ddf4a904711b1d8205d9f55c118", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "12375bd744494f41be2bd864bac4c5d5"}, {"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a66d122477c64c02b70a413276452bed"}, {"url": "http://192.168.24.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f013ed60d58f48b2a2fe5c15df91fa08"}], "type": "object-store", "id": "7da80e7fc8b44dc4b302dbf3bd4cf12c", "name": "swift"}, {"endpoints": [{"url": "http://192.168.24.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "16e55c12241f4ac18e896db4e1a61649"}, {"url": "http://192.168.24.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2e96fdd236eb40d0ae96603bd5792420"}, {"url": "http://192.168.24.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "93753cd9c4df4705bc9d1400918434e8"}], "type": "network", "id": "9bb9c0cd038241228a0a5c450343cbbd", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.24.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4348e9dbbcc14972b6dc1e1bd71d873c"}, {"url": "http://192.168.24.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59659611cdcc4a1a9a04205e8c99acfd"}, {"url": "http://192.168.24.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "59942a334d8541daa79b0ad68b6c186c"}], "type": "workflowv2", "id": "af9164fadde348588d7b5cd0e9ac30ad", "name": "mistral"}, {"endpoints": [{"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "927f45b99634430e8d5da5645313d838"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bd4266a54f9d456da85c11ca6dd24674"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ed92f2fce02442cbb26d18c09b6413ec"}], "type": "orchestration", "id": "c5040fa43d7d409bacb81c808b5d3d3e", "name": "heat"}, {"endpoints": [{"url": "http://192.168.24.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "37763251b4724be5bc2d16ac45caadc0"}, {"url": "http://192.168.24.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "481cd150fbbd485aa76d113b5a3749bf"}, {"url": "http://192.168.24.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "485ca1edc1ff4ec2b596048f7ce2b758"}], "type": "image", "id": "dbf7da3ec4fa4d2fb80f6605ced454e1", "name": "glance"}, {"endpoints": [{"url": "http://192.168.24.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "03f1a42237aa4e3b94153ddf4ece48a7"}, {"url": "http://192.168.24.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "41a07725ebdb453ea5cc189d747aec08"}, {"url": "http://192.168.24.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "699a2598556b46fca25071b98aa0d519"}], "type": "identity", "id": "dbfd7492733148368f6658cb8b86ee98", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.24.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "062a1f6f2a8843fdabdcfdef99a05625"}, {"url": "http://192.168.24.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0c8588622e86486b99ba8aa11213e138"}, {"url": "http://192.168.24.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9a915ec5999c43028e244722a1aab7a4"}], "type": "baremetal-introspection", "id": "e5493fc1f9914421802bdac54c9657fd", "name": "ironic-inspector"}, {"endpoints": [{"url": "ws://192.168.24.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "34366ac2b1f64f2398a5f38bef7b507d"}, {"url": "ws://192.168.24.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4d9ec45cd18749caa1f4d32929bae046"}, {"url": "ws://192.168.24.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6a061669d91d42fe9db38bf18c7e22fa"}], "type": "messaging-websocket", "id": "e5ec63923f0b4784aeb79c52d0f23a58", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.24.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09631c5799b64490af6f5a48c3d43232"}, {"url": "http://192.168.24.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0f4171a2e74f424898c7fba209b1c2e8"}, {"url": "http://192.168.24.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a1dfc6fab96842889cb7cb6a6b443ba4"}], "type": "baremetal", "id": "ef6aaf16eaf140dcb901309f9548282b", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.24.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1b2cf765c34c4d5494f571113321dfd2"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2abb7a435bf0448cb2e215f7685eb05a"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "35dcc2a70587405e99d9c34cf5134ec8"}], "type": "compute", "id": "fdba5af5b3ee45fab371773c1c17352f", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "289972c402df4bb39da6ed2135e8f56d"}, "audit_ids": ["Q44za928Q2K67unMn8cctA"], "issued_at": "2017-06-23T15:23:49.000000Z"}}
REQ: curl -g -i -X GET http://192.168.24.1:6385/v1/nodes/?associated=False&maintenance=False -H "X-OpenStack-Ironic-API-Version: 1.29" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}0432ffe14c6eb9a3dcc9bf378bfa0a6ca7899b08"
Starting new HTTP connection (1): 192.168.24.1
"GET /v1/nodes/?associated=False&maintenance=False HTTP/1.1" 200 507
RESP: [200] Date: Fri, 23 Jun 2017 15:23:49 GMT Server: Apache X-OpenStack-Ironic-API-Minimum-Version: 1.1 X-OpenStack-Ironic-API-Maximum-Version: 1.31 X-OpenStack-Ironic-API-Version: 1.29 Openstack-Request-Id: req-6ae9f7a7-c92f-49b6-93ac-24fecb2716ba Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 507 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json 
RESP BODY: {"nodes": [{"instance_uuid": null, "uuid": "ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-0"}, {"instance_uuid": null, "uuid": "abbf94c4-be5f-462a-b75f-9b0a2758d519", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/abbf94c4-be5f-462a-b75f-9b0a2758d519", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/abbf94c4-be5f-462a-b75f-9b0a2758d519", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-1"}, {"instance_uuid": null, "uuid": "d912eed1-3c9e-4be6-b5fb-b95f83165f74", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/d912eed1-3c9e-4be6-b5fb-b95f83165f74", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/d912eed1-3c9e-4be6-b5fb-b95f83165f74", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-2"}, {"instance_uuid": null, "uuid": "cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "compute-0"}, {"instance_uuid": null, "uuid": "c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power on", "name": "compute-1"}, {"instance_uuid": null, "uuid": "c2436eb7-0031-4570-bd95-4de5a24d7784", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/c2436eb7-0031-4570-bd95-4de5a24d7784", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/c2436eb7-0031-4570-bd95-4de5a24d7784", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-0"}, {"instance_uuid": null, "uuid": "312dc5b7-c823-4ab6-8bee-bf3b286dade1", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/312dc5b7-c823-4ab6-8bee-bf3b286dade1", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/312dc5b7-c823-4ab6-8bee-bf3b286dade1", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-1"}, {"instance_uuid": null, "uuid": "e87a10d6-7668-4a93-be84-87673e458816", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/e87a10d6-7668-4a93-be84-87673e458816", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/e87a10d6-7668-4a93-be84-87673e458816", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-2"}]}

Starting introspection of manageable nodes
Instantiating workflow engine client: <class 'mistralclient.api.v2.client.Client'>
REQ: curl -g -i -X GET http://192.168.24.1:5000/ -H "User-Agent: osc-lib/1.6.0 keystoneauth1/2.20.0 python-requests/2.11.1 CPython/2.7.5" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}0432ffe14c6eb9a3dcc9bf378bfa0a6ca7899b08"
"GET / HTTP/1.1" 300 598
RESP: [300] Date: Fri, 23 Jun 2017 15:23:50 GMT Server: Apache Vary: X-Auth-Token Content-Length: 598 Keep-Alive: timeout=15, max=97 Connection: Keep-Alive Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.24.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.24.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Waiting for introspection to finish...
Instantiating messaging websocket client: ws://192.168.24.1:9000
{u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 97, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 246, in take_action
    queue_name=queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 150, in introspect_manageable_nodes
    with tripleoclients.messaging_websocket(queue_name) as ws:
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 200, in messaging_websocket
    return WebsocketClient(self._instance, queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 97, in __init__
    self.send('queue_create', {'queue_name': queue_name})
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 123, in send
    raise RuntimeError(data)
RuntimeError: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}
clean_up StartBaremetalIntrospectionBulk: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 97, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 246, in take_action
    queue_name=queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 150, in introspect_manageable_nodes
    with tripleoclients.messaging_websocket(queue_name) as ws:
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 200, in messaging_websocket
    return WebsocketClient(self._instance, queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 97, in __init__
    self.send('queue_create', {'queue_name': queue_name})
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 123, in send
    raise RuntimeError(data)
RuntimeError: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}

END return value: 1



-----------------


This looks like it should not be ironic specific, please feel free to change the component

Comment 2 Dmitry Tantsur 2017-06-26 15:05:47 UTC

It's not ironic-specific indeed, and these commands are not even provided by ironic. And I've seen it with commands that do not involve ironic. Moving to tripleoclient for triaging.

Comment 3 Emilien Macchi 2017-06-26 15:09:50 UTC

What is the status of MySQL database ? Can you access to it? can you provide more logs (sosreport maybe) - I want to see if you're having memory or storage issues (maybe some crontab doesn't work on the undercloud?).

Comment 5 Emilien Macchi 2017-07-05 17:33:40 UTC

are you running telemetry services? if not, maybe https://bugs.launchpad.net/tripleo/+bug/1701357 is related to this bug.

Comment 7 Dan Yasny 2017-07-11 08:18:11 UTC

I do not see any telemetry services in the endpoint list on the undercloud, and overcloud isn't deployed yet

Comment 10 Thomas Hervé 2017-09-28 10:47:47 UTC

It should be fixed now.

Comment 14 Dan Yasny 2017-12-05 14:04:41 UTC

Couldn't reproduce on OSP12

Comment 17 errata-xmlrpc 2017-12-13 21:33:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.