Bug 1464247 - MySQL server has gone away error reported when issuing an ironic command after the setup has been idle for a while
MySQL server has gone away error reported when issuing an ironic command afte...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-zaqar (Show other bugs)
12.0 (Pike)
x86_64 Linux
medium Severity medium
: rc
: 12.0 (Pike)
Assigned To: Thomas Hervé
Ronnie Rasouli
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-22 15:24 EDT by Dan Yasny
Modified: 2018-02-05 14:07 EST (History)
17 users (show)

See Also:
Fixed In Version: openstack-zaqar-5.0.1-0.20170905222047.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:33:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1691951 None None None 2017-09-28 06:46 EDT
OpenStack gerrit 473612 None None None 2017-09-28 06:46 EDT
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-15 20:43:25 EST

  None (edit)
Description Dan Yasny 2017-06-22 15:24:56 EDT
Description of problem:
This has already happened several times, but seems to be hard to reproduce consistently.

A setup left alone for a long time (overnight tipycally) will fail to respond to an ironic related command producing a MySQL error.

A second attempt at the same command usually works.

Simple API queries (e.g. ironic node-list) also work.

Examples:
(openstack) baremetal introspection bulk start
This command is deprecated. Please use "openstack overcloud node introspect" to introspect manageable nodes instead.
Setting nodes for introspection to manageable...
Starting introspection of manageable nodes
Waiting for introspection to finish...
{u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'d97b577e-3812-4998-9afb-4710cd33bd37\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'd97b577e-3812-4998-9afb-4710cd33bd37'}, u'api': u'v2', u'headers': {u'Client-ID': u'e8c19088-723a-4621-b945-719155daa3bd', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}


 openstack baremetal import --json /home/stack/instackenv.json
This command is deprecated. Please use "openstack overcloud node import" to register nodes instead.
{u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'8f6d7d3087824450ade39c5854e7b7d1\', \'name\': u\'52cd9312-e11e-49f8-bb75-07859bacdd58\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'52cd9312-e11e-49f8-bb75-07859bacdd58'}, u'api': u'v2', u'headers': {u'Client-ID': u'88d661b5-c0d1-4887-85ea-21871ad0d836', u'X-Project-ID': u'8f6d7d3087824450ade39c5854e7b7d1'}}}

Version-Release number of selected component (if applicable):

puppet-ironic-11.2.0-0.20170611220351.0fa87ae.el7ost.noarch
python-ironic-lib-2.7.0-0.20170331023537.8d899bd.el7ost.noarch
python-ironicclient-1.13.0-0.20170512184144.7cceefe.el7ost.noarch
python-ironic-inspector-client-1.12.0-0.20170331023022.045fd9e.el7ost.noarch
python-PyMySQL-0.7.9-2.el7ost.noarch
openstack-nova-compute-16.0.0-0.20170609171025.el7ost.noarch
openstack-ironic-conductor-8.0.1-0.20170610002959.adf3461.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tempest-16.0.1-0.20170612053404.c0e31f1.el7ost.noarch
openstack-tripleo-common-containers-7.1.1-0.20170612015630.414f94b.el7ost.noarch
openstack-tripleo-ui-7.1.1-0.20170609131448.202695f.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170609171025.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170611233361.el7ost.noarch
openstack-swift-object-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-swift-proxy-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170612043222.a289f55.el7ost.noarch
python-openstackclient-3.11.0-0.20170505192416.el7ost.noarch
openstack-mistral-api-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-selinux-0.8.7-1.el7ost.noarch
openstack-keystone-12.0.0-0.20170602202335.8628235.el7ost.noarch
openstack-tripleo-image-elements-7.0.0-0.20170607161959.401d861.el7ost.noarch
puppet-openstack_extras-11.2.0-0.20170609110653.d9f8351.el7ost.noarch
openstack-neutron-common-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-ironic-inspector-5.1.1-0.20170608115120.a46e708.el7ost.noarch
python-openstacksdk-0.9.16-0.20170608090447.bfc88e7.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170605222006.25e08b3.el7ost.noarch
puppet-openstacklib-11.2.0-0.20170609110906.a99d5dc.el7ost.noarch
openstack-tripleo-common-7.1.1-0.20170612015630.414f94b.el7ost.noarch
openstack-swift-account-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-heat-common-9.0.0-0.20170612043222.a289f55.el7ost.noarch
openstack-ironic-api-8.0.1-0.20170610002959.adf3461.el7ost.noarch
openstack-mistral-executor-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-tripleo-validations-7.1.1-0.20170609083517.e120452.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170609171025.el7ost.noarch
openstack-glance-15.0.0-0.20170609191014.61046c8.el7ost.noarch
openstack-nova-common-16.0.0-0.20170609171025.el7ost.noarch
openstack-heat-api-9.0.0-0.20170612043222.a289f55.el7ost.noarch
python-openstack-mistral-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-mistral-common-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-zaqar-5.0.0-0.20170609114413.51299ba.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170609171025.el7ost.noarch
openstack-swift-container-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch
openstack-neutron-11.0.0-0.20170611190934.01cc269.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170612043222.a289f55.el7ost.noarch
openstack-ironic-common-8.0.1-0.20170610002959.adf3461.el7ost.noarch
openstack-mistral-engine-5.0.0-0.20170607103828.8d51753.el7ost.noarch
openstack-nova-api-16.0.0-0.20170609171025.el7ost.noarch
mariadb-5.5.56-2.el7.x86_64
mariadb-server-5.5.56-2.el7.x86_64
mariadb-libs-5.5.56-2.el7.x86_64

How reproducible:
hard to reproduce since it take a very long idle period

Steps to Reproduce:
1. install undercloud
2. leave it for a long time (~12 hours seems to work)
3. try to run one of the above commands (there may be others)

Actual results:
see the failures above

Expected results:
should always work

Additional info:
leaving a setup idle now so I can rerun this with --debug tomorrow
Comment 1 Dan Yasny 2017-06-23 11:27:52 EDT
(undercloud) [stack@undercloud-0 ~]$ openstack --debug baremetal introspection bulk start
START with options: [u'--debug', u'baremetal', u'introspection', u'bulk', u'start']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', auth_type='password', auth_url='http://192.168.24.1:5000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, old_profile=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.29', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', roles='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
orchestration API version 1, cmd group openstack.orchestration.v1
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
key_manager API version 1, cmd group openstack.key_manager.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
baremetal API version 1.29, cmd group openstack.baremetal.v1
messaging API version 2, cmd group openstack.messaging.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
dns API version 2, cmd group openstack.dns.v2
data_processing API version 1.1, cmd group openstack.data_processing.v1
database API version 1, cmd group openstack.database.v1
container_infra API version 1, cmd group openstack.container_infra.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
command: baremetal introspection bulk start -> tripleoclient.v1.baremetal.StartBaremetalIntrospectionBulk (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.24.1:5000/', 'password': '***', 'project_domain_name': 'Default'}
Get auth_ref
REQ: curl -g -i -X GET http://192.168.24.1:5000/ -H "Accept: application/json" -H "User-Agent: osc-lib/1.6.0 keystoneauth1/2.20.0 python-requests/2.11.1 CPython/2.7.5"
Starting new HTTP connection (1): 192.168.24.1
"GET / HTTP/1.1" 300 598
RESP: [300] Date: Fri, 23 Jun 2017 15:23:49 GMT Server: Apache Vary: X-Auth-Token Content-Length: 598 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.24.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.24.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Making authentication request to http://192.168.24.1:5000/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 8174
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "ccda3744034848d2a0da0ec5eadcd821", "name": "admin"}], "expires_at": "2017-06-23T19:23:49.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "55f170434da2433d8d5c498f51efcc11", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.24.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11cece1679ad4d78acb58c22c914f58b"}, {"url": "http://192.168.24.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba56a2fb27a74c62b14625e96e325786"}, {"url": "http://192.168.24.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "d9d1d844f69b4f33aa3db31a9b6fbda0"}], "type": "messaging", "id": "5d5df786da294c3497330070dce18792", "name": "zaqar"}, {"endpoints": [{"url": "http://192.168.24.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4894e60793ec4a45a479205750b6d547"}, {"url": "http://192.168.24.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "979b44ec318540b994c0c532320ffe85"}, {"url": "http://192.168.24.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "c9142a68eb704c3eb3e6a1113746063c"}], "type": "placement", "id": "6c8bb1d3d3ef42b6ac85c4bc71412e94", "name": "placement"}, {"endpoints": [{"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8eb5261e2d55476e83de82c22ee62600"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a6446773bef34ac286856107b69da81f"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef43b528dc5e411a867b8b4a0ffa23be"}], "type": "cloudformation", "id": "76432ddf4a904711b1d8205d9f55c118", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "12375bd744494f41be2bd864bac4c5d5"}, {"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a66d122477c64c02b70a413276452bed"}, {"url": "http://192.168.24.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f013ed60d58f48b2a2fe5c15df91fa08"}], "type": "object-store", "id": "7da80e7fc8b44dc4b302dbf3bd4cf12c", "name": "swift"}, {"endpoints": [{"url": "http://192.168.24.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "16e55c12241f4ac18e896db4e1a61649"}, {"url": "http://192.168.24.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2e96fdd236eb40d0ae96603bd5792420"}, {"url": "http://192.168.24.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "93753cd9c4df4705bc9d1400918434e8"}], "type": "network", "id": "9bb9c0cd038241228a0a5c450343cbbd", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.24.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4348e9dbbcc14972b6dc1e1bd71d873c"}, {"url": "http://192.168.24.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59659611cdcc4a1a9a04205e8c99acfd"}, {"url": "http://192.168.24.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "59942a334d8541daa79b0ad68b6c186c"}], "type": "workflowv2", "id": "af9164fadde348588d7b5cd0e9ac30ad", "name": "mistral"}, {"endpoints": [{"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "927f45b99634430e8d5da5645313d838"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bd4266a54f9d456da85c11ca6dd24674"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ed92f2fce02442cbb26d18c09b6413ec"}], "type": "orchestration", "id": "c5040fa43d7d409bacb81c808b5d3d3e", "name": "heat"}, {"endpoints": [{"url": "http://192.168.24.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "37763251b4724be5bc2d16ac45caadc0"}, {"url": "http://192.168.24.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "481cd150fbbd485aa76d113b5a3749bf"}, {"url": "http://192.168.24.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "485ca1edc1ff4ec2b596048f7ce2b758"}], "type": "image", "id": "dbf7da3ec4fa4d2fb80f6605ced454e1", "name": "glance"}, {"endpoints": [{"url": "http://192.168.24.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "03f1a42237aa4e3b94153ddf4ece48a7"}, {"url": "http://192.168.24.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "41a07725ebdb453ea5cc189d747aec08"}, {"url": "http://192.168.24.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "699a2598556b46fca25071b98aa0d519"}], "type": "identity", "id": "dbfd7492733148368f6658cb8b86ee98", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.24.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "062a1f6f2a8843fdabdcfdef99a05625"}, {"url": "http://192.168.24.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0c8588622e86486b99ba8aa11213e138"}, {"url": "http://192.168.24.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9a915ec5999c43028e244722a1aab7a4"}], "type": "baremetal-introspection", "id": "e5493fc1f9914421802bdac54c9657fd", "name": "ironic-inspector"}, {"endpoints": [{"url": "ws://192.168.24.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "34366ac2b1f64f2398a5f38bef7b507d"}, {"url": "ws://192.168.24.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4d9ec45cd18749caa1f4d32929bae046"}, {"url": "ws://192.168.24.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6a061669d91d42fe9db38bf18c7e22fa"}], "type": "messaging-websocket", "id": "e5ec63923f0b4784aeb79c52d0f23a58", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.24.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09631c5799b64490af6f5a48c3d43232"}, {"url": "http://192.168.24.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0f4171a2e74f424898c7fba209b1c2e8"}, {"url": "http://192.168.24.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a1dfc6fab96842889cb7cb6a6b443ba4"}], "type": "baremetal", "id": "ef6aaf16eaf140dcb901309f9548282b", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.24.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1b2cf765c34c4d5494f571113321dfd2"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2abb7a435bf0448cb2e215f7685eb05a"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "35dcc2a70587405e99d9c34cf5134ec8"}], "type": "compute", "id": "fdba5af5b3ee45fab371773c1c17352f", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "289972c402df4bb39da6ed2135e8f56d"}, "audit_ids": ["jkhxv3EZSo6rg2K5rvT9Vg"], "issued_at": "2017-06-23T15:23:49.000000Z"}}
run(Namespace(run_validations=False))
take_action(Namespace(run_validations=False))
This command is deprecated. Please use "openstack overcloud node introspect" to introspect manageable nodes instead.
Instantiating baremetal client: <class 'ironicclient.v1.client.Client'>
Baremetal API version: 1.9
Setting nodes for introspection to manageable...
Moving available/enroll nodes to manageable state.
Making authentication request to http://192.168.24.1:5000/v3/auth/tokens
"POST /v3/auth/tokens HTTP/1.1" 201 8174
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "ccda3744034848d2a0da0ec5eadcd821", "name": "admin"}], "expires_at": "2017-06-23T19:23:49.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "55f170434da2433d8d5c498f51efcc11", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.24.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11cece1679ad4d78acb58c22c914f58b"}, {"url": "http://192.168.24.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba56a2fb27a74c62b14625e96e325786"}, {"url": "http://192.168.24.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "d9d1d844f69b4f33aa3db31a9b6fbda0"}], "type": "messaging", "id": "5d5df786da294c3497330070dce18792", "name": "zaqar"}, {"endpoints": [{"url": "http://192.168.24.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4894e60793ec4a45a479205750b6d547"}, {"url": "http://192.168.24.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "979b44ec318540b994c0c532320ffe85"}, {"url": "http://192.168.24.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "c9142a68eb704c3eb3e6a1113746063c"}], "type": "placement", "id": "6c8bb1d3d3ef42b6ac85c4bc71412e94", "name": "placement"}, {"endpoints": [{"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8eb5261e2d55476e83de82c22ee62600"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a6446773bef34ac286856107b69da81f"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef43b528dc5e411a867b8b4a0ffa23be"}], "type": "cloudformation", "id": "76432ddf4a904711b1d8205d9f55c118", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "12375bd744494f41be2bd864bac4c5d5"}, {"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a66d122477c64c02b70a413276452bed"}, {"url": "http://192.168.24.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f013ed60d58f48b2a2fe5c15df91fa08"}], "type": "object-store", "id": "7da80e7fc8b44dc4b302dbf3bd4cf12c", "name": "swift"}, {"endpoints": [{"url": "http://192.168.24.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "16e55c12241f4ac18e896db4e1a61649"}, {"url": "http://192.168.24.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2e96fdd236eb40d0ae96603bd5792420"}, {"url": "http://192.168.24.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "93753cd9c4df4705bc9d1400918434e8"}], "type": "network", "id": "9bb9c0cd038241228a0a5c450343cbbd", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.24.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4348e9dbbcc14972b6dc1e1bd71d873c"}, {"url": "http://192.168.24.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59659611cdcc4a1a9a04205e8c99acfd"}, {"url": "http://192.168.24.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "59942a334d8541daa79b0ad68b6c186c"}], "type": "workflowv2", "id": "af9164fadde348588d7b5cd0e9ac30ad", "name": "mistral"}, {"endpoints": [{"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "927f45b99634430e8d5da5645313d838"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bd4266a54f9d456da85c11ca6dd24674"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ed92f2fce02442cbb26d18c09b6413ec"}], "type": "orchestration", "id": "c5040fa43d7d409bacb81c808b5d3d3e", "name": "heat"}, {"endpoints": [{"url": "http://192.168.24.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "37763251b4724be5bc2d16ac45caadc0"}, {"url": "http://192.168.24.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "481cd150fbbd485aa76d113b5a3749bf"}, {"url": "http://192.168.24.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "485ca1edc1ff4ec2b596048f7ce2b758"}], "type": "image", "id": "dbf7da3ec4fa4d2fb80f6605ced454e1", "name": "glance"}, {"endpoints": [{"url": "http://192.168.24.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "03f1a42237aa4e3b94153ddf4ece48a7"}, {"url": "http://192.168.24.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "41a07725ebdb453ea5cc189d747aec08"}, {"url": "http://192.168.24.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "699a2598556b46fca25071b98aa0d519"}], "type": "identity", "id": "dbfd7492733148368f6658cb8b86ee98", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.24.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "062a1f6f2a8843fdabdcfdef99a05625"}, {"url": "http://192.168.24.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0c8588622e86486b99ba8aa11213e138"}, {"url": "http://192.168.24.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9a915ec5999c43028e244722a1aab7a4"}], "type": "baremetal-introspection", "id": "e5493fc1f9914421802bdac54c9657fd", "name": "ironic-inspector"}, {"endpoints": [{"url": "ws://192.168.24.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "34366ac2b1f64f2398a5f38bef7b507d"}, {"url": "ws://192.168.24.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4d9ec45cd18749caa1f4d32929bae046"}, {"url": "ws://192.168.24.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6a061669d91d42fe9db38bf18c7e22fa"}], "type": "messaging-websocket", "id": "e5ec63923f0b4784aeb79c52d0f23a58", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.24.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09631c5799b64490af6f5a48c3d43232"}, {"url": "http://192.168.24.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0f4171a2e74f424898c7fba209b1c2e8"}, {"url": "http://192.168.24.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a1dfc6fab96842889cb7cb6a6b443ba4"}], "type": "baremetal", "id": "ef6aaf16eaf140dcb901309f9548282b", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.24.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1b2cf765c34c4d5494f571113321dfd2"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2abb7a435bf0448cb2e215f7685eb05a"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "35dcc2a70587405e99d9c34cf5134ec8"}], "type": "compute", "id": "fdba5af5b3ee45fab371773c1c17352f", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "289972c402df4bb39da6ed2135e8f56d"}, "audit_ids": ["Q44za928Q2K67unMn8cctA"], "issued_at": "2017-06-23T15:23:49.000000Z"}}
REQ: curl -g -i -X GET http://192.168.24.1:6385/v1/nodes/?associated=False&maintenance=False -H "X-OpenStack-Ironic-API-Version: 1.29" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}0432ffe14c6eb9a3dcc9bf378bfa0a6ca7899b08"
Starting new HTTP connection (1): 192.168.24.1
"GET /v1/nodes/?associated=False&maintenance=False HTTP/1.1" 200 507
RESP: [200] Date: Fri, 23 Jun 2017 15:23:49 GMT Server: Apache X-OpenStack-Ironic-API-Minimum-Version: 1.1 X-OpenStack-Ironic-API-Maximum-Version: 1.31 X-OpenStack-Ironic-API-Version: 1.29 Openstack-Request-Id: req-6ae9f7a7-c92f-49b6-93ac-24fecb2716ba Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 507 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json 
RESP BODY: {"nodes": [{"instance_uuid": null, "uuid": "ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-0"}, {"instance_uuid": null, "uuid": "abbf94c4-be5f-462a-b75f-9b0a2758d519", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/abbf94c4-be5f-462a-b75f-9b0a2758d519", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/abbf94c4-be5f-462a-b75f-9b0a2758d519", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-1"}, {"instance_uuid": null, "uuid": "d912eed1-3c9e-4be6-b5fb-b95f83165f74", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/d912eed1-3c9e-4be6-b5fb-b95f83165f74", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/d912eed1-3c9e-4be6-b5fb-b95f83165f74", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-2"}, {"instance_uuid": null, "uuid": "cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "compute-0"}, {"instance_uuid": null, "uuid": "c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power on", "name": "compute-1"}, {"instance_uuid": null, "uuid": "c2436eb7-0031-4570-bd95-4de5a24d7784", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/c2436eb7-0031-4570-bd95-4de5a24d7784", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/c2436eb7-0031-4570-bd95-4de5a24d7784", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-0"}, {"instance_uuid": null, "uuid": "312dc5b7-c823-4ab6-8bee-bf3b286dade1", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/312dc5b7-c823-4ab6-8bee-bf3b286dade1", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/312dc5b7-c823-4ab6-8bee-bf3b286dade1", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-1"}, {"instance_uuid": null, "uuid": "e87a10d6-7668-4a93-be84-87673e458816", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/e87a10d6-7668-4a93-be84-87673e458816", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/e87a10d6-7668-4a93-be84-87673e458816", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-2"}]}

Starting introspection of manageable nodes
Instantiating workflow engine client: <class 'mistralclient.api.v2.client.Client'>
REQ: curl -g -i -X GET http://192.168.24.1:5000/ -H "User-Agent: osc-lib/1.6.0 keystoneauth1/2.20.0 python-requests/2.11.1 CPython/2.7.5" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}0432ffe14c6eb9a3dcc9bf378bfa0a6ca7899b08"
"GET / HTTP/1.1" 300 598
RESP: [300] Date: Fri, 23 Jun 2017 15:23:50 GMT Server: Apache Vary: X-Auth-Token Content-Length: 598 Keep-Alive: timeout=15, max=97 Connection: Keep-Alive Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.24.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.24.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Waiting for introspection to finish...
Instantiating messaging websocket client: ws://192.168.24.1:9000
{u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 97, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 246, in take_action
    queue_name=queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 150, in introspect_manageable_nodes
    with tripleoclients.messaging_websocket(queue_name) as ws:
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 200, in messaging_websocket
    return WebsocketClient(self._instance, queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 97, in __init__
    self.send('queue_create', {'queue_name': queue_name})
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 123, in send
    raise RuntimeError(data)
RuntimeError: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}
clean_up StartBaremetalIntrospectionBulk: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 97, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 246, in take_action
    queue_name=queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 150, in introspect_manageable_nodes
    with tripleoclients.messaging_websocket(queue_name) as ws:
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 200, in messaging_websocket
    return WebsocketClient(self._instance, queue_name)
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 97, in __init__
    self.send('queue_create', {'queue_name': queue_name})
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 123, in send
    raise RuntimeError(data)
RuntimeError: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}}

END return value: 1



-----------------


This looks like it should not be ironic specific, please feel free to change the component
Comment 2 Dmitry Tantsur 2017-06-26 11:05:47 EDT
It's not ironic-specific indeed, and these commands are not even provided by ironic. And I've seen it with commands that do not involve ironic. Moving to tripleoclient for triaging.
Comment 3 Emilien Macchi 2017-06-26 11:09:50 EDT
What is the status of MySQL database ? Can you access to it? can you provide more logs (sosreport maybe) - I want to see if you're having memory or storage issues (maybe some crontab doesn't work on the undercloud?).
Comment 5 Emilien Macchi 2017-07-05 13:33:40 EDT
are you running telemetry services? if not, maybe https://bugs.launchpad.net/tripleo/+bug/1701357 is related to this bug.
Comment 7 Dan Yasny 2017-07-11 04:18:11 EDT
I do not see any telemetry services in the endpoint list on the undercloud, and overcloud isn't deployed yet
Comment 10 Thomas Hervé 2017-09-28 06:47:47 EDT
It should be fixed now.
Comment 14 Dan Yasny 2017-12-05 09:04:41 EST
Couldn't reproduce on OSP12
Comment 17 errata-xmlrpc 2017-12-13 16:33:29 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.