Bug 1464247
| Summary: | MySQL server has gone away error reported when issuing an ironic command after the setup has been idle for a while | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Dan Yasny <dyasny> |
| Component: | openstack-zaqar | Assignee: | Thomas Hervé <therve> |
| Status: | CLOSED ERRATA | QA Contact: | Ronnie Rasouli <rrasouli> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 12.0 (Pike) | CC: | apevec, aschultz, dyasny, emacchi, hbrock, jschluet, jslagle, lhh, mburns, mlammon, nlevinki, rbrady, rhel-osp-director-maint, rrasouli, srevivo, therve, tvignaud |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-zaqar-5.0.1-0.20170905222047.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 21:33:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(undercloud) [stack@undercloud-0 ~]$ openstack --debug baremetal introspection bulk start START with options: [u'--debug', u'baremetal', u'introspection', u'bulk', u'start'] options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', auth_type='password', auth_url='http://192.168.24.1:5000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, old_profile=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.29', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', roles='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} compute API version 2, cmd group openstack.compute.v2 network API version 2, cmd group openstack.network.v2 image API version 2, cmd group openstack.image.v2 volume API version 2, cmd group openstack.volume.v2 identity API version 3, cmd group openstack.identity.v3 object_store API version 1, cmd group openstack.object_store.v1 neutronclient API version 2, cmd group openstack.neutronclient.v2 orchestration API version 1, cmd group openstack.orchestration.v1 metric API version 1, cmd group openstack.metric.v1 alarming API version 2, cmd group openstack.alarming.v2 key_manager API version 1, cmd group openstack.key_manager.v1 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 baremetal API version 1.29, cmd group openstack.baremetal.v1 messaging API version 2, cmd group openstack.messaging.v2 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 tripleoclient API version 1, cmd group openstack.tripleoclient.v1 dns API version 2, cmd group openstack.dns.v2 data_processing API version 1.1, cmd group openstack.data_processing.v1 database API version 1, cmd group openstack.database.v1 container_infra API version 1, cmd group openstack.container_infra.v1 Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} command: baremetal introspection bulk start -> tripleoclient.v1.baremetal.StartBaremetalIntrospectionBulk (auth=True) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.168.24.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.29', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} Using auth plugin: password Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.24.1:5000/', 'password': '***', 'project_domain_name': 'Default'} Get auth_ref REQ: curl -g -i -X GET http://192.168.24.1:5000/ -H "Accept: application/json" -H "User-Agent: osc-lib/1.6.0 keystoneauth1/2.20.0 python-requests/2.11.1 CPython/2.7.5" Starting new HTTP connection (1): 192.168.24.1 "GET / HTTP/1.1" 300 598 RESP: [300] Date: Fri, 23 Jun 2017 15:23:49 GMT Server: Apache Vary: X-Auth-Token Content-Length: 598 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.24.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.24.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}} Making authentication request to http://192.168.24.1:5000/v3/auth/tokens "POST /v3/auth/tokens HTTP/1.1" 201 8174 {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "ccda3744034848d2a0da0ec5eadcd821", "name": "admin"}], "expires_at": "2017-06-23T19:23:49.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "55f170434da2433d8d5c498f51efcc11", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.24.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11cece1679ad4d78acb58c22c914f58b"}, {"url": "http://192.168.24.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba56a2fb27a74c62b14625e96e325786"}, {"url": "http://192.168.24.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "d9d1d844f69b4f33aa3db31a9b6fbda0"}], "type": "messaging", "id": "5d5df786da294c3497330070dce18792", "name": "zaqar"}, {"endpoints": [{"url": "http://192.168.24.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4894e60793ec4a45a479205750b6d547"}, {"url": "http://192.168.24.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "979b44ec318540b994c0c532320ffe85"}, {"url": "http://192.168.24.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "c9142a68eb704c3eb3e6a1113746063c"}], "type": "placement", "id": "6c8bb1d3d3ef42b6ac85c4bc71412e94", "name": "placement"}, {"endpoints": [{"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8eb5261e2d55476e83de82c22ee62600"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a6446773bef34ac286856107b69da81f"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef43b528dc5e411a867b8b4a0ffa23be"}], "type": "cloudformation", "id": "76432ddf4a904711b1d8205d9f55c118", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "12375bd744494f41be2bd864bac4c5d5"}, {"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a66d122477c64c02b70a413276452bed"}, {"url": "http://192.168.24.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f013ed60d58f48b2a2fe5c15df91fa08"}], "type": "object-store", "id": "7da80e7fc8b44dc4b302dbf3bd4cf12c", "name": "swift"}, {"endpoints": [{"url": "http://192.168.24.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "16e55c12241f4ac18e896db4e1a61649"}, {"url": "http://192.168.24.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2e96fdd236eb40d0ae96603bd5792420"}, {"url": "http://192.168.24.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "93753cd9c4df4705bc9d1400918434e8"}], "type": "network", "id": "9bb9c0cd038241228a0a5c450343cbbd", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.24.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4348e9dbbcc14972b6dc1e1bd71d873c"}, {"url": "http://192.168.24.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59659611cdcc4a1a9a04205e8c99acfd"}, {"url": "http://192.168.24.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "59942a334d8541daa79b0ad68b6c186c"}], "type": "workflowv2", "id": "af9164fadde348588d7b5cd0e9ac30ad", "name": "mistral"}, {"endpoints": [{"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "927f45b99634430e8d5da5645313d838"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bd4266a54f9d456da85c11ca6dd24674"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ed92f2fce02442cbb26d18c09b6413ec"}], "type": "orchestration", "id": "c5040fa43d7d409bacb81c808b5d3d3e", "name": "heat"}, {"endpoints": [{"url": "http://192.168.24.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "37763251b4724be5bc2d16ac45caadc0"}, {"url": "http://192.168.24.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "481cd150fbbd485aa76d113b5a3749bf"}, {"url": "http://192.168.24.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "485ca1edc1ff4ec2b596048f7ce2b758"}], "type": "image", "id": "dbf7da3ec4fa4d2fb80f6605ced454e1", "name": "glance"}, {"endpoints": [{"url": "http://192.168.24.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "03f1a42237aa4e3b94153ddf4ece48a7"}, {"url": "http://192.168.24.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "41a07725ebdb453ea5cc189d747aec08"}, {"url": "http://192.168.24.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "699a2598556b46fca25071b98aa0d519"}], "type": "identity", "id": "dbfd7492733148368f6658cb8b86ee98", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.24.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "062a1f6f2a8843fdabdcfdef99a05625"}, {"url": "http://192.168.24.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0c8588622e86486b99ba8aa11213e138"}, {"url": "http://192.168.24.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9a915ec5999c43028e244722a1aab7a4"}], "type": "baremetal-introspection", "id": "e5493fc1f9914421802bdac54c9657fd", "name": "ironic-inspector"}, {"endpoints": [{"url": "ws://192.168.24.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "34366ac2b1f64f2398a5f38bef7b507d"}, {"url": "ws://192.168.24.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4d9ec45cd18749caa1f4d32929bae046"}, {"url": "ws://192.168.24.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6a061669d91d42fe9db38bf18c7e22fa"}], "type": "messaging-websocket", "id": "e5ec63923f0b4784aeb79c52d0f23a58", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.24.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09631c5799b64490af6f5a48c3d43232"}, {"url": "http://192.168.24.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0f4171a2e74f424898c7fba209b1c2e8"}, {"url": "http://192.168.24.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a1dfc6fab96842889cb7cb6a6b443ba4"}], "type": "baremetal", "id": "ef6aaf16eaf140dcb901309f9548282b", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.24.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1b2cf765c34c4d5494f571113321dfd2"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2abb7a435bf0448cb2e215f7685eb05a"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "35dcc2a70587405e99d9c34cf5134ec8"}], "type": "compute", "id": "fdba5af5b3ee45fab371773c1c17352f", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "289972c402df4bb39da6ed2135e8f56d"}, "audit_ids": ["jkhxv3EZSo6rg2K5rvT9Vg"], "issued_at": "2017-06-23T15:23:49.000000Z"}} run(Namespace(run_validations=False)) take_action(Namespace(run_validations=False)) This command is deprecated. Please use "openstack overcloud node introspect" to introspect manageable nodes instead. Instantiating baremetal client: <class 'ironicclient.v1.client.Client'> Baremetal API version: 1.9 Setting nodes for introspection to manageable... Moving available/enroll nodes to manageable state. Making authentication request to http://192.168.24.1:5000/v3/auth/tokens "POST /v3/auth/tokens HTTP/1.1" 201 8174 {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "ccda3744034848d2a0da0ec5eadcd821", "name": "admin"}], "expires_at": "2017-06-23T19:23:49.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "55f170434da2433d8d5c498f51efcc11", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.24.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "11cece1679ad4d78acb58c22c914f58b"}, {"url": "http://192.168.24.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ba56a2fb27a74c62b14625e96e325786"}, {"url": "http://192.168.24.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "d9d1d844f69b4f33aa3db31a9b6fbda0"}], "type": "messaging", "id": "5d5df786da294c3497330070dce18792", "name": "zaqar"}, {"endpoints": [{"url": "http://192.168.24.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4894e60793ec4a45a479205750b6d547"}, {"url": "http://192.168.24.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "979b44ec318540b994c0c532320ffe85"}, {"url": "http://192.168.24.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "c9142a68eb704c3eb3e6a1113746063c"}], "type": "placement", "id": "6c8bb1d3d3ef42b6ac85c4bc71412e94", "name": "placement"}, {"endpoints": [{"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "8eb5261e2d55476e83de82c22ee62600"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a6446773bef34ac286856107b69da81f"}, {"url": "http://192.168.24.1:8000/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "ef43b528dc5e411a867b8b4a0ffa23be"}], "type": "cloudformation", "id": "76432ddf4a904711b1d8205d9f55c118", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "12375bd744494f41be2bd864bac4c5d5"}, {"url": "http://192.168.24.1:8080/v1/AUTH_55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "a66d122477c64c02b70a413276452bed"}, {"url": "http://192.168.24.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f013ed60d58f48b2a2fe5c15df91fa08"}], "type": "object-store", "id": "7da80e7fc8b44dc4b302dbf3bd4cf12c", "name": "swift"}, {"endpoints": [{"url": "http://192.168.24.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "16e55c12241f4ac18e896db4e1a61649"}, {"url": "http://192.168.24.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "2e96fdd236eb40d0ae96603bd5792420"}, {"url": "http://192.168.24.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "93753cd9c4df4705bc9d1400918434e8"}], "type": "network", "id": "9bb9c0cd038241228a0a5c450343cbbd", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.24.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "4348e9dbbcc14972b6dc1e1bd71d873c"}, {"url": "http://192.168.24.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "59659611cdcc4a1a9a04205e8c99acfd"}, {"url": "http://192.168.24.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "59942a334d8541daa79b0ad68b6c186c"}], "type": "workflowv2", "id": "af9164fadde348588d7b5cd0e9ac30ad", "name": "mistral"}, {"endpoints": [{"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "927f45b99634430e8d5da5645313d838"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "bd4266a54f9d456da85c11ca6dd24674"}, {"url": "http://192.168.24.1:8004/v1/55f170434da2433d8d5c498f51efcc11", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "ed92f2fce02442cbb26d18c09b6413ec"}], "type": "orchestration", "id": "c5040fa43d7d409bacb81c808b5d3d3e", "name": "heat"}, {"endpoints": [{"url": "http://192.168.24.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "37763251b4724be5bc2d16ac45caadc0"}, {"url": "http://192.168.24.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "481cd150fbbd485aa76d113b5a3749bf"}, {"url": "http://192.168.24.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "485ca1edc1ff4ec2b596048f7ce2b758"}], "type": "image", "id": "dbf7da3ec4fa4d2fb80f6605ced454e1", "name": "glance"}, {"endpoints": [{"url": "http://192.168.24.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "03f1a42237aa4e3b94153ddf4ece48a7"}, {"url": "http://192.168.24.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "41a07725ebdb453ea5cc189d747aec08"}, {"url": "http://192.168.24.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "699a2598556b46fca25071b98aa0d519"}], "type": "identity", "id": "dbfd7492733148368f6658cb8b86ee98", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.24.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "062a1f6f2a8843fdabdcfdef99a05625"}, {"url": "http://192.168.24.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0c8588622e86486b99ba8aa11213e138"}, {"url": "http://192.168.24.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "9a915ec5999c43028e244722a1aab7a4"}], "type": "baremetal-introspection", "id": "e5493fc1f9914421802bdac54c9657fd", "name": "ironic-inspector"}, {"endpoints": [{"url": "ws://192.168.24.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "34366ac2b1f64f2398a5f38bef7b507d"}, {"url": "ws://192.168.24.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "4d9ec45cd18749caa1f4d32929bae046"}, {"url": "ws://192.168.24.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6a061669d91d42fe9db38bf18c7e22fa"}], "type": "messaging-websocket", "id": "e5ec63923f0b4784aeb79c52d0f23a58", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://192.168.24.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "09631c5799b64490af6f5a48c3d43232"}, {"url": "http://192.168.24.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0f4171a2e74f424898c7fba209b1c2e8"}, {"url": "http://192.168.24.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "a1dfc6fab96842889cb7cb6a6b443ba4"}], "type": "baremetal", "id": "ef6aaf16eaf140dcb901309f9548282b", "name": "ironic"}, {"endpoints": [{"url": "http://192.168.24.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1b2cf765c34c4d5494f571113321dfd2"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2abb7a435bf0448cb2e215f7685eb05a"}, {"url": "http://192.168.24.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "35dcc2a70587405e99d9c34cf5134ec8"}], "type": "compute", "id": "fdba5af5b3ee45fab371773c1c17352f", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "289972c402df4bb39da6ed2135e8f56d"}, "audit_ids": ["Q44za928Q2K67unMn8cctA"], "issued_at": "2017-06-23T15:23:49.000000Z"}} REQ: curl -g -i -X GET http://192.168.24.1:6385/v1/nodes/?associated=False&maintenance=False -H "X-OpenStack-Ironic-API-Version: 1.29" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}0432ffe14c6eb9a3dcc9bf378bfa0a6ca7899b08" Starting new HTTP connection (1): 192.168.24.1 "GET /v1/nodes/?associated=False&maintenance=False HTTP/1.1" 200 507 RESP: [200] Date: Fri, 23 Jun 2017 15:23:49 GMT Server: Apache X-OpenStack-Ironic-API-Minimum-Version: 1.1 X-OpenStack-Ironic-API-Maximum-Version: 1.31 X-OpenStack-Ironic-API-Version: 1.29 Openstack-Request-Id: req-6ae9f7a7-c92f-49b6-93ac-24fecb2716ba Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 507 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json RESP BODY: {"nodes": [{"instance_uuid": null, "uuid": "ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/ede82800-b510-48d1-ab5b-ee4a2d04b1ab", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-0"}, {"instance_uuid": null, "uuid": "abbf94c4-be5f-462a-b75f-9b0a2758d519", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/abbf94c4-be5f-462a-b75f-9b0a2758d519", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/abbf94c4-be5f-462a-b75f-9b0a2758d519", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-1"}, {"instance_uuid": null, "uuid": "d912eed1-3c9e-4be6-b5fb-b95f83165f74", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/d912eed1-3c9e-4be6-b5fb-b95f83165f74", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/d912eed1-3c9e-4be6-b5fb-b95f83165f74", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "controller-2"}, {"instance_uuid": null, "uuid": "cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/cf3ed0cd-bf03-44c1-a737-4eaccff176b6", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "compute-0"}, {"instance_uuid": null, "uuid": "c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/c410ef2f-f454-44d3-af7d-ffe1ef41f60f", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power on", "name": "compute-1"}, {"instance_uuid": null, "uuid": "c2436eb7-0031-4570-bd95-4de5a24d7784", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/c2436eb7-0031-4570-bd95-4de5a24d7784", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/c2436eb7-0031-4570-bd95-4de5a24d7784", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-0"}, {"instance_uuid": null, "uuid": "312dc5b7-c823-4ab6-8bee-bf3b286dade1", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/312dc5b7-c823-4ab6-8bee-bf3b286dade1", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/312dc5b7-c823-4ab6-8bee-bf3b286dade1", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-1"}, {"instance_uuid": null, "uuid": "e87a10d6-7668-4a93-be84-87673e458816", "links": [{"href": "http://192.168.24.1:6385/v1/nodes/e87a10d6-7668-4a93-be84-87673e458816", "rel": "self"}, {"href": "http://192.168.24.1:6385/nodes/e87a10d6-7668-4a93-be84-87673e458816", "rel": "bookmark"}], "maintenance": false, "provision_state": "manageable", "power_state": "power off", "name": "ceph-2"}]} Starting introspection of manageable nodes Instantiating workflow engine client: <class 'mistralclient.api.v2.client.Client'> REQ: curl -g -i -X GET http://192.168.24.1:5000/ -H "User-Agent: osc-lib/1.6.0 keystoneauth1/2.20.0 python-requests/2.11.1 CPython/2.7.5" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}0432ffe14c6eb9a3dcc9bf378bfa0a6ca7899b08" "GET / HTTP/1.1" 300 598 RESP: [300] Date: Fri, 23 Jun 2017 15:23:50 GMT Server: Apache Vary: X-Auth-Token Content-Length: 598 Keep-Alive: timeout=15, max=97 Connection: Keep-Alive Content-Type: application/json RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.24.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.24.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}} Waiting for introspection to finish... Instantiating messaging websocket client: ws://192.168.24.1:9000 {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}} Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 97, in run return self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 246, in take_action queue_name=queue_name) File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 150, in introspect_manageable_nodes with tripleoclients.messaging_websocket(queue_name) as ws: File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 200, in messaging_websocket return WebsocketClient(self._instance, queue_name) File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 97, in __init__ self.send('queue_create', {'queue_name': queue_name}) File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 123, in send raise RuntimeError(data) RuntimeError: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}} clean_up StartBaremetalIntrospectionBulk: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}} Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 97, in run return self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 246, in take_action queue_name=queue_name) File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 150, in introspect_manageable_nodes with tripleoclients.messaging_websocket(queue_name) as ws: File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 200, in messaging_websocket return WebsocketClient(self._instance, queue_name) File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 97, in __init__ self.send('queue_create', {'queue_name': queue_name}) File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 123, in send raise RuntimeError(data) RuntimeError: {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'b7433ff9-4abd-4f3a-9322-4ec63491a19e\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'b7433ff9-4abd-4f3a-9322-4ec63491a19e'}, u'api': u'v2', u'headers': {u'Client-ID': u'f200dfe5-31a3-40d7-8d94-d9f6d97e332a', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}} END return value: 1 ----------------- This looks like it should not be ironic specific, please feel free to change the component It's not ironic-specific indeed, and these commands are not even provided by ironic. And I've seen it with commands that do not involve ironic. Moving to tripleoclient for triaging. What is the status of MySQL database ? Can you access to it? can you provide more logs (sosreport maybe) - I want to see if you're having memory or storage issues (maybe some crontab doesn't work on the undercloud?). are you running telemetry services? if not, maybe https://bugs.launchpad.net/tripleo/+bug/1701357 is related to this bug. I do not see any telemetry services in the endpoint list on the undercloud, and overcloud isn't deployed yet It should be fixed now. Couldn't reproduce on OSP12 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |
Description of problem: This has already happened several times, but seems to be hard to reproduce consistently. A setup left alone for a long time (overnight tipycally) will fail to respond to an ironic related command producing a MySQL error. A second attempt at the same command usually works. Simple API queries (e.g. ironic node-list) also work. Examples: (openstack) baremetal introspection bulk start This command is deprecated. Please use "openstack overcloud node introspect" to introspect manageable nodes instead. Setting nodes for introspection to manageable... Starting introspection of manageable nodes Waiting for introspection to finish... {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'55f170434da2433d8d5c498f51efcc11\', \'name\': u\'d97b577e-3812-4998-9afb-4710cd33bd37\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'd97b577e-3812-4998-9afb-4710cd33bd37'}, u'api': u'v2', u'headers': {u'Client-ID': u'e8c19088-723a-4621-b945-719155daa3bd', u'X-Project-ID': u'55f170434da2433d8d5c498f51efcc11'}}} openstack baremetal import --json /home/stack/instackenv.json This command is deprecated. Please use "openstack overcloud node import" to register nodes instead. {u'body': {u'exception': u'(pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, \'Broken pipe\'))") [SQL: u\'INSERT INTO `Queues` (project, name, metadata) VALUES (%(project)s, %(name)s, %(metadata)s)\'] [parameters: {\'project\': u\'8f6d7d3087824450ade39c5854e7b7d1\', \'name\': u\'52cd9312-e11e-49f8-bb75-07859bacdd58\', \'metadata\': bytearray(b\'{}\')}]', u'error': u'Unexpected error.'}, u'headers': {u'status': 500}, u'request': {u'action': u'queue_create', u'body': {u'queue_name': u'52cd9312-e11e-49f8-bb75-07859bacdd58'}, u'api': u'v2', u'headers': {u'Client-ID': u'88d661b5-c0d1-4887-85ea-21871ad0d836', u'X-Project-ID': u'8f6d7d3087824450ade39c5854e7b7d1'}}} Version-Release number of selected component (if applicable): puppet-ironic-11.2.0-0.20170611220351.0fa87ae.el7ost.noarch python-ironic-lib-2.7.0-0.20170331023537.8d899bd.el7ost.noarch python-ironicclient-1.13.0-0.20170512184144.7cceefe.el7ost.noarch python-ironic-inspector-client-1.12.0-0.20170331023022.045fd9e.el7ost.noarch python-PyMySQL-0.7.9-2.el7ost.noarch openstack-nova-compute-16.0.0-0.20170609171025.el7ost.noarch openstack-ironic-conductor-8.0.1-0.20170610002959.adf3461.el7ost.noarch openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch openstack-tempest-16.0.1-0.20170612053404.c0e31f1.el7ost.noarch openstack-tripleo-common-containers-7.1.1-0.20170612015630.414f94b.el7ost.noarch openstack-tripleo-ui-7.1.1-0.20170609131448.202695f.el7ost.noarch openstack-nova-conductor-16.0.0-0.20170609171025.el7ost.noarch openstack-tripleo-heat-templates-7.0.0-0.20170611233361.el7ost.noarch openstack-swift-object-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch openstack-swift-proxy-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch openstack-heat-api-cfn-9.0.0-0.20170612043222.a289f55.el7ost.noarch python-openstackclient-3.11.0-0.20170505192416.el7ost.noarch openstack-mistral-api-5.0.0-0.20170607103828.8d51753.el7ost.noarch openstack-selinux-0.8.7-1.el7ost.noarch openstack-keystone-12.0.0-0.20170602202335.8628235.el7ost.noarch openstack-tripleo-image-elements-7.0.0-0.20170607161959.401d861.el7ost.noarch puppet-openstack_extras-11.2.0-0.20170609110653.d9f8351.el7ost.noarch openstack-neutron-common-11.0.0-0.20170611190934.01cc269.el7ost.noarch openstack-neutron-openvswitch-11.0.0-0.20170611190934.01cc269.el7ost.noarch openstack-ironic-inspector-5.1.1-0.20170608115120.a46e708.el7ost.noarch python-openstacksdk-0.9.16-0.20170608090447.bfc88e7.el7ost.noarch openstack-tripleo-puppet-elements-7.0.0-0.20170605222006.25e08b3.el7ost.noarch puppet-openstacklib-11.2.0-0.20170609110906.a99d5dc.el7ost.noarch openstack-tripleo-common-7.1.1-0.20170612015630.414f94b.el7ost.noarch openstack-swift-account-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch openstack-heat-common-9.0.0-0.20170612043222.a289f55.el7ost.noarch openstack-ironic-api-8.0.1-0.20170610002959.adf3461.el7ost.noarch openstack-mistral-executor-5.0.0-0.20170607103828.8d51753.el7ost.noarch openstack-tripleo-validations-7.1.1-0.20170609083517.e120452.el7ost.noarch openstack-nova-scheduler-16.0.0-0.20170609171025.el7ost.noarch openstack-glance-15.0.0-0.20170609191014.61046c8.el7ost.noarch openstack-nova-common-16.0.0-0.20170609171025.el7ost.noarch openstack-heat-api-9.0.0-0.20170612043222.a289f55.el7ost.noarch python-openstack-mistral-5.0.0-0.20170607103828.8d51753.el7ost.noarch openstack-mistral-common-5.0.0-0.20170607103828.8d51753.el7ost.noarch openstack-zaqar-5.0.0-0.20170609114413.51299ba.el7ost.noarch openstack-neutron-ml2-11.0.0-0.20170611190934.01cc269.el7ost.noarch openstack-nova-placement-api-16.0.0-0.20170609171025.el7ost.noarch openstack-swift-container-2.14.1-0.20170608204711.d46b0f2.el7ost.noarch openstack-neutron-11.0.0-0.20170611190934.01cc269.el7ost.noarch openstack-heat-engine-9.0.0-0.20170612043222.a289f55.el7ost.noarch openstack-ironic-common-8.0.1-0.20170610002959.adf3461.el7ost.noarch openstack-mistral-engine-5.0.0-0.20170607103828.8d51753.el7ost.noarch openstack-nova-api-16.0.0-0.20170609171025.el7ost.noarch mariadb-5.5.56-2.el7.x86_64 mariadb-server-5.5.56-2.el7.x86_64 mariadb-libs-5.5.56-2.el7.x86_64 How reproducible: hard to reproduce since it take a very long idle period Steps to Reproduce: 1. install undercloud 2. leave it for a long time (~12 hours seems to work) 3. try to run one of the above commands (there may be others) Actual results: see the failures above Expected results: should always work Additional info: leaving a setup idle now so I can rerun this with --debug tomorrow