Bug 1870126 - tripleo validator healthcheck-service-status thread lock
Summary: tripleo validator healthcheck-service-status thread lock
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z2
: 16.1 (Train on RHEL 8.2)
Assignee: Cédric Jeanneret
QA Contact: David Rosenfeld
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-19 11:52 UTC by David Vallee Delisle
Modified: 2023-12-15 18:56 UTC (History)
10 users (show)

Fixed In Version: python-tripleoclient-12.3.2-1.20200914164926.72161e0.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-28 15:39:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-30859 0 None None None 2023-12-15 18:56:55 UTC
Red Hat Product Errata RHEA-2020:4284 0 None None None 2020-10-28 15:39:30 UTC

Description David Vallee Delisle 2020-08-19 11:52:47 UTC
Description of problem:
When running tripleo validator run, we get a thread lock [1]

Version-Release number of selected component (if applicable):
openstack-tripleo-validations-11.3.2-0.20200318124452.3fd14c9.el8ost.noarch

How reproducible:
at least 50% of the time

Additional info:

[1]
~~~
(undercloud) [stack@undercloud-0 ~]$ strace -fff -tt -s 9999 -o tripleo_validator openstack --debug tripleo validator run --validation healthcheck-service-status --workers 1                                                                                                                                                                                                                                                               START with options: --debug tripleo validator run --validation healthcheck-service-status --workers 1                                                                                                                                                                                                                                                                                                                                       options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='password', auth_url='https://192.168.24.2:13000', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_placement_api_version='1.0', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)                                                                                                                                                                                Auth plugin password selected                                                                                                                                                                                                                                                                                                                                                                                                               auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_ver
sion': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3
', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'placement_api_version': '1.0', 'metrics_api_ver
sion': '1', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []}                                                                                                                                                                                                                                                                                                                                       
defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_ap
i_version': '1', 'secgroup_source': 'neutron', 'status': 'active'}
cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1
', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queue
s_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'placement_api_version': '1.0', 'metrics_api_version': '1
', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []}
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3                     
image API version 2, cmd group openstack.image.v2  
network API version 2, cmd group openstack.network.v2                   
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3                                    
messaging API version 2, cmd group openstack.messaging.v2       
database API version 1, cmd group openstack.database.v1                            
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1            
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2                    
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1       
baremetal API version 1.58, cmd group openstack.baremetal.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1                    
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2                                                                   
key_manager API version 1, cmd group openstack.key_manager.v1
event API version 2, cmd group openstack.event.v2                               
placement API version 1.0, cmd group openstack.placement.v1
metric API version 1, cmd group openstack.metric.v1                              
alarming API version 2, cmd group openstack.alarming.v2
Auth plugin password selected                                 
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_ver
sion': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3
', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'placement_api_version': '1.0', 'metrics_api_ver
sion': '1', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []}
command: tripleo validator run -> tripleoclient.v1.tripleo_validator.TripleOValidatorRun (auth=True)
Auth plugin password selected     
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_ver
sion': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'https://192.168.24.2:13000', 'username': 'admin', 'password': '***',
 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2'
, 'placement_api_version': '1.0', 'metrics_api_version': '1', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []}
Using auth plugin: password                                                    
Using parameters {'auth_url': 'https://192.168.24.2:13000', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET https://192.168.24.2:13000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.2 keystoneauth1/3.17.2 python-requests/2.20.0 CPython/3.6.8"
Starting new HTTPS connection (1): 192.168.24.2:13000
https://192.168.24.2:13000 "GET / HTTP/1.1" 300 269
RESP: [300] Content-Length: 269 Content-Type: application/json Date: Tue, 18 Aug 2020 20:19:05 GMT Location: https://192.168.24.2:13000/v3/ Server: Apache Vary: X-Auth-Token x-openstack-request-id: req-888a2f77-9aea-48cf-a619-a2f0abf55a9d                                                                                                                                                                                             
RESP BODY: {"versions": {"values": [{"id": "v3.13", "status": "stable", "updated": "2019-07-19T00:00:00Z", "links": [{"rel": "self", "href": "https://192.168.24.2:13000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}                                                                                                                                                   
GET call to https://192.168.24.2:13000/ used request id req-888a2f77-9aea-48cf-a619-a2f0abf55a9d
Making authentication request to https://192.168.24.2:13000/v3/auth/tokens
https://192.168.24.2:13000 "POST /v3/auth/tokens HTTP/1.1" 201 7654
{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "b6fa029392ba414387548b512ca091f4", "name": "admin", "password_expires_at": null}, "audit_ids": ["Q4zlTQgvTl2mXnudSWtd7Q"], "expires_at": "2020-08-19T00:19:06.000000Z", "issued_at": "2020-08-18T20:19:06.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "539d39acc19445f4b6f30238a12f890a", "name": "admin"}, "is_domain": false, "roles": [{"id": "b3e16a546a1642e8b64b746662c16554", "name": "admin"}, {"id": "a726a0c5b78c46fc90bc39789bf6a268", "name": "_member_"}, {"id": "f4a975e4a5a4492c8c747acfb258e874", "name": "reader"}, {"id": "52409f9e5c684b219eaf79bc7bc7f301", "name": "member"}], "catalog": [{"endpoints": [{"id": "afa75a1bae2c4502be6588f0d329471a", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8080", "region": "regionOne"}, {"id": "b39788313385475694c40bcfd5ec4fc9", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13808/v1/AUTH_539d39acc19445f4b6f30238a12f890a", "region": "regionOne"}, {"id": "d0762401098d4a9ab10dcce9447b9443", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8080/v1/AUTH_539d39acc19445f4b6f30238a12f890a", "region": "regionOne"}], "id":
"065b6b7d9e5a4248b772728837ae8ee0", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "4a31800264204f748a6fca2a379abea5", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8004/v1/539d39acc19445f4b6f30238a12f890a", "region": "regionOne"}, {"id": "704ff69a9e414804a0333b6bb1e443af", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8004/v1/539d39acc19445f4b6f30238a12f890a", "region": "regionOne"}, {"id": "fb480f3c9b69475585faa58e960b46a4", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13004/v1/539d39acc19445f4b6f30238a12f890a", "region": "regionOne"}], "id": "0e0404152d25411580fa05e7d8bc8ffb", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "c972885c1ee043ed8fbe3181f0968b7e", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}, {"id": "d1f345c286514580a30a55cfe703ea8b", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8778/placement", "region": "regionOne"}, {"id": "fa5f935874784c008771eb582f4a1eca", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13778/placement", "region": "regionOne"}], "id": "127af4b91ce44934a4bf9ec40dc75aaf", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "0159d1b1d0bd404bab3146f7a2a2be08", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8989/v2", "region": "regionOne"}, {"id": "40de246264fe4f818056f65d5eb8c97d", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13989/v2", "region": "regionOne"}, {"id": "69049f8b3b104e1eb2fe214424f19853", "interface":
"admin", "region_id": "regionOne", "url": "http://192.168.24.3:8989/v2", "region": "regionOne"}], "id": "2896e98d5ce9449588a824435aa374c6", "type": "workflowv2", "name": "mistral"}, {"endpoints": [{"id": "117991fe99da4d4892c8deecb8ce4380", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne"}, {"id": "8b3b48d145d4477db7c70587b51cec47", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13696", "region": "regionOne"}, {"id": "8c09e541d81a400cb59e0a3e534f0973", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne"}], "id": "2cd25a89311746fbaf760ca10027f1c6", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "2e4e69388763461daf640547190bdd63", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:5000", "region": "regionOne"}, {"id": "3f559d22fb284d108b108481ab5e32c0", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13000", "region": "regionOne"}, {"id": "b4bfe9c77b704a5096e487b25b6739d3", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:35357", "region": "regionOne"}], "id": "6c7fb2ba11834f7a96850e0614419902", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "c767716af1134053952137fbdf63a375", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}, {"id": "f46fd5c98a6846e3b1094f92d17be5c1", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8774/v2.1", "region": "regionOne"}, {"id": "fddd1d71b096470a92c931bc16b5b6ea", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13774/v2.1", "region": "regionOne"}], "id": "72e1b090e2d148249ed143c6e435c587", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "5084f9c08aca41bbb049d426381ec4b7", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13888", "region": "regionOne"}, {"id": "8e1e8281a7584788b9b6f4adc99923b9", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:8888", "region": "regionOne"}, {"id": "c4699e01faf74c6383f0500d51cb9528", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:8888", "region": "regionOne"}], "id": "87fdae101861435ba99bea5e2342ff49", "type": "messaging", "name": "zaqar"}, {"endpoints": [{"id": "b29c9945016c4053a747e9225950a4ed", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:6385", "region": "regionOne"}, {"id": "cf48dc1438f645e5a736a79abf8b2ac2", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13385", "region": "regionOne"}, {"id": "e35bdd8d20884693ad47958431eaf86c", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:6385", "region": "regionOne"}], "id": "a32141c1e7244b6fa2b82522550e27f3", "type": "baremetal", "name": "ironic"}, {"endpoints": [{"id": "1e686b0cf94f4813b6487f99927a269f", "interface": "public", "region_id": "regionOne", "url": "wss://192.168.24.2:9000", "region": "regionOne"}, {"id": "c0f950bbc6834ee2b639ba45d862383f", "interface": "admin", "region_id": "regionOne", "url": "ws://192.168.24.3:9000", "region": "regionOne"}, {"id": "e5b6096e457c4681aff2d4fad8572217", "interface": "internal", "region_id": "regionOne", "url": "ws://192.168.24.3:9000", "region": "regionOne"}], "id": "c9b8d982cfcc4bd7b89208b56d5d7b2b", "type": "messaging-websocket", "name": "zaqar-websocket"}, {"endpoints": [{"id": "25d8d6a5d22d4244a60977ac3a2f52b2", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}, {"id": "4fddc00752fb46fe919b9ba4e1ea20c4", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:9292", "region": "regionOne"}, {"id": "f944f75c34a249eea454d4c6468959dc", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13292", "region": "regionOne"}], "id": "cd6bbe0e59c94869a2d386256d18f3bb", "type": "image", "name": "glance"}, {"endpoints": [{"id": "37961d5a05a84fbfbcbfc6035f2ceadf", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne"}, {"id": "a488b8f6a36748d297bd1f9db244d1ad", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13050", "region": "regionOne"}, {"id": "ef3ce7cee5334730a2e9ffd659bc1179", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne"}], "id": "fa5121535b694ed6b2eba4715027dd4a", "type": "baremetal-introspection", "name": "ironic-inspector"}]}}                                                 
run(Namespace(extra_vars={}, extra_vars_file='', group=[], plan='overcloud', validation_name=['healthcheck-service-status'], workers=1))
Running cmd (subprocess): /usr/bin/tripleo-ansible-inventory --stack overcloud --ansible_ssh_user heat-admin --undercloud-connection local --undercloud-key-file /var/lib/mistral/.ssh/tripleo-admin-rsa --static-yaml-inventory /home/stack/tripleo-ansible-inventory.yaml                                                                                                                                                                
CMD "/usr/bin/tripleo-ansible-inventory --stack overcloud --ansible_ssh_user heat-admin --undercloud-connection local --undercloud-key-file /var/lib/mistral/.ssh/tripleo-admin-rsa --static-yaml-inventory /home/stack/tripleo-ansible-inventory.yaml" returned: 0 in 3.802s                                                                                                                                                              
Running the validations with Ansible
^CEND return value: 1
Traceback (most recent call last):
  File "/usr/bin/openstack", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 179, in main
    return OpenStackShell().run(argv)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 460, in take_action
    self._run_validator_run(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 438, in _run_validator_run
    for playbook in playbooks
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 611, in __exit__
    self.shutdown(wait=True)
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 152, in shutdown
    t.join()
  File "/usr/lib64/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/usr/lib64/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 40, in _python_exit
    t.join()
  File "/usr/lib64/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/usr/lib64/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
~~~

Comment 2 Cédric Jeanneret 2020-08-20 13:47:10 UTC
Hello David,

Care to add "-w 2" to the command? The validator uses "old" code in osp-16.1, we're working hard in order to get the newer thing in place for a zstream.

It's probably linked to this code:
https://github.com/openstack/python-tripleoclient/blob/stable/train/tripleoclient/v1/tripleo_validator.py#L484

Cheers,

C.

Comment 4 David Vallee Delisle 2020-08-21 11:58:39 UTC
Cédric,

That was my impression as well. I put the strace files and sosreport here [1].

Thanks,

DVD

[1] http://file.rdu.redhat.com/~dvalleed/bz1870126/

Comment 5 Cédric Jeanneret 2020-08-24 09:15:58 UTC
Adding Mathieu, and creating an env to check if passing -w X would be a solution.

Comment 7 Cédric Jeanneret 2020-09-14 05:19:56 UTC
Hello,

I think this issue is now resolved with the latest things we've pushed for z2 - we've switch the whole code to the newer version that doesn't use "thread".

Care to confirm once we get a proper build with the new code?

Cheers,

C.

Comment 8 David Vallee Delisle 2020-09-15 19:16:44 UTC
Thanks Cédric,

It looks like it's working now. [1].

My only concern now is the ResourceWarning. Is there a way to make it disapear? I know it might cause our cautious customers to open cases.

Thanks,

DVD


[1]
~~~
(undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --validation healthcheck-service-status --workers 1                                                                                                                                                                                                                                                                                                                    
/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py:437: ResourceWarning: unclosed file <_io.BufferedReader name=7>
  gathering_policy=gathering_policy)
+--------------------------------------+----------------------------+--------+-----------------------+-------------------------------------+---------------------+-------------+
| UUID                                 | Validations                | Status | Host Group(s)         | Status by Host                      | Unreachable Host(s) | Duration    |
+--------------------------------------+----------------------------+--------+-----------------------+-------------------------------------+---------------------+-------------+
| 5254005a-bc9d-4647-fd51-000000000008 | healthcheck-service-status | PASSED | undercloud, overcloud | compute-0, controller-0, undercloud |                     | 0:00:23.494 |
+--------------------------------------+----------------------------+--------+-----------------------+-------------------------------------+---------------------+-------------+
~~~

Comment 9 Cédric Jeanneret 2020-09-16 06:09:25 UTC
Hello David,

that warning isn't due to the validator itself, it's somewhere in openstack clients. There have been some work on them upstream, not sure about the backport status though.. There are such messages for other CLI calls as well :(.

I propose to close the current BZ and, if needed, create a new one for those warnings (though they are harmless, it's python3 being stricter than python2.7).

Cheers,

C.

Comment 17 errata-xmlrpc 2020-10-28 15:39:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4284


Note You need to log in before you can comment on or make changes to this bug.