Description of problem: Deleting inspection rule fails: Unexpected exception for https://192.168.24.2:13050/v1/rules/535c8a47-3523-4b03-9e2d-be70c0388fae: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) Server side there is no error, and the rule is in fact deleted. But the client report an error. Version-Release number of selected component (if applicable): REPOSITORY TAG IMAGE ID rh-osbs/rhosp17-openstack-ironic-inspector 17.0_20211207.1 ed248b487475 python3-ironic-inspector-client-4.5.0-0.20210528005713.3c03e21.el8ost.noarch python3-openstackclient-5.5.1-0.20210904061812.53f1efa.el8ost.noarch python3-ironicclient-4.6.3-0.20211022061911.7a77646.el8ost.noarch python3-openstacksdk-0.55.0-0.20210527232302.964fec6.el8ost.noarch python3-ironicclient-4.6.3-0.20211022061911.7a77646.el8ost.noarch python3-ironic-lib-4.6.3-0.20211126101848.972eda9.el8ost.noarch openstack-ironic-inspector-dnsmasq-10.6.2-0.20211006041849.06c6575.el8ost.noarch openstack-ironic-common-17.0.5-0.20211201041855.fc0f1c9.el8ost.noarch openstack-ironic-inspector-10.6.2-0.20211006041849.06c6575.el8ost.noarch How reproducible: 100% Steps to Reproduce: 1. Import inspection rule $ cat << EOF > /tmp/rule.json [ { "description": "Set default IPMI credentials", "conditions": [ {"op": "eq", "field": "data://auto_discovered", "value": true} ], "actions": [ {"action": "set-attribute", "path": "driver_info/ipmi_username", "value": "admin"}, {"action": "set-attribute", "path": "driver_info/ipmi_password", "value": "password"}, {"action": "set-attribute", "path": "driver_info/ipmi_address", "value": "172.16.0.32"}, {"action": "set-attribute", "path": "driver_info/ipmi_port", "value": "6234"} ] } ] $ openstack baremetal introspection rule import /tmp/rule.json 2. Delete inspeciton rule $ openstack baremetal introspection rule delete <RULE_UUID> Actual results: Unexpected exception for https://192.168.24.2:13050/v1/rules/1dc1fde7-e4ed-4ed9-939c-48a61c4617e2: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) Expected results: The client should not error. Additional info: Manually deleting a rule using curl reveals error: "curl: (18) transfer closed with outstanding read data remaining" $ curl -g -i -X delete https://192.168.24.2:13050/v1/rules/67736c21-d09c-488f-b63f-fd0c754141af -H "User-Agent: openstacksdk/0.55.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8" -H "X-Auth-Token: $TOKEN_ID" -H "X-OpenStack-Ironic-Inspector-API-Version: 1.0" HTTP/1.1 204 NO CONTENT Content-Type: text/plain; charset=utf-8 X-Openstack-Ironic-Inspector-Api-Minimum-Version: 1.0 X-Openstack-Ironic-Inspector-Api-Maximum-Version: 1.17 Date: Mon, 10 Jan 2022 19:16:18 GMT Transfer-Encoding: chunked curl: (18) transfer closed with outstanding read data remaining client '--debug' output: ------------------------ (undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection rule delete 1dc1fde7-e4ed-4ed9-939c-48a61c4617e2 --debug START with options: baremetal introspection rule delete 1dc1fde7-e4ed-4ed9-939c-48a61c4617e2 --debug options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='', cacert=None, cert='', client_id='', client_secret='***', cloud='undercloud', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.69', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_placement_api_version='1.0', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_share_api_version='2.63', os_tripleoclient_api_version='2', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='', project_id='', project_name='', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='', user_id='', username='', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': '', 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'auth_url': 'https://192.168.24.2:13000', 'password': '***', 'username': 'admin', 'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'identity_api_version': '3', 'region_name': 'regionOne', 'volume_api_version': '3', 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'cloud': 'undercloud', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'beta_command': False, 'orchestration_api_version': '1', 'queues_api_version': '2', 'workflow_api_version': '2', 'database_api_version': '1', 'share_api_version': '2.63', 'dns_api_version': '2', 'container_infra_api_version': '1', 'key_manager_api_version': '1', 'alarming_api_version': '2', 'baremetal_api_version': '1.69', 'loadbalancer_api_version': '2.0', 'data_processing_api_version': '1.1', 'tripleoclient_api_version': '2', 'metrics_api_version': '1', 'placement_api_version': '1.0', 'auth_type': 'password', 'networks': []} defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'} cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': '', 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'auth_url': 'https://192.168.24.2:13000', 'password': '***', 'username': 'admin', 'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'identity_api_version': '3', 'region_name': 'regionOne', 'volume_api_version': '3', 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'cloud': 'undercloud', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'beta_command': False, 'orchestration_api_version': '1', 'queues_api_version': '2', 'workflow_api_version': '2', 'database_api_version': '1', 'share_api_version': '2.63', 'dns_api_version': '2', 'container_infra_api_version': '1', 'key_manager_api_version': '1', 'alarming_api_version': '2', 'baremetal_api_version': '1.69', 'loadbalancer_api_version': '2.0', 'data_processing_api_version': '1.1', 'tripleoclient_api_version': '2', 'metrics_api_version': '1', 'placement_api_version': '1.0', 'auth_type': 'password', 'networks': []} compute API version 2.1, cmd group openstack.compute.v2 identity API version 3, cmd group openstack.identity.v3 image API version 2, cmd group openstack.image.v2 network API version 2, cmd group openstack.network.v2 object_store API version 1, cmd group openstack.object_store.v1 volume API version 3, cmd group openstack.volume.v3 orchestration API version 1, cmd group openstack.orchestration.v1 messaging API version 2, cmd group openstack.messaging.v2 neutronclient API version 2, cmd group openstack.neutronclient.v2 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 database API version 1, cmd group openstack.database.v1 share API version 2.63, cmd group openstack.share.v2 dns API version 2, cmd group openstack.dns.v2 container_infra API version 1, cmd group openstack.container_infra.v1 /usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead. % (name, name, name)) key_manager API version 1, cmd group openstack.key_manager.v1 alarming API version 2, cmd group openstack.alarming.v2 baremetal API version 1.69, cmd group openstack.baremetal.v1 load_balancer API version 2.0, cmd group openstack.load_balancer.v2 data_processing API version 1.1, cmd group openstack.data_processing.v1 Popen(['git', 'version'], cwd=/home/stack, universal_newlines=False, shell=None, istream=None) Popen(['git', 'version'], cwd=/home/stack, universal_newlines=False, shell=None, istream=None) /usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change. To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`. DeprecationWarning tripleoclient API version 2, cmd group openstack.tripleoclient.v2 metric API version 1, cmd group openstack.metric.v1 placement API version 1.0, cmd group openstack.placement.v1 command: baremetal introspection rule delete -> ironic_inspector_client.shell.RuleDeleteCommand (auth=True) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': '', 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'auth_url': 'https://192.168.24.2:13000', 'password': '***', 'username': 'admin', 'user_domain_name': 'Default', 'project_domain_name': 'Default', 'project_name': 'admin'}, 'identity_api_version': '3', 'region_name': 'regionOne', 'volume_api_version': '3', 'additional_user_agent': [('osc-lib', '2.3.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'cloud': 'undercloud', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'beta_command': False, 'orchestration_api_version': '1', 'queues_api_version': '2', 'workflow_api_version': '2', 'database_api_version': '1', 'share_api_version': '2.63', 'dns_api_version': '2', 'container_infra_api_version': '1', 'key_manager_api_version': '1', 'alarming_api_version': '2', 'baremetal_api_version': '1.69', 'loadbalancer_api_version': '2.0', 'data_processing_api_version': '1.1', 'tripleoclient_api_version': '2', 'metrics_api_version': '1', 'placement_api_version': '1.0', 'auth_type': 'password', 'networks': []} Using auth plugin: password Using parameters {'auth_url': 'https://192.168.24.2:13000', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'} Get auth_ref REQ: curl -g -i -X GET https://192.168.24.2:13000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.55.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8" Starting new HTTPS connection (1): 192.168.24.2:13000 https://192.168.24.2:13000 "GET / HTTP/1.1" 300 269 RESP: [300] Content-Length: 269 Content-Type: application/json Date: Mon, 10 Jan 2022 19:06:35 GMT Location: https://192.168.24.2:13000/v3/ Server: Apache Vary: X-Auth-Token x-openstack-request-id: req-89f12400-7ad7-48e5-8f41-6aa5e3775898 RESP BODY: {"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "https://192.168.24.2:13000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}} GET call to https://192.168.24.2:13000/ used request id req-89f12400-7ad7-48e5-8f41-6aa5e3775898 Making authentication request to https://192.168.24.2:13000/v3/auth/tokens https://192.168.24.2:13000 "POST /v3/auth/tokens HTTP/1.1" 201 2966 {"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "1c2073ae52cf4f9ebee433521ccf4adf", "name": "admin", "password_expires_at": null}, "audit_ids": ["lNjrgiPYRJ6HEwnN1BkUrg"], "expires_at": "2022-01-10T23:06:35.000000Z", "issued_at": "2022-01-10T19:06:35.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "e4d9deacaa134bfdb50ee54f823532cd", "name": "admin"}, "is_domain": false, "roles": [{"id": "1aeca53bc68c47909301efd24522aa8a", "name": "admin"}, {"id": "e0f65559df0940389b88dbaf4c35a2cd", "name": "reader"}, {"id": "dcdc172504504005b0c3adc3cf33f07c", "name": "_member_"}, {"id": "85778fe17b9646d0ab3a9724dba757b1", "name": "member"}], "catalog": [{"endpoints": [{"id": "0b5922fabc424582889de1bd01484f48", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne"}, {"id": "2930c1aacc934b6da0dd8ec2f58406a6", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13050", "region": "regionOne"}, {"id": "7794533a316d478e8614c7e08681908e", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:5050", "region": "regionOne"}], "id": "24f883e172244e569602055a9df6ba3d", "type": "baremetal-introspection", "name": "ironic-inspector"}, {"endpoints": [{"id": "1226af5e6414430fa19c11b5cc74e206", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:35357", "region": "regionOne"}, {"id": "542968446f9b4d5daa86fecb21fb3b27", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13000", "region": "regionOne"}, {"id": "a3da1a8067a744eda13df50269ab32f2", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:5000", "region": "regionOne"}], "id": "3ef786401a034f2abd5b07a2b980c91e", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "25c06c0b1c4849ceb8226cd2c69d4290", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13385", "region": "regionOne"}, {"id": "a388803081ba4f369e04e351392572b4", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:6385", "region": "regionOne"}, {"id": "b07f8ccf909d4701a5085f189fb8bbc3", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:6385", "region": "regionOne"}], "id": "6b60983a006048c3abc2d085067bdd78", "type": "baremetal", "name": "ironic"}, {"endpoints": [{"id": "55ca6cfa277741ee8ad99362479c33e4", "interface": "admin", "region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne"}, {"id": "ec062da23f5e4bc3a0bb891bf222c5a1", "interface": "internal", "region_id": "regionOne", "url": "http://192.168.24.3:9696", "region": "regionOne"}, {"id": "ed16a64e41a34bd2937e0747bbf12d76", "interface": "public", "region_id": "regionOne", "url": "https://192.168.24.2:13696", "region": "regionOne"}], "id": "e5616b40de924b34a4cbc800f14b232b", "type": "network", "name": "neutron"}]}} REQ: curl -g -i -X GET https://192.168.24.2:13050 -H "User-Agent: openstacksdk/0.55.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8" Starting new HTTPS connection (1): 192.168.24.2:13050 https://192.168.24.2:13050 "GET / HTTP/1.1" 200 112 RESP: [200] Content-Length: 112 Content-Type: application/json Date: Mon, 10 Jan 2022 19:06:35 GMT X-Openstack-Ironic-Inspector-Api-Maximum-Version: 1.17 X-Openstack-Ironic-Inspector-Api-Minimum-Version: 1.0 RESP BODY: {"versions":[{"id":"1.17","links":[{"href":"https://192.168.24.2:13050/v1","rel":"self"}],"status":"CURRENT"}]} Supported API version range for https://192.168.24.2:13050 is [1.0, 1.17] Requesting DELETE https://192.168.24.2:13050/v1/rules/1dc1fde7-e4ed-4ed9-939c-48a61c4617e2 (API version 1.0) with {} REQ: curl -g -i -X delete https://192.168.24.2:13050/v1/rules/1dc1fde7-e4ed-4ed9-939c-48a61c4617e2 -H "User-Agent: openstacksdk/0.55.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8" -H "X-Auth-Token: {SHA256}72a6a3bcf673764ed66baee0fc59c7c0ca1056ac69cee3b32264f4b67b069b17" -H "X-OpenStack-Ironic-Inspector-API-Version: 1.0" https://192.168.24.2:13050 "DELETE /v1/rules/1dc1fde7-e4ed-4ed9-939c-48a61c4617e2 HTTP/1.1" 204 0 Unexpected exception for https://192.168.24.2:13050/v1/rules/1dc1fde7-e4ed-4ed9-939c-48a61c4617e2: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 696, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b'' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 436, in _error_catcher yield File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 763, in read_chunked self._update_chunk_length() File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 700, in _update_chunk_length raise httplib.IncompleteRead(line) http.client.IncompleteRead: IncompleteRead(0 bytes read) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/requests/models.py", line 753, in generate for chunk in self.raw.stream(chunk_size, decode_content=True): File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 571, in stream for line in self.read_chunked(amt, decode_content=decode_content): File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 792, in read_chunked self._original_response.close() File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python3.6/site-packages/urllib3/response.py", line 454, in _error_catcher raise ProtocolError("Connection broken: %r" % e, e) urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1022, in _send_request resp = self.session.request(method, url, **kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3.6/site-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3.6/site-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/ironic_inspector_client/shell.py", line 271, in take_action client.rules.delete(parsed_args.uuid) File "/usr/lib/python3.6/site-packages/ironic_inspector_client/v1.py", line 481, in delete self._request('delete', '/rules/%s' % uuid) File "/usr/lib/python3.6/site-packages/ironic_inspector_client/common/http.py", line 213, in request raise_exc=False, **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 931, in request resp = send(**kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1042, in _send_request raise exceptions.UnknownConnectionError(msg, e) keystoneauth1.exceptions.connection.UnknownConnectionError: Unexpected exception for https://192.168.24.2:13050/v1/rules/1dc1fde7-e4ed-4ed9-939c-48a61c4617e2: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) clean_up RuleDeleteCommand: Unexpected exception for https://192.168.24.2:13050/v1/rules/1dc1fde7-e4ed-4ed9-939c-48a61c4617e2: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) END return value: 1 ------------------------
I have eliminated haproxy response munging by reproducing the issue when requesting directly from the inspector port: export INSPECTOR_URL="http://192.168.24.3:5050"
I can avoid the error by modifying the delete request call[1] to add stream=True, and trigger it again when setting stream=False. Its as if in this environment having the Transfer-Encoding: chunked response sets python-requests on the chunked request parsing codepath which errors on the empty response. Setting stream=True avoids parsing the response. I tried but failed to reproduce locally with a centos-8.4 container and similar client versions installed. [1] https://opendev.org/openstack/python-ironic-inspector-client/src/branch/master/ironic_inspector_client/v1.py#L481
Possibly it is a server-side issue, a fresh pip-install of python-ironic-inspector-client in a venv has the same issue.
Just updating the findings from today: - Setting header Connection: close doesn't help - the wsgi stack will always set Transfer-Encoding: chunked for http/1.1 requests - forcing a curl request to do http/1.0 fixes the problem, but python requests only supports http/1.1 - inside the ironic-inspector container upgrading eventlet, werkzeug, and flask made no difference At this point I can only assume this is a python3.6 issue. Also I bet the problem goes away when osp-17 jobs move to rhel-9 instead of rhel-8.4
Okay, we've got a root cause finally. It is eventlet, which is stamping an transfer encoding header on all replies which do not contain a content length. Which is an RFC violation. This causes all sorts of fun issues, including this one.
Upstream change to minimize impact of the eventlet issue with ironic-inspector has been merged to stable/wallaby. Backporting.
Verified manually by running through the repro steps and it is working properly: (undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection rule import /tmp/rule.json /usr/lib/python3.9/site-packages/ansible/_vendor/__init__.py:42: UserWarning: One or more Python packages bundled by this ansible-core distribution were already loaded (pyparsing). This may result in undefined behavior. warnings.warn('One or more Python packages bundled by this ansible-core distribution were already ' +--------------------------------------+------------------------------+ | UUID | Description | +--------------------------------------+------------------------------+ | 843383a1-70bc-494c-b485-3e67f4485309 | Set default IPMI credentials | +--------------------------------------+------------------------------+ (undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection rule delete 843383a1-70bc-494c-b485-3e67f4485309 /usr/lib/python3.9/site-packages/ansible/_vendor/__init__.py:42: UserWarning: One or more Python packages bundled by this ansible-core distribution were already loaded (pyparsing). This may result in undefined behavior. warnings.warn('One or more Python packages bundled by this ansible-core distribution were already ' (undercloud) [stack@undercloud-0 ~]$ notice no errors ^
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543