Bug 2039046
| Summary: | introspection rule delete - Unexpected exception ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Harald Jensås <hjensas> |
| Component: | openstack-ironic-inspector | Assignee: | Julia Kreger <jkreger> |
| Status: | CLOSED ERRATA | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 17.0 (Wallaby) | CC: | jkreger, jparoly, sbaker |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 17.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-ironic-inspector-10.6.2-0.20220118051837.8f97076.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-09-21 12:18:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Harald Jensås
2022-01-10 19:43:49 UTC
I have eliminated haproxy response munging by reproducing the issue when requesting directly from the inspector port: export INSPECTOR_URL="http://192.168.24.3:5050" I can avoid the error by modifying the delete request call[1] to add stream=True, and trigger it again when setting stream=False. Its as if in this environment having the Transfer-Encoding: chunked response sets python-requests on the chunked request parsing codepath which errors on the empty response. Setting stream=True avoids parsing the response. I tried but failed to reproduce locally with a centos-8.4 container and similar client versions installed. [1] https://opendev.org/openstack/python-ironic-inspector-client/src/branch/master/ironic_inspector_client/v1.py#L481 Possibly it is a server-side issue, a fresh pip-install of python-ironic-inspector-client in a venv has the same issue. Just updating the findings from today: - Setting header Connection: close doesn't help - the wsgi stack will always set Transfer-Encoding: chunked for http/1.1 requests - forcing a curl request to do http/1.0 fixes the problem, but python requests only supports http/1.1 - inside the ironic-inspector container upgrading eventlet, werkzeug, and flask made no difference At this point I can only assume this is a python3.6 issue. Also I bet the problem goes away when osp-17 jobs move to rhel-9 instead of rhel-8.4 Okay, we've got a root cause finally. It is eventlet, which is stamping an transfer encoding header on all replies which do not contain a content length. Which is an RFC violation. This causes all sorts of fun issues, including this one. Upstream change to minimize impact of the eventlet issue with ironic-inspector has been merged to stable/wallaby. Backporting. Verified manually by running through the repro steps and it is working properly:
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection rule import /tmp/rule.json
/usr/lib/python3.9/site-packages/ansible/_vendor/__init__.py:42: UserWarning: One or more Python packages bundled by this ansible-core distribution were already loaded (pyparsing). This may result in undefined behavior.
warnings.warn('One or more Python packages bundled by this ansible-core distribution were already '
+--------------------------------------+------------------------------+
| UUID | Description |
+--------------------------------------+------------------------------+
| 843383a1-70bc-494c-b485-3e67f4485309 | Set default IPMI credentials |
+--------------------------------------+------------------------------+
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal introspection rule delete 843383a1-70bc-494c-b485-3e67f4485309
/usr/lib/python3.9/site-packages/ansible/_vendor/__init__.py:42: UserWarning: One or more Python packages bundled by this ansible-core distribution were already loaded (pyparsing). This may result in undefined behavior.
warnings.warn('One or more Python packages bundled by this ansible-core distribution were already '
(undercloud) [stack@undercloud-0 ~]$
notice no errors ^
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543 |