Bug 1917261
| Summary: | Satellite job failed on undercloud stage while uploading images to local registry | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jad Haj Yahya <jhajyahy> | ||||
| Component: | openstack-tripleo-common | Assignee: | Alex Schultz <aschultz> | ||||
| Status: | CLOSED ERRATA | QA Contact: | David Rosenfeld <drosenfe> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 16.2 (Train) | CC: | aschultz, cjeanner, kecarter, mburns, slinaber | ||||
| Target Milestone: | rc | Keywords: | Triaged | ||||
| Target Release: | 16.2 (Train on RHEL 8.4) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-common-11.4.1-2.20210206004924.a42a18b.el8ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-09-15 07:11:11 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Jad Haj Yahya
2021-01-18 08:30:58 UTC
Created attachment 1748482 [details]
undercloud-install.log
Attached is a "reformatted" log that should make the issue more obvious than the terrible original output.
It sounds like some internal timeout with the session, as we can see with the following stack-trace (it's the last block in the attached file):
"ERROR:root:Image prepare failed: '<' not supported between instances of 'int' and 'NoneType'"
"concurrent.futures.process._RemoteTraceback: "
"""""
"Traceback (most recent call last):"
" File "/usr/lib64/python3.6/concurrent/futures/process.py"
line 175, in _process_worker"
" r = call_item.fn(*call_item.args, **call_item.kwargs)"
" File "/usr/lib64/python3.6/concurrent/futures/process.py"
line 153, in _process_chunk"
" return [fn(*args) for args in chunk]"
" File "/usr/lib64/python3.6/concurrent/futures/process.py"
line 153, in <listcomp>"
" return [fn(*args) for args in chunk]"
" File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py"
line 2678, in upload_task"
" return uploader.upload_image(task)"
" File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py"
line 1564, in upload_image"
" password=target_password"
" File "/usr/lib/python3.6/site-packages/tenacity/__init__.py"
line 292, in wrapped_f"
" return self.call(f, *args, **kw)"
" File "/usr/lib/python3.6/site-packages/tenacity/__init__.py"
line 358, in call"
" do = self.iter(retry_state=retry_state)"
" File "/usr/lib/python3.6/site-packages/tenacity/__init__.py"
line 319, in iter"
" return fut.result()"
" File "/usr/lib64/python3.6/concurrent/futures/_base.py"
line 425, in result"
" return self.__get_result()"
" File "/usr/lib64/python3.6/concurrent/futures/_base.py"
line 384, in __get_result"
" raise self._exception"
" File "/usr/lib/python3.6/site-packages/tenacity/__init__.py"
line 361, in call"
" result = fn(*args, **kwargs)"
" File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py"
line 732, in authenticate"
" get_cached_bearer_token(self.lock, scope)"
" File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py"
line 316, in get_cached_bearer_token"
" if (now - token_time).seconds < data.get('expires_in'):"
"TypeError: '<' not supported between instances of 'int' and 'NoneType'"
"""""
""
"The above exception was the direct cause of the following exception:"
""
"Traceback (most recent call last):"
" File "/usr/bin/tripleo-container-image-prepare"
line 138, in <module>"
" lock=lock)"
" File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py"
line 241, in container_images_prepare_multi"
" uploader.upload()"
" File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py"
line 573, in upload"
" uploader.run_tasks()"
" File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py"
line 2620, in run_tasks"
" for result in p.map(upload_task, self.upload_tasks):"
" File "/usr/lib64/python3.6/concurrent/futures/process.py"
line 366, in _chain_from_iterable_of_lists"
" for element in iterable:"
" File "/usr/lib64/python3.6/concurrent/futures/_base.py"
line 586, in result_iterator"
" yield fs.pop().result()"
" File "/usr/lib64/python3.6/concurrent/futures/_base.py"
line 432, in result"
" return self.__get_result()"
" File "/usr/lib64/python3.6/concurrent/futures/_base.py"
line 384, in __get_result"
" raise self._exception"
"TypeError: '<' not supported between instances of 'int' and 'NoneType'"], "stdout": ""
"stdout_lines": []}
2021-01-14 17:43:50.962574 | 52540070-da86-bc14-3777-000000000c8d | TIMING | Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log | undercloud-0 | 0:03:59.879271 | 7.20s
New findings: After setting the tag to 16.2 explicitly on containers-prepare-parameter.yaml, the undercloud deployment passed Note: in 16.1 no need for explicitly setting tag to 16.1 That sounds like we might be missing the downstream tag patch. Will look this week. Adding tag bypasses the tag lookup so it doesn't exercise the code that is failing. I'll look into why the authentication code is unhappy. UC deployment via satellite passed without explicit tag Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483 |