2165527 – Upgrade [OSP16.2 -> OSP17.1] overcloud prepare containers failed with 401 client error UNAUTHORIZED

Bug 2165527 - Upgrade [OSP16.2 -> OSP17.1] overcloud prepare containers failed with 401 client error UNAUTHORIZED

Summary: Upgrade [OSP16.2 -> OSP17.1] overcloud prepare containers failed with 401 cli...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	tripleo-ansible
Sub Component:
Version:	17.1 (Wallaby)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	beta
Target Release:	17.1
Assignee:	Lukas Bezdicka
QA Contact:	Khomesh Thakre
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-01-30 11:19 UTC by Juan Badia Payno
Modified:	2023-12-15 04:25 UTC (History)
CC List:	11 users (show)
Fixed In Version:	tripleo-ansible-3.3.1-1.20230224221431.6976f3f.el9ost openstack-tripleo-common-15.4.1-1.20230330012250.0cdddae.el9ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-08-16 01:13:42 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	872194	None	MERGED	Set workers to 1	2023-03-29 13:12:09 UTC
OpenStack gerrit	873039	None	MERGED	Use default thread locking for container image prepare	2023-03-22 09:18:02 UTC
Red Hat Issue Tracker	OSP-21880	None	None	None	2023-01-30 11:20:41 UTC
Red Hat Product Errata	RHEA-2023:4577	None	None	None	2023-08-16 01:14:10 UTC

Description Juan Badia Payno 2023-01-30 11:19:45 UTC

2023-01-27 11:18:36 | PLAY [External upgrade step 2] *************************************************
2023-01-27 11:18:36 | ^M
2023-01-27 11:18:36 | PLAY [External deploy step 1] **************************************************
2023-01-27 11:18:36 | 2023-01-27 11:18:36.332444 | 525400be-5980-2533-9171-000000000406 |       TASK | Container image prepare
2023-01-27 11:20:01 | 2023-01-27 11:20:01.669311 | 525400be-5980-2533-9171-000000000406 |      FATAL | Container image prepare | undercloud | error={"changed": false, "error": "401 Client Error: UNAUTHORIZED for url: https://registry-proxy.engineering.redhat.com/v2/rh-osbs/rhosp17-openstack-nova-libvirt/blobs/sha256:77f7bdc926ca7bd6ce09e9368f6ef5af7aa82a98f379d20b5532c886d106ca5a", "msg": "Error running container image prepare: 401 Client Error: UNAUTHORIZED for url: https://registry-proxy.engineering.redhat.com/v2/rh-osbs/rhosp17-openstack-nova-libvirt/blobs/sha256:77f7bdc926ca7bd6ce09e9368f6ef5af7aa82a98f379d20b5532c886d106ca5a", "params": {}, "success": false}
2023-01-27 11:20:01 | 2023-01-27 11:20:01.673849 | 525400be-5980-2533-9171-000000000406 |     TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:01:28.865738 | 85.34s
2023-01-27 11:20:01 | ^M
2023-01-27 11:20:01 | NO MORE HOSTS LEFT *************************************************************
2023-01-27 11:20:01 | ^M
2023-01-27 11:20:01 | PLAY RECAP *********************************************************************
2023-01-27 11:20:01 | computehci-0               : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2
2023-01-27 11:20:01 | computehci-1               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2
2023-01-27 11:20:01 | computehci-2               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2
2023-01-27 11:20:01 | controller-0               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2
2023-01-27 11:20:01 | controller-1               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2
2023-01-27 11:20:01 | controller-2               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2
2023-01-27 11:20:01 | localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
2023-01-27 11:20:01 | undercloud                 : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=2
2023-01-27 11:20:01 | 2023-01-27 11:20:01.684666 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-01-27 11:20:01 | 2023-01-27 11:20:01.685158 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 7          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-01-27 11:20:01 | 2023-01-27 11:20:01.685657 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:01:28.877558 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-01-27 11:20:01 | 2023-01-27 11:20:01.686135 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2023-01-27 11:20:01 | 2023-01-27 11:20:01.686677 | 525400be-5980-2533-9171-000000000406 |    SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 85.34s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.687135 | 525400be-5980-2533-9171-000000000614 |    SUMMARY |  localhost | Gathering Facts | 0.61s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.687640 | 525400be-5980-2533-9171-000000000024 |    SUMMARY | controller-0 | Set all_nodes data as group_vars for overcloud | 0.07s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.688137 | 525400be-5980-2533-9171-000000000024 |    SUMMARY | computehci-2 | Set all_nodes data as group_vars for overcloud | 0.06s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.688623 | 525400be-5980-2533-9171-00000000001a |    SUMMARY | controller-0 | ansible.builtin.include_vars | 0.06s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.689116 | 525400be-5980-2533-9171-00000000001a |    SUMMARY | undercloud | ansible.builtin.include_vars | 0.06s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.689586 | 525400be-5980-2533-9171-00000000001a |    SUMMARY | controller-1 | ansible.builtin.include_vars | 0.06s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.690071 | 525400be-5980-2533-9171-00000000001a |    SUMMARY | computehci-0 | ansible.builtin.include_vars | 0.06s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.690560 | 525400be-5980-2533-9171-000000000024 |    SUMMARY | computehci-1 | Set all_nodes data as group_vars for overcloud | 0.05s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.691111 | 525400be-5980-2533-9171-00000000001a |    SUMMARY | controller-2 | ansible.builtin.include_vars | 0.05s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.691616 | 525400be-5980-2533-9171-000000000024 |    SUMMARY | computehci-0 | Set all_nodes data as group_vars for overcloud | 0.05s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.692146 | 525400be-5980-2533-9171-000000000024 |    SUMMARY | controller-1 | Set all_nodes data as group_vars for overcloud | 0.04s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.692615 | 525400be-5980-2533-9171-00000000001d |    SUMMARY | computehci-0 | Include Service VIP vars | 0.04s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.693081 | 525400be-5980-2533-9171-00000000001d |    SUMMARY | controller-2 | Include Service VIP vars | 0.04s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.693650 | 525400be-5980-2533-9171-00000000001a |    SUMMARY | computehci-1 | ansible.builtin.include_vars | 0.03s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.694162 | 525400be-5980-2533-9171-00000000001a |    SUMMARY | computehci-2 | ansible.builtin.include_vars | 0.03s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.694756 | 525400be-5980-2533-9171-00000000001d |    SUMMARY | controller-0 | Include Service VIP vars | 0.03s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.695210 | 525400be-5980-2533-9171-00000000001e |    SUMMARY | computehci-2 | Include OVN bridge MAC address variables | 0.03s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.695720 | 525400be-5980-2533-9171-00000000001d |    SUMMARY | controller-1 | Include Service VIP vars | 0.03s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.696197 | 525400be-5980-2533-9171-00000000001e |    SUMMARY | controller-1 | Include OVN bridge MAC address variables | 0.03s
2023-01-27 11:20:01 | 2023-01-27 11:20:01.696745 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-01-27 11:20:01 | 2023-01-27 11:20:01.697272 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-01-27 11:20:01 | 2023-01-27 11:20:01.697785 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2023-01-27 11:20:01 | 2023-01-27 11:20:01.698254 |  The following node(s) had failures: undercloud
2023-01-27 11:20:01 | 2023-01-27 11:20:01.698749 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-01-27 11:20:01 | 2023-01-27 11:20:01.940 458510 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleo07gfd1ei ] cleaned up^[[00m
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh^[[00m
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun Traceback (most recent call last):
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun   File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun     super(Command, self).run(parsed_args)
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun   File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun     return super(Command, self).run(parsed_args)
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun   File "/usr/lib/python3.6/site-packages/cliff/command.py", line 186, in run
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun     return_code = self.take_action(parsed_args) or 0
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_external_upgrade.py", line 146, in take_action
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun     reproduce_command=True
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 735, in run_ansible_playbook
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun     raise RuntimeError(err_msg)
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh
2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun ^[[00m
2023-01-27 11:20:01 | 2023-01-27 11:20:01.943 458510 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh^[[00m
2023-01-27 11:20:01 | 2023-01-27 11:20:01.944 458510 INFO osc_lib.shell [-] END return value: 1^[[00m
2023-01-27 11:20:01 | /usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
2023-01-27 11:20:01 |   % (name, name, name))


cat containers-prepare-parameter.yaml
parameter_defaults:
  ComputeHCIContainerImagePrepare: &id001
  - excludes:
    - nova-compute
    - nova-libvirt
    - ovn-controller
    push_destination: undercloud-0.ctlplane.redhat.local:8787
    set:
      ceph_alertmanager_image: openshift-ose-prometheus-alertmanager
      ceph_alertmanager_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_alertmanager_tag: v4.10
      ceph_grafana_image: grafana
      ceph_grafana_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_grafana_tag: latest
      ceph_image: rhceph
      ceph_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_node_exporter_image: openshift-ose-prometheus-node-exporter
      ceph_node_exporter_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_node_exporter_tag: v4.10
      ceph_prometheus_image: openshift-ose-prometheus
      ceph_prometheus_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_prometheus_tag: v4.10
      ceph_tag: 5-359
      name_prefix: rhosp17-openstack-
      name_suffix: ''
      namespace: registry-proxy.engineering.redhat.com/rh-osbs
      neutron_driver: ovn
      rhel_containers: false
      tag: 17.1_20221129.1
    tag_from_label: '{version}-{release}'
  - includes:
    - nova-compute
    - nova-libvirt
    - ovn-controller
    push_destination: undercloud-0.ctlplane.redhat.local:8787
    set:
      ceph_alertmanager_image: openshift-ose-prometheus-alertmanager
      ceph_alertmanager_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_alertmanager_tag: v4.10
      ceph_grafana_image: grafana
      ceph_grafana_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_grafana_tag: latest
      ceph_image: rhceph
      ceph_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_node_exporter_image: openshift-ose-prometheus-node-exporter
      ceph_node_exporter_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_node_exporter_tag: v4.10
      ceph_prometheus_image: openshift-ose-prometheus
      ceph_prometheus_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_prometheus_tag: v4.10
      ceph_tag: 5-359
      name_prefix: rhosp17-openstack-
      name_suffix: ''
      namespace: registry-proxy.engineering.redhat.com/rh-osbs
      neutron_driver: ovn
      rhel_containers: false
      tag: 17.1_20230116.1
    tag_from_label: '{version}-{release}'
  ContainerImagePrepare:
  - push_destination: undercloud-0.ctlplane.redhat.local:8787
    set:
      ceph_alertmanager_image: openshift-ose-prometheus-alertmanager
      ceph_alertmanager_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_alertmanager_tag: v4.10
      ceph_grafana_image: grafana
      ceph_grafana_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_grafana_tag: latest
      ceph_image: rhceph
      ceph_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_node_exporter_image: openshift-ose-prometheus-node-exporter
      ceph_node_exporter_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_node_exporter_tag: v4.10
      ceph_prometheus_image: openshift-ose-prometheus
      ceph_prometheus_namespace: registry-proxy.engineering.redhat.com/rh-osbs
      ceph_prometheus_tag: v4.10
      ceph_tag: 5-359
      name_prefix: rhosp17-openstack-
      name_suffix: ''
      namespace: registry-proxy.engineering.redhat.com/rh-osbs
      neutron_driver: ovn
      rhel_containers: false
      tag: 17.1_20221129.1
    tag_from_label: '{version}-{release}'
  ControllerContainerImagePrepare: *id001

the /var/log/tripleo-container-image-prepare.log
2023-01-27 10:55:01,885 360065 DEBUG urllib3.connectionpool [  ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-neutron-dhcp-agent/blobs/uploads/ HTTP/1.1" 404 196
2023-01-27 10:55:01,886 360073 DEBUG urllib3.connectionpool [  ] http://undercloud-0.ctlplane.redhat.local:8787 "GET /v2/ HTTP/1.1" 200 2
2023-01-27 10:55:01,886 360065 INFO tripleo_common.image.image_uploader [  ] Non-2xx: id a0af8437e82e0d46f367d162b4aa4788131d6232, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

2023-01-27 10:55:01,886 360072 DEBUG urllib3.connectionpool [  ] http://undercloud-0.ctlplane.redhat.local:8787 "GET /v2/ HTTP/1.1" 200 2
2023-01-27 10:55:01,886 360065 DEBUG urllib3.util.retry [  ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2023-01-27 10:55:01,886 360073 DEBUG tripleo_common.image.image_uploader [  ] http://undercloud-0.ctlplane.redhat.local:8787/v2/ status code 200
2023-01-27 10:55:01,887 360072 DEBUG tripleo_common.image.image_uploader [  ] http://undercloud-0.ctlplane.redhat.local:8787/v2/ status code 200
2023-01-27 10:55:01,888 360070 DEBUG urllib3.connectionpool [  ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-memcached/blobs/uploads/ HTTP/1.1" 404 196
2023-01-27 10:55:01,888 360070 INFO tripleo_common.image.image_uploader [  ] Non-2xx: id 1bf240041c9527f54c913e74541aa9ef0413a7c3, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

2023-01-27 10:55:01,888 360065 DEBUG urllib3.connectionpool [  ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443
2023-01-27 10:55:01,889 360070 DEBUG urllib3.util.retry [  ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2023-01-27 10:55:01,889 360073 DEBUG urllib3.connectionpool [  ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-neutron-openvswitch-agent/blobs/uploads/ HTTP/1.1" 404 196
2023-01-27 10:55:01,889 360072 DEBUG urllib3.connectionpool [  ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-mariadb/blobs/uploads/ HTTP/1.1" 404 196
2023-01-27 10:55:01,889 360073 INFO tripleo_common.image.image_uploader [  ] Non-2xx: id fd2e2aed508732158283e111feb328a5ee3e4c0a, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
2023-01-27 10:55:01,889 360073 INFO tripleo_common.image.image_uploader [  ] Non-2xx: id fd2e2aed508732158283e111feb328a5ee3e4c0a, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

2023-01-27 10:55:01,890 360072 INFO tripleo_common.image.image_uploader [  ] Non-2xx: id 79abc26db456c4f883f9ab8c024a510f064b6b94, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

2023-01-27 10:55:01,890 360073 DEBUG urllib3.util.retry [  ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2023-01-27 10:55:01,890 360072 DEBUG urllib3.util.retry [  ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2023-01-27 10:55:01,891 360070 DEBUG urllib3.connectionpool [  ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443
2023-01-27 10:55:01,892 360073 DEBUG urllib3.connectionpool [  ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443
2023-01-27 10:55:01,893 360072 DEBUG urllib3.connectionpool [  ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443
2023-01-27 10:55:01,962 360065 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4
2023-01-27 10:55:01,963 360065 DEBUG tripleo_common.image.image_uploader [  ] https://registry-proxy.engineering.redhat.com/v2/ status code 401
2023-01-27 10:55:01,963 360065 DEBUG tripleo_common.image.image_uploader [  ] Using bearer token auth
2023-01-27 10:55:01,971 360072 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4
2023-01-27 10:55:01,971 360070 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4
2023-01-27 10:55:01,972 360072 DEBUG tripleo_common.image.image_uploader [  ] https://registry-proxy.engineering.redhat.com/v2/ status code 401
2023-01-27 10:55:01,972 360070 DEBUG tripleo_common.image.image_uploader [  ] https://registry-proxy.engineering.redhat.com/v2/ status code 401
2023-01-27 10:55:01,973 360072 DEBUG tripleo_common.image.image_uploader [  ] Using bearer token auth
2023-01-27 10:55:01,973 360073 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4
2023-01-27 10:55:01,973 360070 DEBUG tripleo_common.image.image_uploader [  ] Using bearer token auth
2023-01-27 10:55:01,974 360073 DEBUG tripleo_common.image.image_uploader [  ] https://registry-proxy.engineering.redhat.com/v2/ status code 401
2023-01-27 10:55:01,974 360073 DEBUG tripleo_common.image.image_uploader [  ] Using bearer token auth
2023-01-27 10:55:02,027 360065 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-neutron-dhcp-agent%3Apull HTTP/1.1" 200 1134
2023-01-27 10:55:02,028 360065 DEBUG tripleo_common.image.image_uploader [  ] Session authenticated: id f3c0a0ccf94c193bb04adbabb35ae955e3b8895f
2023-01-27 10:55:02,052 360073 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-neutron-openvswitch-agent%3Apull HTTP/1.1" 200 1153
2023-01-27 10:55:02,052 360072 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-mariadb%3Apull HTTP/1.1" 200 1105
2023-01-27 10:55:02,054 360073 DEBUG tripleo_common.image.image_uploader [  ] Session authenticated: id 86afc1d77dfc9a486f4c053c6aefa2b577ff04a2
2023-01-27 10:55:02,055 360072 DEBUG tripleo_common.image.image_uploader [  ] Session authenticated: id 95c4e4d73d0b78b852e9c23422db7c3485bab2ca
2023-01-27 10:55:02,074 360070 DEBUG urllib3.connectionpool [  ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-memcached%3Apull HTTP/1.1" 200 1110

Comment 1 Lukas Bezdicka 2023-01-30 16:57:20 UTC

Root cause is:
        with session.get(source_blob_url,
                         stream=True,
                         timeout=30,
                         allow_redirects=False) as blob_req:


timeout seems to be too small, session gets closed.

Comment 2 Lukas Bezdicka 2023-01-30 19:59:05 UTC

Nope, that is not the cause.

Comment 5 Lukas Bezdicka 2023-01-31 10:34:48 UTC

Notice in uploaded logs Auth: where Bearer token gets lost.

Comment 6 Takashi Kajinami 2023-02-01 07:19:20 UTC

Do we know version of the tripleo-common package installed in undercloud ?

I'm not sure what is used to provide that internal registry, but I'd like to check
whether https://review.opendev.org/c/openstack/tripleo-common/+/865247 is included.

Comment 11 Steve Baker 2023-02-07 20:39:23 UTC

If this is being triggered by the tripleo_container_image_prepare ansible module, it is using the ProcessLock[1], which results in a ProcessPoolExecutor. Process based concurrency was only added to support python2 and Mistral[2] so it would be worth checking with Rabi to see if there is a requirement that ansible use it.

It might be worth trying removing the line in [1] to use the default thread based lock.

[1] https://opendev.org/openstack/tripleo-ansible/src/branch/stable/wallaby/tripleo_ansible/ansible_plugins/modules/tripleo_container_image_prepare.py#L147
[2] https://opendev.org/openstack/tripleo-common/commit/60afc0eec44f698dd95d9d6ec80dad94a4b07329

Comment 30 errata-xmlrpc 2023-08-16 01:13:42 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577

Comment 31 Red Hat Bugzilla 2023-12-15 04:25:58 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days

Note You need to log in before you can comment on or make changes to this bug.