Bug 2165527
| Summary: | Upgrade [OSP16.2 -> OSP17.1] overcloud prepare containers failed with 401 client error UNAUTHORIZED | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Juan Badia Payno <jbadiapa> |
| Component: | tripleo-ansible | Assignee: | Lukas Bezdicka <lbezdick> |
| Status: | CLOSED ERRATA | QA Contact: | Khomesh Thakre <kthakre> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 17.1 (Wallaby) | CC: | aromito, jpretori, jslagle, kthakre, lbezdick, lmiccini, mtomaska, ramishra, sbaker, sgolovat, tkajinam |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 17.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | tripleo-ansible-3.3.1-1.20230224221431.6976f3f.el9ost openstack-tripleo-common-15.4.1-1.20230330012250.0cdddae.el9ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-08-16 01:13:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Root cause is:
with session.get(source_blob_url,
stream=True,
timeout=30,
allow_redirects=False) as blob_req:
timeout seems to be too small, session gets closed.
Nope, that is not the cause. Notice in uploaded logs Auth: where Bearer token gets lost. Do we know version of the tripleo-common package installed in undercloud ? I'm not sure what is used to provide that internal registry, but I'd like to check whether https://review.opendev.org/c/openstack/tripleo-common/+/865247 is included. If this is being triggered by the tripleo_container_image_prepare ansible module, it is using the ProcessLock[1], which results in a ProcessPoolExecutor. Process based concurrency was only added to support python2 and Mistral[2] so it would be worth checking with Rabi to see if there is a requirement that ansible use it. It might be worth trying removing the line in [1] to use the default thread based lock. [1] https://opendev.org/openstack/tripleo-ansible/src/branch/stable/wallaby/tripleo_ansible/ansible_plugins/modules/tripleo_container_image_prepare.py#L147 [2] https://opendev.org/openstack/tripleo-common/commit/60afc0eec44f698dd95d9d6ec80dad94a4b07329 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |
2023-01-27 11:18:36 | PLAY [External upgrade step 2] ************************************************* 2023-01-27 11:18:36 | ^M 2023-01-27 11:18:36 | PLAY [External deploy step 1] ************************************************** 2023-01-27 11:18:36 | 2023-01-27 11:18:36.332444 | 525400be-5980-2533-9171-000000000406 | TASK | Container image prepare 2023-01-27 11:20:01 | 2023-01-27 11:20:01.669311 | 525400be-5980-2533-9171-000000000406 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "401 Client Error: UNAUTHORIZED for url: https://registry-proxy.engineering.redhat.com/v2/rh-osbs/rhosp17-openstack-nova-libvirt/blobs/sha256:77f7bdc926ca7bd6ce09e9368f6ef5af7aa82a98f379d20b5532c886d106ca5a", "msg": "Error running container image prepare: 401 Client Error: UNAUTHORIZED for url: https://registry-proxy.engineering.redhat.com/v2/rh-osbs/rhosp17-openstack-nova-libvirt/blobs/sha256:77f7bdc926ca7bd6ce09e9368f6ef5af7aa82a98f379d20b5532c886d106ca5a", "params": {}, "success": false} 2023-01-27 11:20:01 | 2023-01-27 11:20:01.673849 | 525400be-5980-2533-9171-000000000406 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:01:28.865738 | 85.34s 2023-01-27 11:20:01 | ^M 2023-01-27 11:20:01 | NO MORE HOSTS LEFT ************************************************************* 2023-01-27 11:20:01 | ^M 2023-01-27 11:20:01 | PLAY RECAP ********************************************************************* 2023-01-27 11:20:01 | computehci-0 : ok=5 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=2 2023-01-27 11:20:01 | computehci-1 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=2 2023-01-27 11:20:01 | computehci-2 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=2 2023-01-27 11:20:01 | controller-0 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=2 2023-01-27 11:20:01 | controller-1 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=2 2023-01-27 11:20:01 | controller-2 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=2 2023-01-27 11:20:01 | localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 2023-01-27 11:20:01 | undercloud : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=2 2023-01-27 11:20:01 | 2023-01-27 11:20:01.684666 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-01-27 11:20:01 | 2023-01-27 11:20:01.685158 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 7 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-01-27 11:20:01 | 2023-01-27 11:20:01.685657 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:01:28.877558 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-01-27 11:20:01 | 2023-01-27 11:20:01.686135 | UUID | Info | Host | Task Name | Run Time 2023-01-27 11:20:01 | 2023-01-27 11:20:01.686677 | 525400be-5980-2533-9171-000000000406 | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 85.34s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.687135 | 525400be-5980-2533-9171-000000000614 | SUMMARY | localhost | Gathering Facts | 0.61s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.687640 | 525400be-5980-2533-9171-000000000024 | SUMMARY | controller-0 | Set all_nodes data as group_vars for overcloud | 0.07s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.688137 | 525400be-5980-2533-9171-000000000024 | SUMMARY | computehci-2 | Set all_nodes data as group_vars for overcloud | 0.06s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.688623 | 525400be-5980-2533-9171-00000000001a | SUMMARY | controller-0 | ansible.builtin.include_vars | 0.06s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.689116 | 525400be-5980-2533-9171-00000000001a | SUMMARY | undercloud | ansible.builtin.include_vars | 0.06s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.689586 | 525400be-5980-2533-9171-00000000001a | SUMMARY | controller-1 | ansible.builtin.include_vars | 0.06s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.690071 | 525400be-5980-2533-9171-00000000001a | SUMMARY | computehci-0 | ansible.builtin.include_vars | 0.06s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.690560 | 525400be-5980-2533-9171-000000000024 | SUMMARY | computehci-1 | Set all_nodes data as group_vars for overcloud | 0.05s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.691111 | 525400be-5980-2533-9171-00000000001a | SUMMARY | controller-2 | ansible.builtin.include_vars | 0.05s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.691616 | 525400be-5980-2533-9171-000000000024 | SUMMARY | computehci-0 | Set all_nodes data as group_vars for overcloud | 0.05s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.692146 | 525400be-5980-2533-9171-000000000024 | SUMMARY | controller-1 | Set all_nodes data as group_vars for overcloud | 0.04s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.692615 | 525400be-5980-2533-9171-00000000001d | SUMMARY | computehci-0 | Include Service VIP vars | 0.04s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.693081 | 525400be-5980-2533-9171-00000000001d | SUMMARY | controller-2 | Include Service VIP vars | 0.04s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.693650 | 525400be-5980-2533-9171-00000000001a | SUMMARY | computehci-1 | ansible.builtin.include_vars | 0.03s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.694162 | 525400be-5980-2533-9171-00000000001a | SUMMARY | computehci-2 | ansible.builtin.include_vars | 0.03s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.694756 | 525400be-5980-2533-9171-00000000001d | SUMMARY | controller-0 | Include Service VIP vars | 0.03s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.695210 | 525400be-5980-2533-9171-00000000001e | SUMMARY | computehci-2 | Include OVN bridge MAC address variables | 0.03s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.695720 | 525400be-5980-2533-9171-00000000001d | SUMMARY | controller-1 | Include Service VIP vars | 0.03s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.696197 | 525400be-5980-2533-9171-00000000001e | SUMMARY | controller-1 | Include OVN bridge MAC address variables | 0.03s 2023-01-27 11:20:01 | 2023-01-27 11:20:01.696745 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-01-27 11:20:01 | 2023-01-27 11:20:01.697272 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-01-27 11:20:01 | 2023-01-27 11:20:01.697785 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2023-01-27 11:20:01 | 2023-01-27 11:20:01.698254 | The following node(s) had failures: undercloud 2023-01-27 11:20:01 | 2023-01-27 11:20:01.698749 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-01-27 11:20:01 | 2023-01-27 11:20:01.940 458510 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleo07gfd1ei ] cleaned up^[[00m 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh^[[00m 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun Traceback (most recent call last): 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun super(Command, self).run(parsed_args) 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun return super(Command, self).run(parsed_args) 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun File "/usr/lib/python3.6/site-packages/cliff/command.py", line 186, in run 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun return_code = self.take_action(parsed_args) or 0 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_external_upgrade.py", line 146, in take_action 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun reproduce_command=True 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 735, in run_ansible_playbook 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun raise RuntimeError(err_msg) 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh 2023-01-27 11:20:01 | 2023-01-27 11:20:01.941 458510 ERROR tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun ^[[00m 2023-01-27 11:20:01 | 2023-01-27 11:20:01.943 458510 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/tripleo-multi-playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/qe-Cloud-0/config-download/ansible-playbook-command.sh^[[00m 2023-01-27 11:20:01 | 2023-01-27 11:20:01.944 458510 INFO osc_lib.shell [-] END return value: 1^[[00m 2023-01-27 11:20:01 | /usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead. 2023-01-27 11:20:01 | % (name, name, name)) cat containers-prepare-parameter.yaml parameter_defaults: ComputeHCIContainerImagePrepare: &id001 - excludes: - nova-compute - nova-libvirt - ovn-controller push_destination: undercloud-0.ctlplane.redhat.local:8787 set: ceph_alertmanager_image: openshift-ose-prometheus-alertmanager ceph_alertmanager_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_alertmanager_tag: v4.10 ceph_grafana_image: grafana ceph_grafana_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_grafana_tag: latest ceph_image: rhceph ceph_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_node_exporter_image: openshift-ose-prometheus-node-exporter ceph_node_exporter_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_node_exporter_tag: v4.10 ceph_prometheus_image: openshift-ose-prometheus ceph_prometheus_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_prometheus_tag: v4.10 ceph_tag: 5-359 name_prefix: rhosp17-openstack- name_suffix: '' namespace: registry-proxy.engineering.redhat.com/rh-osbs neutron_driver: ovn rhel_containers: false tag: 17.1_20221129.1 tag_from_label: '{version}-{release}' - includes: - nova-compute - nova-libvirt - ovn-controller push_destination: undercloud-0.ctlplane.redhat.local:8787 set: ceph_alertmanager_image: openshift-ose-prometheus-alertmanager ceph_alertmanager_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_alertmanager_tag: v4.10 ceph_grafana_image: grafana ceph_grafana_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_grafana_tag: latest ceph_image: rhceph ceph_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_node_exporter_image: openshift-ose-prometheus-node-exporter ceph_node_exporter_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_node_exporter_tag: v4.10 ceph_prometheus_image: openshift-ose-prometheus ceph_prometheus_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_prometheus_tag: v4.10 ceph_tag: 5-359 name_prefix: rhosp17-openstack- name_suffix: '' namespace: registry-proxy.engineering.redhat.com/rh-osbs neutron_driver: ovn rhel_containers: false tag: 17.1_20230116.1 tag_from_label: '{version}-{release}' ContainerImagePrepare: - push_destination: undercloud-0.ctlplane.redhat.local:8787 set: ceph_alertmanager_image: openshift-ose-prometheus-alertmanager ceph_alertmanager_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_alertmanager_tag: v4.10 ceph_grafana_image: grafana ceph_grafana_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_grafana_tag: latest ceph_image: rhceph ceph_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_node_exporter_image: openshift-ose-prometheus-node-exporter ceph_node_exporter_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_node_exporter_tag: v4.10 ceph_prometheus_image: openshift-ose-prometheus ceph_prometheus_namespace: registry-proxy.engineering.redhat.com/rh-osbs ceph_prometheus_tag: v4.10 ceph_tag: 5-359 name_prefix: rhosp17-openstack- name_suffix: '' namespace: registry-proxy.engineering.redhat.com/rh-osbs neutron_driver: ovn rhel_containers: false tag: 17.1_20221129.1 tag_from_label: '{version}-{release}' ControllerContainerImagePrepare: *id001 the /var/log/tripleo-container-image-prepare.log 2023-01-27 10:55:01,885 360065 DEBUG urllib3.connectionpool [ ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-neutron-dhcp-agent/blobs/uploads/ HTTP/1.1" 404 196 2023-01-27 10:55:01,886 360073 DEBUG urllib3.connectionpool [ ] http://undercloud-0.ctlplane.redhat.local:8787 "GET /v2/ HTTP/1.1" 200 2 2023-01-27 10:55:01,886 360065 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id a0af8437e82e0d46f367d162b4aa4788131d6232, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> 2023-01-27 10:55:01,886 360072 DEBUG urllib3.connectionpool [ ] http://undercloud-0.ctlplane.redhat.local:8787 "GET /v2/ HTTP/1.1" 200 2 2023-01-27 10:55:01,886 360065 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None) 2023-01-27 10:55:01,886 360073 DEBUG tripleo_common.image.image_uploader [ ] http://undercloud-0.ctlplane.redhat.local:8787/v2/ status code 200 2023-01-27 10:55:01,887 360072 DEBUG tripleo_common.image.image_uploader [ ] http://undercloud-0.ctlplane.redhat.local:8787/v2/ status code 200 2023-01-27 10:55:01,888 360070 DEBUG urllib3.connectionpool [ ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-memcached/blobs/uploads/ HTTP/1.1" 404 196 2023-01-27 10:55:01,888 360070 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 1bf240041c9527f54c913e74541aa9ef0413a7c3, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> 2023-01-27 10:55:01,888 360065 DEBUG urllib3.connectionpool [ ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443 2023-01-27 10:55:01,889 360070 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None) 2023-01-27 10:55:01,889 360073 DEBUG urllib3.connectionpool [ ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-neutron-openvswitch-agent/blobs/uploads/ HTTP/1.1" 404 196 2023-01-27 10:55:01,889 360072 DEBUG urllib3.connectionpool [ ] http://undercloud-0.ctlplane.redhat.local:8787 "POST /v2/rh-osbs/rhosp17-openstack-mariadb/blobs/uploads/ HTTP/1.1" 404 196 2023-01-27 10:55:01,889 360073 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id fd2e2aed508732158283e111feb328a5ee3e4c0a, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> 2023-01-27 10:55:01,889 360073 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id fd2e2aed508732158283e111feb328a5ee3e4c0a, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> 2023-01-27 10:55:01,890 360072 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 79abc26db456c4f883f9ab8c024a510f064b6b94, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> 2023-01-27 10:55:01,890 360073 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None) 2023-01-27 10:55:01,890 360072 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None) 2023-01-27 10:55:01,891 360070 DEBUG urllib3.connectionpool [ ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443 2023-01-27 10:55:01,892 360073 DEBUG urllib3.connectionpool [ ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443 2023-01-27 10:55:01,893 360072 DEBUG urllib3.connectionpool [ ] Starting new HTTPS connection (1): registry-proxy.engineering.redhat.com:443 2023-01-27 10:55:01,962 360065 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4 2023-01-27 10:55:01,963 360065 DEBUG tripleo_common.image.image_uploader [ ] https://registry-proxy.engineering.redhat.com/v2/ status code 401 2023-01-27 10:55:01,963 360065 DEBUG tripleo_common.image.image_uploader [ ] Using bearer token auth 2023-01-27 10:55:01,971 360072 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4 2023-01-27 10:55:01,971 360070 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4 2023-01-27 10:55:01,972 360072 DEBUG tripleo_common.image.image_uploader [ ] https://registry-proxy.engineering.redhat.com/v2/ status code 401 2023-01-27 10:55:01,972 360070 DEBUG tripleo_common.image.image_uploader [ ] https://registry-proxy.engineering.redhat.com/v2/ status code 401 2023-01-27 10:55:01,973 360072 DEBUG tripleo_common.image.image_uploader [ ] Using bearer token auth 2023-01-27 10:55:01,973 360073 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/ HTTP/1.1" 401 4 2023-01-27 10:55:01,973 360070 DEBUG tripleo_common.image.image_uploader [ ] Using bearer token auth 2023-01-27 10:55:01,974 360073 DEBUG tripleo_common.image.image_uploader [ ] https://registry-proxy.engineering.redhat.com/v2/ status code 401 2023-01-27 10:55:01,974 360073 DEBUG tripleo_common.image.image_uploader [ ] Using bearer token auth 2023-01-27 10:55:02,027 360065 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-neutron-dhcp-agent%3Apull HTTP/1.1" 200 1134 2023-01-27 10:55:02,028 360065 DEBUG tripleo_common.image.image_uploader [ ] Session authenticated: id f3c0a0ccf94c193bb04adbabb35ae955e3b8895f 2023-01-27 10:55:02,052 360073 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-neutron-openvswitch-agent%3Apull HTTP/1.1" 200 1153 2023-01-27 10:55:02,052 360072 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-mariadb%3Apull HTTP/1.1" 200 1105 2023-01-27 10:55:02,054 360073 DEBUG tripleo_common.image.image_uploader [ ] Session authenticated: id 86afc1d77dfc9a486f4c053c6aefa2b577ff04a2 2023-01-27 10:55:02,055 360072 DEBUG tripleo_common.image.image_uploader [ ] Session authenticated: id 95c4e4d73d0b78b852e9c23422db7c3485bab2ca 2023-01-27 10:55:02,074 360070 DEBUG urllib3.connectionpool [ ] https://registry-proxy.engineering.redhat.com:443 "GET /v2/auth?scope=repository%3Arh-osbs%2Frhosp17-openstack-memcached%3Apull HTTP/1.1" 200 1110