Created attachment 1458420 [details] undercloud-sos Description of problem: I am hitting this inconsistent problem when preparing overcloud images: openstack overcloud container image prepare \ --namespace=registry.access.redhat.com/rhosp13 \ --push-destination=172.16.0.11:8787 \ --prefix=openstack- \ --tag-from-label {version}-{release} \ --output-env-file=/home/stack/templates/overcloud_images.yaml \ --output-images-file /home/stack/local_registry_images.yaml Error inspecting image: docker://registry.access.redhat.com/rhosp13/openstack-aodh-listener:latest time="2018-07-12T09:24:28-04:00" level=fatal msg="Get https://registry.access.redhat.com/v2/rhosp13/openstack-aodh-listener/manifests/latest: read tcp 10.9.65.105:60672->10.4.204.31:443: read: connection reset by peer" The inconsistency is that it might work fine on a first try, but it might as well error out on first 2 tries and work on the 3rd try. It almost seems like there is multiple docker repository with not all of them being operational and load balancing puts me on random ones and not re-tries at failure. I am repeating this process over and over again and it's all scripted on my end, so there is little to no room for user error (unless my script is wrong). Also the service that the error complains about is usually different. In this case it's openstack-aodh-listener, but I have seen others as well Version-Release number of selected component (if applicable): OSP13 How reproducible: inconsistent Steps to Reproduce: 1.Run the upgrade process from OSP10 to OSP13 2.Invoke openstack overcloud container image prepare as part of the process Actual results: Error inspecting image: docker://registry.access.redhat.com/rhosp13/openstack....read: connection reset by peer" Expected results: successful completion .. maybe even after multiple re-tries if necessary Additional info: attaching sos from undercloud
I think what we want here is a backport of https://review.openstack.org/#/c/579734/. I'll look at this one today.
FYI - I just hit it again on another env with different service: Error inspecting image: docker://registry.access.redhat.com/rhosp13/openstack-neutron-metadata-agent:latest time="2018-07-12T12:39:36-04:00" level=fatal msg="Get https://registry.access.redhat.com/v2/rhosp13/openstack-neutron-metadata-agent/manifests/latest: read tcp 10.9.65.117:51226->10.4.204.31:443: read: connection reset by peer"
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible. If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-". To add draft documentation text: * Select the documentation type from the "Doc Type" drop down field. * A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2574