Bug 1600559

Created attachment 1458420 [details]
undercloud-sos

Description of problem:
I am hitting this inconsistent problem when preparing overcloud images:

openstack overcloud container image prepare \
  --namespace=registry.access.redhat.com/rhosp13 \
  --push-destination=172.16.0.11:8787 \
  --prefix=openstack- \
  --tag-from-label {version}-{release} \
  --output-env-file=/home/stack/templates/overcloud_images.yaml \
  --output-images-file /home/stack/local_registry_images.yaml

Error inspecting image: docker://registry.access.redhat.com/rhosp13/openstack-aodh-listener:latest
time="2018-07-12T09:24:28-04:00" level=fatal msg="Get https://registry.access.redhat.com/v2/rhosp13/openstack-aodh-listener/manifests/latest: read tcp 10.9.65.105:60672->10.4.204.31:443: read: connection reset by peer" 

The inconsistency is that it might work fine on a first try, but it might as well error out on first 2 tries and work on the 3rd try.
It almost seems like there is multiple docker repository with not all of them being operational and load balancing puts me on random ones and not re-tries at failure.

I am repeating this process over and over again and it's all scripted on my end, so there is little to no room for user error (unless my script is wrong).

Also the service that the error complains about is usually different. In this case it's openstack-aodh-listener, but I have seen others as well


Version-Release number of selected component (if applicable):
OSP13

How reproducible:
inconsistent

Steps to Reproduce:
1.Run the upgrade process from OSP10 to OSP13
2.Invoke openstack overcloud container image prepare as part of the process


Actual results:
Error inspecting image: docker://registry.access.redhat.com/rhosp13/openstack....read: connection reset by peer" 

Expected results:
successful completion .. maybe even after multiple re-tries if necessary 

Additional info:
attaching sos from undercloud