Description of problem: Deployment fails during 'Run tripleo-container-image-prepare' task: 2018-10-09 15:55:46,975 p=31988 u=mistral | TASK [Run tripleo-container-image-prepare] ************************************* 2018-10-09 15:55:46,975 p=31988 u=mistral | Tuesday 09 October 2018 15:55:46 -0400 (0:00:00.317) 0:04:04.977 ******* 2018-10-09 15:56:51,837 p=31988 u=mistral | [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo 2018-10-09 15:56:51,838 p=31988 u=mistral | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "sudo /usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.HFJkZf-role-data --environment-file /tmp/ansible.0W4OVR-prepare-param --cleanup partial 2> tripleo-container-image-prepare.log", "delta": "0:01:04.714610", "end": "2018-10-09 15:56:51.821230", "msg": "non-zero return code", "rc": 1, "start": "2018-10-09 15:55:47.106620", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2018-10-09 15:56:51,838 p=31988 u=mistral | NO MORE HOSTS LEFT ************************************************************* 2018-10-09 15:56:51,839 p=31988 u=mistral | PLAY RECAP ********************************************************************* 2018-10-09 15:56:51,839 p=31988 u=mistral | openshift-infra-0 : ok=82 changed=35 unreachable=0 failed=0 2018-10-09 15:56:51,839 p=31988 u=mistral | openshift-infra-1 : ok=82 changed=35 unreachable=0 failed=0 2018-10-09 15:56:51,839 p=31988 u=mistral | openshift-infra-2 : ok=83 changed=35 unreachable=0 failed=0 2018-10-09 15:56:51,839 p=31988 u=mistral | openshift-master-0 : ok=87 changed=37 unreachable=0 failed=0 2018-10-09 15:56:51,839 p=31988 u=mistral | openshift-master-1 : ok=86 changed=37 unreachable=0 failed=0 2018-10-09 15:56:51,839 p=31988 u=mistral | openshift-master-2 : ok=86 changed=37 unreachable=0 failed=0 2018-10-09 15:56:51,839 p=31988 u=mistral | openshift-worker-0 : ok=82 changed=35 unreachable=0 failed=0 2018-10-09 15:56:51,840 p=31988 u=mistral | openshift-worker-1 : ok=82 changed=35 unreachable=0 failed=0 2018-10-09 15:56:51,840 p=31988 u=mistral | openshift-worker-2 : ok=83 changed=35 unreachable=0 failed=0 2018-10-09 15:56:51,840 p=31988 u=mistral | undercloud : ok=7 changed=4 unreachable=0 failed=1 2018-10-09 15:56:51,840 p=31988 u=mistral | Tuesday 09 October 2018 15:56:51 -0400 (0:01:04.864) 0:05:09.842 ******* 2018-10-09 15:56:51,840 p=31988 u=mistral | =============================================================================== Checking tripleo-container-image-prepare.log we can see that it cannot find the 'registry.access.redhat.com/openshift3/ose-sti-builder' image: INFO:tripleo_common.image.image_uploader:Running skopeo inspect docker://registry.access.redhat.com/openshift3/ose-sti-builder:v3.10 WARNING:tripleo_common.image.image_uploader:Completed upload for image registry.access.redhat.com/rhgs3/rhgs-server-rhel7:latest INFO:tripleo_common.image.image_uploader:imagename: registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7:latest INFO:tripleo_common.image.image_uploader:Running skopeo inspect docker://registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7:latest INFO:tripleo_common.image.image_uploader:Pulling registry.access.redhat.com/openshift3/ose-sti-builder WARNING:tripleo_common.image.image_uploader:docker pull failed: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\"" INFO:tripleo_common.image.image_uploader:Running skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhgs3/rhgs-gluster-block-prov-rhel7:latest INFO:tripleo_common.image.image_uploader:Pulling registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7 INFO:tripleo_common.image.image_uploader:Pulling registry.access.redhat.com/openshift3/ose-sti-builder INFO:tripleo_common.image.image_uploader:Pushing 192.168.24.1:8787/rhgs3/rhgs-gluster-block-prov-rhel7 WARNING:tripleo_common.image.image_uploader:Completed upload for image registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7:latest WARNING:tripleo_common.image.image_uploader:docker pull failed: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\"" INFO:tripleo_common.image.image_uploader:Pulling registry.access.redhat.com/openshift3/ose-sti-builder WARNING:tripleo_common.image.image_uploader:docker pull failed: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\"" INFO:tripleo_common.image.image_uploader:Pulling registry.access.redhat.com/openshift3/ose-sti-builder WARNING:tripleo_common.image.image_uploader:docker pull failed: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\"" INFO:tripleo_common.image.image_uploader:Pulling registry.access.redhat.com/openshift3/ose-sti-builder WARNING:tripleo_common.image.image_uploader:docker pull failed: error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\"" Traceback (most recent call last): File "/usr/bin/tripleo-container-image-prepare", line 100, in <module> env, roles_data, cleanup=args.cleanup) File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 198, in container_images_prepare_multi uploader.upload() File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 132, in upload uploader.run_tasks() File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 521, in run_tasks for result in p.map(docker_upload, self.upload_tasks): File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator yield future.result() File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 549, in docker_upload return DockerImageUploader.upload_image(*args) File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 257, in upload_image DockerImageUploader._pull(dockerc, repo, tag=source_tag) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 241, in wrapped_f return self.call(f, *args, **kw) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 330, in call start_time=start_time) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 297, in iter raise retry_exc.reraise() File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 136, in reraise raise self.last_attempt.result() File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 333, in call result = fn(*args, **kwargs) File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 295, in _pull raise ImageUploaderException('Could not pull image %s' % image) tripleo_common.image.exception.ImageUploaderException: Could not pull image registry.access.redhat.com/openshift3/ose-sti-builder Version-Release number of selected component (if applicable): openstack-tripleo-common-9.3.1-0.20180923215327.d22cb3e.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. openstack overcloud deploy \ --stack openshift \ --templates \ -r /home/stack/openshift_roles_data.yaml \ -n /home/stack/network_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/openshift.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/openshift-cns.yaml \ -e /home/stack/openshift_env.yaml \ -e /home/stack/containers-prepare-parameter.yaml Actual results: Fails because ose-sti-builder 3.10 image doesn't exist. Expected results: No failure. Additional info: (undercloud) [stack@undercloud-0 ~]$ cat /home/stack/openshift_env.yaml resource_registry: OS::TripleO::Services::HAproxy: /usr/share/openstack-tripleo-heat-templates/docker/services/haproxy.yaml OS::TripleO::Services::Keepalived: /usr/share/openstack-tripleo-heat-templates//docker/services/keepalived.yaml OS::TripleO::NodeUserData: /home/stack/firstboot.yaml OS::TripleO::OpenShiftMaster::Net::SoftwareConfig: /home/stack/master-nic.yaml OS::TripleO::OpenShiftWorker::Net::SoftwareConfig: /home/stack/worker-nic.yaml OS::TripleO::OpenShiftInfra::Net::SoftwareConfig: /home/stack/infra-nic.yaml parameter_defaults: CloudName: openshift.localdomain OvercloudOpenShiftMasterFlavor: master OpenShiftMasterHostnameFormat: '%stackname%-master-%index%' OvercloudOpenShiftWorkerFlavor: worker OpenShiftWorkerHostnameFormat: '%stackname%-worker-%index%' OvercloudOpenShiftInfraFlavor: infra OpenShiftInfraHostnameFormat: '%stackname%-infra-%index%' OpenShiftMasterCount: 3 OpenShiftWorkerCount: 3 OpenShiftInfraCount: 3 NtpServer: ["clock.redhat.com","clock2.redhat.com"] ControlPlaneDefaultRoute: 192.168.24.1 EC2MetadataIp: 192.168.24.1 ControlPlaneSubnetCidr: 24 DnsServers: - 10.0.0.1 OpenShiftGlobalVariables: openshift_master_identity_providers: - name: allow_all login: 'true' challenge: true kind: AllowAllPasswordIdentityProvider openshift_disable_check: docker_image_availability openshift_deployment_type: openshift-enterprise openshift_master_default_subdomain: apps.openshift.localdomain (undercloud) [stack@undercloud-0 ~]$ cat /home/stack/containers-prepare-parameter.yaml # Generated with the following on 2018-10-09T11:50:15.456687 # # openstack tripleo container image prepare default --output-env-file /home/stack/containers-prepare-parameter.yaml --local-push-destination # parameter_defaults: ContainerImagePrepare: - push_destination: true set: ceph_image: rhceph ceph_namespace: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 ceph_tag: 3-13 name_prefix: openstack- name_suffix: '' namespace: rhos-qe-mirror-rdu2.usersys.redhat.com:5000/rhosp14 neutron_driver: null openshift_base_image: ose openshift_cockpit_image: registry-console openshift_cockpit_namespace: registry.access.redhat.com/openshift3 openshift_cockpit_tag: v3.10 openshift_etcd_image: etcd openshift_etcd_namespace: registry.access.redhat.com/rhel7 openshift_etcd_tag: latest openshift_gluster_block_image: rhgs-gluster-block-prov-rhel7 openshift_gluster_image: rhgs-server-rhel7 openshift_gluster_namespace: registry.access.redhat.com/rhgs3 openshift_gluster_tag: latest openshift_heketi_image: rhgs-volmanager-rhel7 openshift_heketi_namespace: registry.access.redhat.com/rhgs3 openshift_heketi_tag: latest openshift_namespace: registry.access.redhat.com/openshift3 openshift_tag: v3.10 tag: 2018-10-08.4
This should be fixed with https://review.openstack.org/#/c/607194/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045