Description of problem: OSP15 overcloud 3cont_3data_3mess_1comp (no ceph), deployment fails with : error pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume Version-Release number of selected component (if applicable): RHOS_TRUNK-15.0-RHEL-8-20190701.n.0 Steps to Reproduce: rerun : https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/features/job/DFG-pidone-features-15_director-rhel-virthost-3cont_3data_3mess_1comp-ipv4-geneve-composable_roles/2/artifact/ Additional info: overcloud / undercloud logs and stack home and files are at : https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/features/job/DFG-pidone-features-15_director-rhel-virthost-3cont_3data_3mess_1comp-ipv4-geneve-composable_roles/2/artifact/
from overcloud_install.log: TASK [tripleo-container-tag : Pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1 image] *** Tuesday 02 July 2019 21:06:43 +0000 (0:00:02.754) 0:13:23.947 ********** skipping: [compute-0] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [database-0] => {"changed": false, "skip_reason": "Conditional result was False"} fatal: [controller-0]: FAILED! => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1", "delta": "0:00:00.256499", "end": "2019-07-02 21:06:44.643437", "msg": "non-zero return code", "rc": 125, "start": "2019-07-02 21:06:44.386938", "stderr": "Trying to pull 192.168.24.1: 8787/rhosp15/openstack-cinder-volume:20190625.1...Failed\nerror pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/ openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title >\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\"", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed", "error pulling image \"192.168.24.1:8 787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rho sp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-vol ume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\""], "stdout": "", "stdout_lines": []} skipping: [database-1] => {"changed": false, "skip_reason": "Conditional result was False"} fatal: [controller-1]: FAILED! => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1", "delta": "0:00:00.287698", "end": "2019-07-02 21:06:44.875930", "msg": "non-zero return code", "rc": 125, "start": "2019-07-02 21:06:44.588232", "stderr": "Trying to pull 192.168.24.1: 8787/rhosp15/openstack-cinder-volume:20190625.1...Failed\nerror pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/ openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title >\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\"", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed", "error pulling image \"192.168.24.1:8 787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 201906 25.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\""], "stdout": "", "stdout_lines": []} skipping: [database-2] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [messaging-0] => {"changed": false, "skip_reason": "Conditional result was False"} fatal: [controller-2]: FAILED! => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1", "delta": "0:00:00.278776", "end": "2019-07-02 21:06:45.056877", "msg": "non-zero return code", "rc": 125, "start": "2019-07-02 21:06:44.778101", "stderr": "Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed\nerror pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\"", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed", "error pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\""], "stdout": "", "stdout_lines": []}
some standard url checks for the repo : [stack@undercloud-0 ~]$ ping registry.redhat.io PING e14353.g.akamaiedge.net (23.222.44.197) 56(84) bytes of data. 64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=1 ttl=52 time=61.3 ms 64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=2 ttl=52 time=70.10 ms 64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=3 ttl=52 time=61.4 ms 64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=4 ttl=52 time=61.3 ms 64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=5 ttl=52 time=61.3 ms ^C --- e14353.g.akamaiedge.net ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 179ms rtt min/avg/max/mdev = 61.258/63.256/70.990/3.867 ms [stack@undercloud-0 ~]$ curl -v registry.redhat.io/rhceph-beta/rhceph-4-rhel8 * Trying 23.222.44.197... * TCP_NODELAY set * Connected to registry.redhat.io (23.222.44.197) port 80 (#0) > GET /rhceph-beta/rhceph-4-rhel8 HTTP/1.1 > Host: registry.redhat.io > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 302 Moved Temporarily < Location: https://registry.redhat.io/rhceph-beta/rhceph-4-rhel8 < Server: BigIP < Content-Length: 0 < Expires: Wed, 03 Jul 2019 08:24:17 GMT < Cache-Control: max-age=0, no-cache, no-store < Pragma: no-cache < Date: Wed, 03 Jul 2019 08:24:17 GMT < Connection: keep-alive < * Connection #0 to host registry.redhat.io left intact [stack@undercloud-0 ~]$ echo $? 0
*** This bug has been marked as a duplicate of bug 1726171 ***