Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1726567

Summary: OSP15 overcloud 3cont_3data_3mess_1comp (no ceph), deployment fails with : error pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume
Product: Red Hat OpenStack Reporter: pkomarov
Component: openstack-containersAssignee: Emilien Macchi <emacchi>
Status: CLOSED DUPLICATE QA Contact: Marius Cornea <mcornea>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 15.0 (Stein)CC: abeekhof, emacchi, johfulto, lmiccini, m.andre, michele
Target Milestone: ---Keywords: AutomationBlocker, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-05 12:43:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1716627    
Bug Blocks:    

Description pkomarov 2019-07-03 07:56:39 UTC 
Description of problem:
OSP15 overcloud 3cont_3data_3mess_1comp (no ceph), deployment fails with : error pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume

Version-Release number of selected component (if applicable):
RHOS_TRUNK-15.0-RHEL-8-20190701.n.0

Steps to Reproduce:
rerun : 
https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/features/job/DFG-pidone-features-15_director-rhel-virthost-3cont_3data_3mess_1comp-ipv4-geneve-composable_roles/2/artifact/


Additional info:
overcloud / undercloud logs and stack home and files are at : 
https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/features/job/DFG-pidone-features-15_director-rhel-virthost-3cont_3data_3mess_1comp-ipv4-geneve-composable_roles/2/artifact/
Comment 1 pkomarov 2019-07-03 07:59:44 UTC 
from overcloud_install.log:

TASK [tripleo-container-tag : Pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1 image] ***
Tuesday 02 July 2019  21:06:43 +0000 (0:00:02.754)       0:13:23.947 ********** 
skipping: [compute-0] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [database-0] => {"changed": false, "skip_reason": "Conditional result was False"}
fatal: [controller-0]: FAILED! => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1", "delta": "0:00:00.256499", "end": "2019-07-02 21:06:44.643437", "msg": "non-zero return code", "rc": 125, "start": "2019-07-02 21:06:44.386938", "stderr": "Trying to pull 192.168.24.1:
8787/rhosp15/openstack-cinder-volume:20190625.1...Failed\nerror pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/
openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title
>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\"", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed", "error pulling image \"192.168.24.1:8
787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rho
sp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-vol
ume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\""], "stdout": "", "stdout_lines": []}
skipping: [database-1] => {"changed": false, "skip_reason": "Conditional result was False"}
fatal: [controller-1]: FAILED! => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1", "delta": "0:00:00.287698", "end": "2019-07-02 21:06:44.875930", "msg": "non-zero return code", "rc": 125, "start": "2019-07-02 21:06:44.588232", "stderr": "Trying to pull 192.168.24.1:
8787/rhosp15/openstack-cinder-volume:20190625.1...Failed\nerror pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/
openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title
>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\"", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed", "error pulling image \"192.168.24.1:8
787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 201906
25.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\""], "stdout": "", "stdout_lines": []}
skipping: [database-2] => {"changed": false, "skip_reason": "Conditional result was False"}
skipping: [messaging-0] => {"changed": false, "skip_reason": "Conditional result was False"}
fatal: [controller-2]: FAILED! => {"changed": true, "cmd": "podman pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1", "delta": "0:00:00.278776", "end": "2019-07-02 21:06:45.056877", "msg": "non-zero return code", "rc": 125, "start": "2019-07-02 21:06:44.778101", "stderr": "Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed\nerror pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\"", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1...Failed", "error pulling image \"192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-cinder-volume:20190625.1: Error reading manifest 20190625.1 in 192.168.24.1:8787/rhosp15/openstack-cinder-volume: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-cinder-volume/manifests/20190625.1 was not found on this server.</p>\\n</body></html>\\n\""], "stdout": "", "stdout_lines": []}
Comment 4 pkomarov 2019-07-03 08:26:09 UTC 
some standard url checks for the repo : 

[stack@undercloud-0 ~]$ ping registry.redhat.io
PING e14353.g.akamaiedge.net (23.222.44.197) 56(84) bytes of data.
64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=1 ttl=52 time=61.3 ms
64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=2 ttl=52 time=70.10 ms
64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=3 ttl=52 time=61.4 ms
64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=4 ttl=52 time=61.3 ms
64 bytes from a23-222-44-197.deploy.static.akamaitechnologies.com (23.222.44.197): icmp_seq=5 ttl=52 time=61.3 ms
^C
--- e14353.g.akamaiedge.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 179ms
rtt min/avg/max/mdev = 61.258/63.256/70.990/3.867 ms
[stack@undercloud-0 ~]$ curl -v registry.redhat.io/rhceph-beta/rhceph-4-rhel8
*   Trying 23.222.44.197...
* TCP_NODELAY set
* Connected to registry.redhat.io (23.222.44.197) port 80 (#0)
> GET /rhceph-beta/rhceph-4-rhel8 HTTP/1.1
> Host: registry.redhat.io
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 302 Moved Temporarily
< Location: https://registry.redhat.io/rhceph-beta/rhceph-4-rhel8
< Server: BigIP
< Content-Length: 0
< Expires: Wed, 03 Jul 2019 08:24:17 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Wed, 03 Jul 2019 08:24:17 GMT
< Connection: keep-alive
< 
* Connection #0 to host registry.redhat.io left intact
[stack@undercloud-0 ~]$ echo $?
0
Comment 6 John Fulton 2019-07-05 12:43:55 UTC 

*** This bug has been marked as a duplicate of bug 1726171 ***