Bug 1420207

Summary: ceph-ansible fails to pull downstream docker image
Product: [Red Hat Storage] Red Hat Storage Console Reporter: krishnaram Karthick <kramdoss>
Component: ceph-ansibleAssignee: Sébastien Han <shan>
Status: CLOSED ERRATA QA Contact: Rachana Patel <racpatel>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2CC: adeza, aschoen, ceph-eng-bugs, dang, gmeno, hchen, hnallurv, icolle, jim.curtis, kdreyer, nthomas, pprakash, rcyriac, sankarshan, seb
Target Milestone: ---Keywords: Reopened
Target Release: 2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-ansible-2.2.0-1.el7scon Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-19 13:16:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1315538    

Description krishnaram Karthick 2017-02-08 07:05:12 UTC
Description of problem:

with the current naming convention of downstream packages, ceph ansible is unable to pull docker images. ansible fills in the default values when trying to pull images 

case 1: when ceph_mon_docker_image_tagis not updated in groups_vars/all file, "latest" tag is used by default

ceph_mon_docker_username: rhceph
ceph_mon_docker_imagename: ceph-2-rhel-7-docker-candidate-20170206164959
ceph_osd_docker_username: rhceph
ceph_osd_docker_imagename: ceph-2-rhel-7-docker-candidate-20170206164959


TASK [ceph-mon : pull ceph daemon image] ***************************************
fatal: [ceph-monitor01]: FAILED! => {"changed": false, "cmd": ["docker", "pull", "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/ceph-2-rhel-7-docker-candidate-20170206164959:latest"], "delta": "0:00:01.599078", "end": "2017-02-08 12:23:18.673616", "failed": true, "rc": 1, "start": "2017-02-08 12:23:17.074538", "stderr": "unknown: Not Found", "stdout": "Trying to pull repository brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/ceph-2-rhel-7-docker-candidate-20170206164959 ... ", "stdout_lines": ["Trying to pull repository brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/ceph-2-rhel-7-docker-candidate-20170206164959 ... "], "warnings": []}
	to retry, use: --limit @/usr/share/ceph-ansible/site-docker.retry

PLAY RECAP *********************************************************************
ceph-monitor01             : ok=17   changed=2    unreachable=0    failed=1   




case 2: when ceph_mon_docker_image_tag is set to "", it still fails 

ceph_mon_docker_username: rhceph
ceph_mon_docker_imagename: ceph-2-rhel-7-docker-candidate-20170206164959
ceph_mon_docker_image_tag: ""
ceph_osd_docker_username: rhceph
ceph_osd_docker_imagename: ceph-2-rhel-7-docker-candidate-20170206164959
ceph_osd_docker_image_tag: ""


TASK [ceph-mon : pull ceph daemon image] ***************************************
fatal: [ceph-monitor01]: FAILED! => {"changed": false, "cmd": ["docker", "pull", "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/ceph-2-rhel-7-docker-candidate-20170206164959:"], "delta": "0:00:00.022068", "end": "2017-02-08 12:26:05.529494", "failed": true, "rc": 1, "start": "2017-02-08 12:26:05.507426", "stderr": "Error parsing reference: \"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/ceph-2-rhel-7-docker-candidate-20170206164959:\" is not a valid repository/tag", "stdout": "", "stdout_lines": [], "warnings": []}

case 3: when docker_imagename is skipped

ceph_mon_docker_username: rhceph
#ceph_mon_docker_imagename: ceph-2-rhel-7-docker-candidate-20170206164959
ceph_mon_docker_image_tag: ceph-2-rhel-7-docker-candidate-20170206164959
ceph_osd_docker_username: rhceph
#ceph_osd_docker_imagename: ceph-2-rhel-7-docker-candidate-20170206164959
ceph_osd_docker_image_tag: ceph-2-rhel-7-docker-candidate-20170206164959


TASK [ceph-mon : pull ceph daemon image] ***************************************
fatal: [ceph-monitor01]: FAILED! => {"changed": false, "cmd": ["docker", "pull", "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/daemon:ceph-2-rhel-7-docker-candidate-20170206164959"], "delta": "0:00:01.594566", "end": "2017-02-08 11:59:24.825071", "failed": true, "rc": 1, "start": "2017-02-08 11:59:23.230505", "stderr": "unknown: Not Found", "stdout": "Trying to pull repository brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/daemon ... ", "stdout_lines": ["Trying to pull repository brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph/daemon ... "], "warnings": []}
	to retry, use: --limit @/usr/share/ceph-ansible/site-docker.retry

PLAY RECAP *********************************************************************
ceph-monitor01             : ok=17   changed=2    unreachable=0    failed=1   


case 4: when username is skipped,

ceph_mon_docker_imagename: rhceph
ceph_mon_docker_image_tag: ceph-2-rhel-7-docker-candidate-20170206164959
ceph_osd_docker_imagename: rhceph
ceph_osd_docker_image_tag: ceph-2-rhel-7-docker-candidate-20170206164959


TASK [ceph-mon : pull ceph daemon image] ***************************************
fatal: [ceph-monitor01]: FAILED! => {"changed": false, "cmd": ["docker", "pull", "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ceph/rhceph:ceph-2-rhel-7-docker-candidate-20170206164959"], "delta": "0:00:01.600035", "end": "2017-02-08 12:31:43.261556", "failed": true, "rc": 1, "start": "2017-02-08 12:31:41.661521", "stderr": "unknown: Not Found", "stdout": "Trying to pull repository brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ceph/rhceph ... ", "stdout_lines": ["Trying to pull repository brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ceph/rhceph ... "], "warnings": []}


Version-Release number of selected component (if applicable):
[root@dhcp46-176 ceph-ansible]# rpm -qa | grep 'ceph-ansible'
ceph-ansible-2.1.6-1.el7scon.noarch


How reproducible:
Always

Steps to Reproduce:
Provided above

Actual results:
unable to pull docker images using ceph ansible

Expected results:
image pull should be made possible when installing ceph docker using ceph ansible.

Additional info:

Comment 2 seb 2017-02-08 10:54:02 UTC
Images should always have a tag, even if you don't specify anything as tag docker will try to fetch the "latest" tag. Make sure your image candidate is tagged and it should work fine.

Comment 3 Ken Dreyer (Red Hat) 2017-02-08 14:59:43 UTC
It looks like "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/ceph/rhceph" is the image, and "ceph-2-rhel-7-docker-candidate-20170206164959" is the tag.

What happens when you try:
ceph_mon_docker_username: ceph
ceph_mon_docker_imagename: rhceph
ceph_mon_docker_image_tag: ceph-2-rhel-7-docker-candidate-20170206164959

?

Comment 5 Andrew Schoen 2017-02-08 16:03:46 UTC
If you want to pull from a different docker registry other than docker.io you will also need to set `ceph_docker_registry`.

https://github.com/ceph/ceph-ansible/blob/stable-2.1/roles/ceph-docker-common/defaults/main.yml#L3

Comment 6 seb 2017-02-08 16:24:54 UTC
I'm testing this and I believe there is an hidden bug. Looking into that.

Comment 7 seb 2017-02-09 10:53:46 UTC
Upstream PR for this: https://github.com/ceph/ceph-ansible/pull/1302

Comment 16 errata-xmlrpc 2017-06-19 13:16:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1496