Bug 1594974

Summary: [ceph-ansible 3.0.33-2redhat1] mgr repo not setup with ceph-ansible when using rhcs downstream settings
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vasu Kulkarni <vakulkar>
Component: Ceph-AnsibleAssignee: Sébastien Han <shan>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 3.0CC: aschoen, ceph-eng-bugs, gmeno, hnallurv, kdreyer, nthomas, sankarshan, seb, shan, tserlin, vakulkar, vpoliset
Target Milestone: z4Flags: vakulkar: automate_bug+
Target Release: 3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-ansible-3.0.37-1.el7cp Ubuntu: ceph-ansible_3.0.37-2redhat1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-11 18:11:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vasu Kulkarni 2018-06-25 21:41:58 UTC 
Description of problem:

1. setup xenial node to use rhcs packages
ceph_origin: repository
ceph_repository: rhcs
ceph_repository_type: cdn
ceph_rhcs_cdn_debian_repo: https://xxxxx:xxxx@rhcs.download.redhat.com
ceph_rhcs_version: 3

2. hosts config is 
[mons]
ceph-clacroix-run400-node1-mon monitor_interface=ens3
ceph-clacroix-run400-node3-mon monitor_interface=ens3
ceph-clacroix-run400-node2-mon monitor_interface=ens3
[mgrs]
ceph-clacroix-run400-node7-mgr monitor_interface=ens3
[osds]
ceph-clacroix-run400-node5-osd monitor_interface=ens3  devices='["/dev/vdb", "/dev/vdc", "/dev/vdd"]'
ceph-clacroix-run400-node4-osd monitor_interface=ens3  devices='["/dev/vdb", "/dev/vdc", "/dev/vdd"]'
ceph-clacroix-run400-node6-osd monitor_interface=ens3  devices='["/dev/vdb", "/dev/vdc", "/dev/vdd"]'
[rgws]
ceph-clacroix-run400-node9-rgw radosgw_interface=ens3
[clients]
ceph-clacroix-run400-node10-client client_interface=ens3


After cluster is setup, mgr is not active, debugging further shows that mgr doesn't have repo setup properly unlike mon/osd etc and mgr is using upstream packages.


more check should be added to ceph-ansible to ensure rhcs packages are properly installed.



cephuser@ceph-clacroix-run400-node1-mon:~$ sudo ceph --version
ceph version 12.2.1-42redhat1xenial (c6d85fd953226c9e8168c9abe81f499d66cc2716) luminous (stable)
cephuser@ceph-clacroix-run400-node1-mon:~$ sudo ceph -s
  cluster:
    id:     19dc7064-49b0-42e0-beef-f8e14b163927
    health: HEALTH_WARN
            no active mgr
            clock skew detected on mon.ceph-clacroix-run400-node1-mon, mon.ceph-clacroix-run400-node3-mon
 
  services:
    mon: 3 daemons, quorum ceph-clacroix-run400-node2-mon,ceph-clacroix-run400-node1-mon,ceph-clacroix-run400-node3-mon
    mgr: no daemons active
    osd: 9 osds: 9 up, 9 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 bytes
    usage:   0 kB used, 0 kB / 0 kB avail
    pgs:
Comment 4 Vasu Kulkarni 2018-06-25 22:57:27 UTC 
This has to be fixed in z4 folks not z5.
Comment 5 Sébastien Han 2018-06-26 14:34:53 UTC 
Does that mean there is a downstream repo for ceph-mgr?
Thanks.

Something like rhel-7-server-rhceph-3-mgr-rpms?
Comment 6 Vasu Kulkarni 2018-06-26 15:15:42 UTC 
Sebastein,

This issue is on xenial using rhcs_cdn packages , Manohar found this to be a doc issue and we are missing following in docs, I think a better approach is to link 3-release folder to 3-updates, I will check with Ken or RCM on this.



1. #ceph_rhcs_cdn_debian_repo: https://customername:customerpasswd@rhcs.download.redhat.com
2. #ceph_rhcs_cdn_debian_repo_version: "/3-release/" # for GA, later for updates use /3-updates/
Comment 7 Sébastien Han 2018-06-26 15:43:49 UTC 
I'm not sure I'm following, but let me know what to do and I'll do it or submit a PR on ceph-ansible :).

Thanks.
Comment 8 Vasu Kulkarni 2018-06-26 18:21:43 UTC 
ok changing from 3-release to 3-udpates didnt work, although we got new ceph version,

sebastein to explain this, i think you are assuming rhel nodes thats why you are saying rhel-7-server-rhceph-3-mgr-rpms, but for xenial its in /etc/apt/sources.list.d/ where ceph-ansible creates the repo based on vars

on other nodes eg: osd, mon it creates the repo

$ ls -lt /etc/apt/sources.list.d/rhcs_download_redhat_com_3_updates_MON.list 
-rw-r--r-- 1 root root 116 Jun 26 17:12 /etc/apt/sources.list.d/rhcs_download_redhat_com_3_updates_MON.list

$ sudo ceph --version
ceph version 12.2.4-14redhat1xenial (03fd19535b3701f3322c68b5f424335d6fc8dd66) luminous (stable)

But on mgr, we can see there is no repo and it ends up using upstream packages

mgr:~$ ls -lt /etc/apt/sources.list.d/
total 0

$ sudo ceph --version
ceph version 10.2.9 (2ee413f77150c0f375ff6f10edd6c8f9c7d060d0)
Comment 9 Vasu Kulkarni 2018-06-26 21:44:50 UTC 
I gave this a try here https://github.com/ceph/ceph-ansible/pull/2814
Comment 10 Harish NV Rao 2018-07-02 04:18:21 UTC 
@Gregory & @Sebastien, is this going to be fixed in z4? If not, can you please change the target release?
Comment 11 Sébastien Han 2018-07-02 09:36:21 UTC 
I'll be in z4.
We are going to make a new release upstream today which will correspond to z4.

Thanks.
Comment 12 seb 2018-07-03 15:10:15 UTC 
https://github.com/ceph/ceph-ansible/releases/tag/v3.0.37
Comment 18 errata-xmlrpc 2018-07-11 18:11:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2177