Bug 1219344 - 1.3.0: "ceph-deploy calamari connect <node>" fails
Summary: 1.3.0: "ceph-deploy calamari connect <node>" fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Build
Version: 1.3.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: rc
: 1.3.0
Assignee: Alfredo Deza
QA Contact: ceph-qe-bugs
URL:
Whiteboard:
Depends On: 1210038
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-07 06:40 UTC by Harish NV Rao
Modified: 2015-06-24 15:52 UTC (History)
6 users (show)

Fixed In Version: ceph-deploy-1.5.24-1.el7cp
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-24 15:52:52 UTC
Embargoed:


Attachments (Terms of Use)
CLI output of installation commands (78.78 KB, text/plain)
2015-05-07 06:40 UTC, Harish NV Rao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:1183 0 normal SHIPPED_LIVE Ceph bug fix and enhancement update 2015-06-24 19:49:46 UTC

Description Harish NV Rao 2015-05-07 06:40:52 UTC
Created attachment 1022951 [details]
CLI output of installation commands

Description of problem:

The "ceph-deploy calamari connect <node>" failed with following error:

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephuser/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.23): /usr/bin/ceph-deploy calamari connect Mon
[ceph_deploy][ERROR ] RuntimeError: no calamari-minion repo found


Version-Release number of selected component (if applicable):1.3.0


How reproducible: always


Steps to Reproduce:
1. Begin fresh installation by following the doc: https://gitlab.cee.redhat.com/jowilkin/red-hat-ceph-storage-installation-guide-rhel/blob/master/calamari.adoc and the instructions given in the attachment to 1211368 for installing from puddles.

2. Install calamari server and do calamari-ctl initialize

3. Follow the doc "https://gitlab.cee.redhat.com/jowilkin/red-hat-ceph-storage-installation-guide-rhel/blob/master/quick-ceph-deploy.adoc" and Run following commands:
ceph-deploy new <initial-monitor-node(s)>
ceph-deploy install 
ceph-deploy mon create-initial

3. Run "ceph-deploy calamari connect <node>"

Actual results:
The "ceph-deploy calamari connect <node>" failed with following error:

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephuser/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.23): /usr/bin/ceph-deploy calamari connect Mon
[ceph_deploy][ERROR ] RuntimeError: no calamari-minion repo found

Expected results:
command should have completed successfully

Additional info: None

Comment 2 Alfredo Deza 2015-05-07 12:30:11 UTC
The way this is supposed to work is that the cephdeploy.conf file should have a "[calamari-minion]" repository section.

This is automatically created for the user when installing via the "ice_setup.py" script because that script takes care of creating paths and moving packages and dealing with repos. 

In that process it creates the section for the user (that you seem to be missing).

In short: if the user is not going via the ice_setup.py, this is not supposed to work *unless* a "[calamari-minion]" repo section with all details needed exists in the cephdeploy.conf file.

Comment 3 Travis Rhoden 2015-05-07 13:42:44 UTC
Alfredo,

Your description in Comment #2 was true before, but it is no longer for Stockwell.  As I mentioned in the email thread, there is no longer a "calamari-minions" repo needed at all.  All the dependencies for a calamari minion are present in the ceph-mon *and* the ceph-osd channels.

Therefore, ceph-deploy does not need to do any installation of repo files for calamari-minions, and ice-setup should create/host such a repo.  ceph-deploy just needs to do the install as if the repo is already there.

In that sense, this install *should* work without ice_setup being run.

So ceph-deploy does need to modified such that it does not do any of those steps when using RHEL + RCHS.  Basically, these lines need to be disabled for RHCS: https://github.com/ceph/ceph-deploy/blob/master/ceph_deploy/calamari.py#L23-L80

Comment 4 Travis Rhoden 2015-05-07 13:44:21 UTC
(In reply to Travis Rhoden from comment #3)

> Therefore, ceph-deploy does not need to do any installation of repo files
> for calamari-minions, and ice-setup should create/host such a repo. 
> ceph-deploy just needs to do the install as if the repo is already there.
> 

Just to clarify, I meant "ice-setup should **NOT** create/host such a repo..."

Comment 5 Harish NV Rao 2015-05-08 06:02:21 UTC
changing the severity as this is blocking calamari UI testing.

Comment 9 Harish NV Rao 2015-05-13 16:17:38 UTC
Thanks Travis! I will try the work around.

Comment 12 Harish NV Rao 2015-05-14 17:09:57 UTC
I am also facing the same problem as mentioned by Hemanth after applying the workaround.

I am getting following message in GUI:

 "5 Ceph servers are connected to Calamari, but no Ceph cluster has been created yet. Please use ceph-deploy to create a cluster; please see the Inktank Ceph Enterprise documentation for more details."

Calamari and Admin node IP: http://10.8.128.6
user = TEST12, password = ghj

Comment 13 Christina Meno 2015-05-14 17:14:25 UTC
checking the calamari host I see that the ceph module has not been sync'd to the minions via salt.

I'll continue to investigate why.

[cephuser@Admin ~]$ sudo salt-key -L
Accepted Keys:
Admin
magna029
magna033
magna040
magna086
Unaccepted Keys:
Rejected Keys:
[cephuser@Admin ~]$ sudo salt '*' ceph.get_heartbeats
magna086:
    'ceph.get_heartbeats' is not available.
magna029:
    'ceph.get_heartbeats' is not available.
magna033:
    'ceph.get_heartbeats' is not available.
magna040:
    'ceph.get_heartbeats' is not available.
Admin:
    ----------
    - boot_time:
        1430872612
    - ceph_version:
        0.94.1-8.el7cp
    - services:
        ----------
    ----------

Comment 14 Christina Meno 2015-05-14 17:22:00 UTC
The issue here is that the cluster is running a different version of salt from EPEL. This is not a supported setup.

I recommend downgrading salt-minion and re-trying.

gmeno$ ssh shadow_man.redhat.com
Last login: Wed May  6 14:18:15 2015 from 10.8.128.2
[shadow_man@osd1 ~]$ salt-minion --version
salt-minion 2014.7.5 (Helium)
[shadow_man@osd1 ~]$


[shadow_man@osd1 ~]$ yum repolist
Loaded plugins: priorities, product-id, subscription-manager
Repo rhel-7-server-extras-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5269745257163545448-key.pem
Repo rhel-7-server-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5269745257163545448-key.pem
Repo rhel-7-server-optional-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5269745257163545448-key.pem
lab-extras                                                                                                                                                            10/10
rhel-7-fcgi-ceph                                                                                                                                                        3/3
rhel-7-server-ceph-1.3-calamari-test                                                                                                                                  15/15
rhel-7-server-ceph-1.3-installer-test                                                                                                                                   1/1
rhel-7-server-ceph-1.3-mon-test                                                                                                                                       26/26
rhel-7-server-ceph-1.3-osd-test                                                                                                                                       23/23
repo id                                                                       repo name                                                                               status!epel/x86_64                                                                  Extra Packages for Enterprise Linux                                                     7,819
!lab-extras                                                                   lab-extras                                                                                 10
!rhel-7-fcgi-ceph                                                             RHEL 7 Local fastcgi Repo                                                                 2+1
rhel-7-server-ceph-1.3-calamari-test                                          Server-RH7-CEPH-CALAMARI-1.3                                                               15
rhel-7-server-ceph-1.3-installer-test                                         Server-RH7-CEPH-INSTALLER-1.3                                                               1
rhel-7-server-ceph-1.3-mon-test                                               Server-RH7-CEPH-MON-1.3                                                                    26
rhel-7-server-ceph-1.3-osd-test                                               Server-RH7-CEPH-OSD-1.3                                                                    23
rhel-7-server-extras-rpms/x86_64                                              Red Hat Enterprise Linux 7 Server - Extras (RPMs)                                          52
rhel-7-server-optional-rpms/7Server/x86_64                                    Red Hat Enterprise Linux 7 Server - Optional (RPMs)                                     5,804
rhel-7-server-rpms/7Server/x86_64                                             Red Hat Enterprise Linux 7 Server (RPMs)                                                6,990
repolist: 20,742
[shadow_man@osd1 ~]$ yum info salt-minion
Loaded plugins: priorities, product-id, subscription-manager
Repo rhel-7-server-extras-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5269745257163545448-key.pem
Repo rhel-7-server-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5269745257163545448-key.pem
Repo rhel-7-server-optional-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5269745257163545448-key.pem
Installed Packages
Name        : salt-minion
Arch        : noarch
Version     : 2014.7.5
Release     : 1.el7
Size        : 28 k
Repo        : installed
From repo   : epel
Summary     : Client component for Salt, a parallel remote execution system
URL         : http://saltstack.org/
License     : ASL 2.0
Description : The Salt minion is the agent component of Salt. It listens for instructions
            : from the master, runs jobs, and returns results back to the master.

[shadow_man@osd1 ~]$

Comment 15 Harish NV Rao 2015-05-14 18:29:06 UTC
I did following on all the nodes that had different salt-minion version:

1. Disabled the EPEL:
    a) sudo vi /etc/yum.repos.d/epel.repo
    b) change enabled=1 to enabled=0

2. installed correct version of salt-minion:
    a) sudo yum distro-sync salt\*

3. sudo service salt-minion restart (without this GUI did not show dashboard)

Then opened calamari UI http://10.8.128.6. 

It showed the Dashboard!!! Able to see the details of Ceph cluster!!!

Thanks a lot for the quick response and help.

Regards,
Harish

Comment 18 Hemanth Kumar 2015-06-01 14:02:25 UTC
ceph-deploy calamari connect <node> executed successfully with the new ISO puddle generated on 26th May.. Moving to verified state.
---------------------------------------------------------
Output of ceph-deploy calamari connect <MON_Node>

http://pastebin.test.redhat.com/286920

Comment 20 errata-xmlrpc 2015-06-24 15:52:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1183


Note You need to log in before you can comment on or make changes to this bug.