Bug 1320547

Summary: OSD not added to cluster after install and configure.
Product: Red Hat Ceph Storage Reporter: Tejas <tchandra>
Component: Ceph-InstallerAssignee: Alfredo Deza <adeza>
Status: CLOSED DUPLICATE QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: adeza, aschoen, ceph-eng-bugs, gmeno, hnallurv, nthomas, sankarshan, tchandra
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-01 06:25:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
tasks ID dump none

Description Tejas 2016-03-23 13:12:18 UTC 
Description of problem:
After configuring the cluster using ceph-installer, the OSD are not showing up as part of the cluster.
the Ceph files are correctly copied on the OSD nodes.
I have used the reference doc : http://docs.ceph.com/ceph-installer/docs/

[ceph-installer@magna082 ~]$ sudo ceph -v
ceph version 10.0.4-2.el7cp (61adb020219fbad4508050b5f0a792246ba74dae)
[ceph-installer@magna082 ~]$ 
[ceph-installer@magna082 ~]$ 
[ceph-installer@magna082 ~]$ sudo ceph status
2016-03-23 13:09:32.712027 7f11007b2700 -1 asok(0x7f10f8000f80) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-clients/ceph-client.admin.6536.139710856958112.asok': (2) No such file or directory
    cluster b22da224-c464-4a7e-95f5-9a87e8041202
     health HEALTH_ERR
            64 pgs are stuck inactive for more than 300 seconds
            64 pgs stuck inactive
            no osds
     monmap e1: 1 mons at {magna082=10.8.128.82:6789/0}
            election epoch 4, quorum 0 magna082
     osdmap e1: 0 osds: 0 up, 0 in
            flags sortbitwise
      pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
                  64 creating
[ceph-installer@magna082 ~]$ 
[ceph-installer@magna082 ~]$ 
[ceph-installer@magna082 ~]$ 
[ceph-installer@magna082 ~]$ sudo ceph osd tree
2016-03-23 13:10:01.209017 7f308e811700 -1 asok(0x7f3088000f80) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-clients/ceph-client.admin.6579.139846416863392.asok': (2) No such file or directory
ID WEIGHT TYPE NAME    UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1      0 root default                                   
[ceph-installer@magna082 ~]$

my setup is:
[mon]
magna082.ceph.redhat.com

[osd]
magna081
magna083


Version-Release number of selected component (if applicable): 
Ceph-installer 1.0.1

How reproducible:
Always
Comment 2 Christina Meno 2016-03-23 15:58:29 UTC 
Would you please provide additional details? I think we're looking for command history and error logs which are missing in the report.
Comment 3 Christina Meno 2016-03-23 15:59:47 UTC 
Alfredo, Would you please advise what information do we need to understand what's going on here?
Comment 4 Christina Meno 2016-03-23 16:01:44 UTC 
Is this a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1320574
Comment 5 Tejas 2016-03-24 10:30:47 UTC 
the setup is still in the same state:
[installer]
magna103.ceph.redhat.com

[mon]
magna082.ceph.redhat.com

[osd]
magna081
magna083
Comment 6 Alfredo Deza 2016-03-24 20:04:43 UTC 
@Tejas, we need to see what requests are being sent, in what order, with what output from the /api/tasks/{ID}

The ticket description here doesn't provide enough to be able to understand what is going on.
Comment 7 Tejas 2016-03-28 09:47:55 UTC 
@Alfredo, could you please let know where I can find the /api/tasks/{ID} logs.
Also any other logs that  are required?
Comment 8 Alfredo Deza 2016-03-28 11:04:43 UTC 
@Tejas the tasks are documented here: http://docs.ceph.com/ceph-installer/docs/#tasks

The section on the supported requests has a /api/tasks/ which shows all available: http://docs.ceph.com/ceph-installer/docs/#get--api-tasks-

The important things to understand what is going on is to: capture all requests going to the API and match them with their corresponding ID so that the metadata about the operation can be inspected (stdout, stderr, ansible command, etc...)
Comment 9 Tejas 2016-03-28 12:13:56 UTC 
Created attachment 1140845 [details]
tasks ID dump
Comment 10 Alfredo Deza 2016-03-28 13:09:51 UTC 
Looking at the output it seems like the network subnet is not correct:

    "public_network": "10.8.128.81/21"

Maybe you meant: 

    "public_network": "10.8.128.0/21"
Comment 11 Tejas 2016-04-01 06:25:04 UTC 

*** This bug has been marked as a duplicate of bug 1320574 ***