Bug 1347137

Summary: Calamari cli endpoint api returns errors for rados df
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: anmol babu <anbabu>
Component: CalamariAssignee: Boris Ranto <branto>
Calamari sub component: Back-end QA Contact: Vasishta <vashastr>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: medium CC: anbabu, ceph-eng-bugs, gmeno, hnallurv, john.spray, kdreyer, uboppana
Version: 2.0   
Target Milestone: rc   
Target Release: 2.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: calamari-server-1.4.9-1.el7cp Ubuntu: calamari-server_1.4.9-2redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-22 19:26:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description anmol babu 2016-06-16 06:48:36 UTC
Description of problem:
Calamari cli endpoint api returns errors for rados df

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create cluster
2. Create some rbds
3. Use calamari cli end point to get rados df --cluster <cluster_name> --format=json

Actual results:
Api: https://<calamari_node>:8002/api/v2/cluster/{cluster-id}/cli?format=api
Body:
{"command":["rados", "df", "--cluster", "<cluster_name>", "--format=json"]}

Response:
POST /api/v2/cluster/4c6ac35f-cfe3-48ea-a88e-dd3db7ac7b6c/cli?format=api
HTTP 200 OK
Vary: Accept
Content-Type: text/html; charset=utf-8
Allow: POST, OPTIONS

{
    "out": "", 
    "err": "no valid command found; 10 closest matches:\nosd setmaxosd <int[0-]>\nosd pause\nosd crush rule rm <name>\nosd crush tree\nosd crush rule create-simple <name> <root> <type> {firstn|indep}\nosd crush rule create-erasure <name> {<profile>}\nosd crush get-tunable straw_calc_version\nosd crush show-tunables\nosd crush tunables legacy|argonaut|bobtail|firefly|hammer|jewel|optimal|default\nosd crush set-tunable straw_calc_version <int>\nError EINVAL: invalid command\n", 
    "status": 22
}

Expected results:
Valid response

Additional info:

Comment 2 Christina Meno 2016-06-16 14:54:44 UTC
I don't expect that this works.
The CLI endpoint only supports commands that begin with ceph or rbd.

What is this bug's impact to storage console?

Comment 3 anmol babu 2016-06-16 15:35:27 UTC
Earlier, we used salt to get this job done so every 10 mins, we used to get the count of objects and count of degraded objects using this command but through salt. But, the problem we are facing is salt is not very reliable for triggering periodic jobs and so now, we are moving all such periodic salt-calls to happen through calamari api(cli end-point). As part of this exercise, we would like to have this information (object count and degraded object count) also served through the calamari api giving us a json reponse.

Comment 5 Christina Meno 2016-06-16 23:04:54 UTC
ok, I can understand that AND the old way you were getting the data will have to suffice till 2.1

Comment 7 Boris Ranto 2016-10-03 17:14:43 UTC
Upstream PR:

https://github.com/ceph/calamari/pull/490

Comment 8 John Spray 2016-10-04 12:06:49 UTC
I wouldn't recommend calling the CLI binary to get the "rados df" data.  The CLI tool itself is just calling into librados, so for Calamari to provide the data, Calamari should just call into librados too.

If you create a REST endpoint that serves these stats (and does so by invoking a remote librados call), then it will be straightforward to port that forward to ceph-mgr, as at that point we'll just have these stats in memory anyway (ceph-mgr keeps a PGMap handy).

Comment 9 Boris Ranto 2016-10-11 20:52:48 UTC
New re-worked PR based on John's comments:

https://github.com/ceph/calamari/pull/492

Comment 10 Boris Ranto 2016-10-12 12:58:18 UTC
(Tested) PR for 1.4 branch:

https://github.com/ceph/calamari/pull/494

Comment 15 Vasishta 2016-10-26 16:32:43 UTC
Hi Anmol,

'stats' cli endpoints mentioned by Boris in Comment 14 have been tried and they are working fine and they are giving information that 'rados df' would give. 

Sample of information displayed by /api/v2/cluster/<fsid>/pool/stats (to get stats of all the pools) looks like -

{
    "name": "test_rbd", 
    "num_objects_unfound": 0, 
    "num_objects_missing_on_primary": 0, 
    "num_object_clones": 0, 
    "num_objects": 31, 
    "num_object_copies": 93, 
    "num_bytes": 510, 
    "num_rd_kb": 30, 
    "num_wr_kb": 20, 
    "num_kb": 1, 
    "num_wr": 80, 
    "num_objects_degraded": 0, 
    "num_rd": 69
}

Moving this bug to VERIFIED state, Please let me know if there are any concerns or issues.

Regards,
Vasishta

Comment 17 errata-xmlrpc 2016-11-22 19:26:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2815.html