Description of problem: 'ceph health' cannot be seen in mgr restful module. Version-Release number of selected component (if applicable): RHCS version 3.2 How reproducible: always Steps to Reproduce: 1. Enable and configure mgr restful module 2. Try to access mgr restful module on a browser using url : https://<ceph-mgr>:8003/config/cluster Actual results: 'ceph health' information is not available in mgr restful module. Expected results: 'ceph health' should be there in ceph cluster information. Additional info: you cannot see 'ceph health' in mgr restful but in ceph-rest-api you can. ceph-rest-api is deprecated in RHCS version 3.2.
Hi Servensha, the ceph-rest-api was superseded by the /request endpoint in the restful module. You can get more information about the endpoint by reading the upstream documentation http://docs.ceph.com/docs/master/mgr/restful/#the-request-endpoint or reading through this thread on ceph-devel: https://marc.info/?l=ceph-devel&m=154773125114644&w=2 Basically, what you need to do is this: [root@node2 ~]# python Python 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> _auth=('admin', '...') >>> result = requests.post('https://node2:8003/request?wait=1', json={'prefix': 'health'}, auth=_auth, verify=False) >>> print result.json()['finished'][0]['outb'] u'HEALTH_OK\n' >>> Following the mailing thread on ceph-devel should give you more insight on how this works and what is going on when you use the endpoint.
Hello, Customer wants to have exact same functions in mgr-resful-module which is present in deprecated ceph-rest-api. The following things related to rgw are not included in mgr-restful-module: rgw successful requests rgw failed requests rgw performance details (read/write iops, latency, throughput etc) rgw active connections rgw active thread number In mgr-module we can see the information about particular request using url => https://10.74.255.33:8003/request and https://10.74.255.33:8003/request/<id> , but we are unable to see the set of rgw failed request, rgw successful requests, rgw active thread number , connections and rgw performance details. The information about other arguments regarding rgw can be seen using url => https://10.74.255.33:8003/config/cluster But as we can see information related to monitor and osd separately using url => https://10.74.255.33:8003/mon and https://10.74.255.33:8003/osd respectively. Is there any way in mgr-module which gives information regarding rgw about above mentioned points? Also cu mentioned that crushmap structure is present in ceph-rest-api. We can se info about crush map using => https://10.74.255.33:8003/crush/rule but it gives output like below : 0 max_size 10 min_size 1 osd_count 6 rule_id 0 rule_name "replicated_rule" ruleset 0 steps 0 item -1 item_name "default" op "take" 1 num 0 op "chooseleaf_firstn" type "host" 2 op "emit" type 1 Can we have whole crush map structure instead of only this much of info? The functions which cu mentioned present in ceph-rest-api : Df - the ability of viewing cluster wise consumption(used, available, overall) Pg stats - viewing the pg stats (state, number etc) Features - monitor clients connected to ceph cluster Health - monitor cluster's health Mon_status - see monitor status (quorum, clock sqew, daemon status etc) Nodes ls - view nodes participating in the cluster Device classes - viewing device classes Crush dump - crushmap structure Osd df - the ability of viewing disk wise consumption(used, available, overall) Osd down - viewing down osds Osd perf - viewing performance details on osds(apply latency, commit latency, request latency, read/write iops, latency, throughput etc etc) Osd perf histogram- same Osd tree - osd tree structure Pool ls - viewing pools Pool stats - performance details upon pools (read/write iops, latency, throughput etc) Daemons (mon/mgr/osd/rgw daemon status) My question is do I have to raise an RFE to have some features from ceph-rest-api to mgr-restful-module. It seems there is some gap between ceph-rest-api and mgr-module. Please let me know. Thank you! Best regards, Servesha
Hello, I followed below doc to refer to mgr-module. https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/ceph_management_api/questions-and-answers Thanks Best regards, Servesha
Hello, I had a call with the customer. There are some major gaps between ceph-rest-api and mgr-module which we have to fill up. Customer has already upgraded to RHCS 3.2. He is still on ceph-rest-api because, there are gaps and so that he cannot move to mgr-module. ceph-rest-api is deprecated in RHCS 3.2. We cannot really force the customer to use mgr-module also can't provide support exception. Their requirement can't be fullfilled ; mgr-module doesn't fetch as much information as ceph-rest-api. I have already mentioned detailed information about what needs to be added in mgr-module. [ Please refer to Comment #2 ] Could we please have functionalities as it was present in ceph-rest-api ? Kind Regards, Servesha
(In reply to Servesha from comment #2) > Hello, > Hey, > Customer wants to have exact same functions in mgr-resful-module which is > present in deprecated ceph-rest-api. > The following things related to rgw are not included in mgr-restful-module: > > rgw successful requests > rgw failed requests > rgw performance details (read/write iops, latency, throughput etc) > rgw active connections > rgw active thread number > > In mgr-module we can see the information about particular request using url > => https://10.74.255.33:8003/request and > https://10.74.255.33:8003/request/<id> , but we are unable to see the set of > rgw failed request, rgw successful requests, rgw active thread number , > connections and rgw performance details. > > The information about other arguments regarding rgw can be seen using url => > https://10.74.255.33:8003/config/cluster > > But as we can see information related to monitor and osd separately using > url => https://10.74.255.33:8003/mon and https://10.74.255.33:8003/osd > respectively. Is there any way in mgr-module which gives information > regarding rgw about above mentioned points? > You are right they are not available in the restful module. The prometheus exporter module should provide these (and more) though. This could be an RFE. However, please note that depending on what the customer is trying to achieve, they could be more or less successful if they used the prometheus exporter module (and a prometheus server) instead of the restful module. > Also cu mentioned that crushmap structure is present in ceph-rest-api. We > can se info about crush map using => https://10.74.255.33:8003/crush/rule > but it gives output like below : > > 0 > max_size 10 > min_size 1 > osd_count 6 > rule_id 0 > rule_name "replicated_rule" > ruleset 0 > steps > 0 > item -1 > item_name "default" > op "take" > 1 > num 0 > op "chooseleaf_firstn" > type "host" > 2 > op "emit" > type 1 > > Can we have whole crush map structure instead of only this much of info? > Is this what they were looking for? [root@node2 ~]# python Python 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> _auth=('admin', '...') >>> result = requests.post('https://node2:8003/request?wait=1', json={'prefix': 'osd crush dump'}, auth=_auth, verify=False) >>> print result.json()['finished'][0]['outb'] ... If not, then there might be other prefixes like 'osd crush rule dump' or 'osd crush rule ls' that they are looking for. > The functions which cu mentioned present in ceph-rest-api : > Df - the ability of viewing cluster wise consumption(used, available, > overall) > Pg stats - viewing the pg stats (state, number etc) > Features - monitor clients connected to ceph cluster > Health - monitor cluster's health > Mon_status - see monitor status (quorum, clock sqew, daemon status etc) > Nodes ls - view nodes participating in the cluster > Device classes - viewing device classes > Crush dump - crushmap structure > Osd df - the ability of viewing disk wise consumption(used, available, > overall) > Osd down - viewing down osds > Osd perf - viewing performance details on osds(apply latency, commit > latency, request latency, read/write iops, latency, throughput etc etc) > Osd perf histogram- same > Osd tree - osd tree structure > Pool ls - viewing pools > Pool stats - performance details upon pools (read/write iops, latency, > throughput etc) > Daemons (mon/mgr/osd/rgw daemon status) > These are also available in the restful module, right? (although they might be available through the /request endpoint) > My question is do I have to raise an RFE to have some features from > ceph-rest-api to mgr-restful-module. It seems there is some gap between > ceph-rest-api and mgr-module. Please let me know. > You can raise an RFE, especially for the RGW bits. > Thank you! > > Best regards, > Servesha
Hello Boris, >> These are also available in the restful module, right? (although they might be available through the /request endpoint) - Yeah I'm sure about some features availability in mgr restful plugin like 'ceph health'. But Everything is not available. For Ex : Osd perf histogram, pool stats, viewing down osds, ceph osd tree, pg stats - not there in mgr restful. If there are some ways to have that information in mgr restful, please let me know. === In the doc I could not see prefixes such as - 'osd crush rule dump' or 'osd crush rule ls'. The doc mentions information about 'crush rule'. please refer to doc link [1]. [1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/ceph_management_api/questions-and-answers#how-can-i-view-crush-rules === For rgw bits, I have raised a RFE - https://bugzilla.redhat.com/show_bug.cgi?id=1699233 === Best Regards, Servesha
What endpoint/command does the customer use to get the rgw data mentioned above? I have been looking into the ceph-rest-api source code and there is nothing special regarding rgw handling. That suggests, the information the customer is looking for might already be covered by the restful's /request endpoint. We just need to figure out what command (prefix) they need to run. Alternatively, if it really is not covered by the current API, we may want to expose all the perf counters in the restful module. That should help us cover the customer request.
Hello Boris, - You're right. rgw related requests are not present in ceph-rest-api. And what covered in newer API is : we can see the information about particular request using url => https://10.74.255.33:8003/request and https://10.74.255.33:8003/request/<id> - but we are unable to see the set of rgw failed request, rgw successful requests, rgw active thread number , connections and rgw performance details. - I think, we should cover things like rgw failed request, rgw successful requests, rgw active thread number , connections and rgw performance details. That will surely cover the customer's request regarding rgw information. Best Regards, Servesha
Hi Servesha, this time I was asking how they can access the rgw related requests using the deprecated ceph-rest-api module/app? I mean, I don't see this data anywhere in the ceph-rest-api -- at least not directly, they could be using some more advanced/hidden method to get that data though and if they are I would like to know which one/how are they getting that information from the deprecated ceph-rest-api. Or is this really a new feature request? I.e. having this new data exposed in the restful ceph-mgr module while it wasn't exposed in the deprecated ceph-rest-api? From what I can see if could they access the data before, I could give them steps on how to do it with the current restful module. If they couldn't access it before I could probably make the perf counters available through the new restful module (if upstream would be ok with that). That should cover at least some (if not all) of the information that you mentioned above. The new rgw data would then look like this: https://pastebin.com/UTAZ4Wdp Regards, Boris
Hello Boris, They are not accessing rgw related data using the deprecated ceph-rest-api. Since it's not available in ceph-rest-api. Yes, This is a new feature request. I.e. having this new data exposed in the restful ceph-mgr module while it wasn't exposed in the deprecated ceph-rest-api. They cannot access rgw related information using either of modules, so they requested for RFE. https://pastebin.com/UTAZ4Wdp The data mentioned in the pastebin covers information for all rgw requests right ? Could we put filters and have data for 1. rgw failed request 2. rgw successful requests etc. Best Regards, Servesha
This exposes the perf counters as described e.g. here: http://docs.ceph.com/docs/master/dev/perf_counters/ This is the data that ceph collects internally about its deamons. What it means is that this is per daemon -- each rgw, osd, etc daemon has its own set of perf counters. My current scratch implementation does support filtering by daemons through regexps with the '/perf?daemon=<regexp>' syntax. The resulting data is a simple JSON so any option should be easily accessible algorithmically. This should also allow you to search for the number of active rgw daemons simply by using '/perf?daemon=rgw.*' endpoint and counting the number of returned daemons. Honestly, I am not sure about the per-counter filters as this would make things fairly slow if you were to gather several pieces of information, here. It would be better to get information about all the daemons that you care about and store and access it as necessary. For any additional information, this would require setting up a /rgw endpoint that could contain this piece of information and it would be a subject to further research on the feasibility of exposing such an information.
We should improve the documentation of the /request endpoint for our next release to better cover the way how ceph-rest-api was deprecated by the restful module.
Hello Boris, Regarding comment #11 - Agreed. It'd be better to get information about all daemons, and then access/store which we need. Also Regarding comment #12 - Yes. We will have to make documentation better in order to eliminate module gap conflicts. Also we can include perf counters (when it's merged), mentioned in the PR. https://github.com/ceph/ceph/pull/27885 Best Regards, Servesha
Hello Boris, May I know status of this bug?
The current status is that the upstream PR was merged, the upstream back-port for Nautilus is currently pending QA. I have back-ported this downstream for 3.3 so that is where you should be able to see the perf counters being exposed by the restful module.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:2538