Description of problem ====================== Pools page which provides a list of pools shows information about maximal available storage capacity for each pool. But this information is valid only when there is no problem with the cluster. Eg. when (some) OSD are removed from cluster map (eg. in case of some problem), the pool item in the Pools list provides incorrect storage capacity information. Version-Release =============== On RHSC 2.0 server: rhscon-ceph-0.0.27-1.el7scon.x86_64 rhscon-core-0.0.28-1.el7scon.x86_64 rhscon-core-selinux-0.0.28-1.el7scon.noarch rhscon-ui-0.0.42-1.el7scon.noarch ceph-ansible-1.0.5-23.el7scon.noarch ceph-installer-1.0.12-3.el7scon.noarch On Ceph Storage nodes: rhscon-agent-0.0.13-1.el7scon.noarch ceph-osd-10.2.2-5.el7cp.x86_64 How reproducible ================ 100 % Steps to Reproduce ================== 1. Install RHSC 2.0 following the documentation. 2. Accept few nodes for the ceph cluster. 3. Create new ceph cluster named 'alpha'. 4. Create 2 RBD (along with new backing pool each time) in the cluster. 5. Check CRUSH cluster map, make sure it's ok and then make a backup of it: ~~~ # ceph --cluster alpha osd getcrushmap -o ceph-crushmap.ok.compiled # crushtool -d ceph-crushmap.ok.compiled -o ceph-crushmap.ok ~~~ 7. Edit CRUSH cluster map so that there are no OSDs in cluster hierarchy which is used by the backing pools of RBDs created in step 4: ~~~ # cp ceph-crushmap.ok ceph-crushmap.err01 # sed -i '/.*item\ osd\.[0-9]\+\ weight\ [0-9\.]\+$/d' ceph-crushmap.err01 $ sed -i 's/weight\ [0-9\.]\+$/weight 0.000/' ceph-crushmap.err01 ~~~ So that for example: ~~~ # diff ceph-crushmap.ok ceph-crushmap.err01 65c65 < # weight 0.010 --- > # weight 0.000 68d67 < item osd.1 weight 0.010 72c71 < # weight 0.010 --- > # weight 0.000 75d73 < item osd.2 weight 0.010 79c77 < # weight 0.010 --- > # weight 0.000 82d79 < item osd.0 weight 0.010 86c83 < # weight 0.010 --- > # weight 0.000 89d85 < item osd.3 weight 0.010 93c89 < # weight 0.040 --- > # weight 0.000 96,99c92,95 < item mbukatov-usm1-node2.os1.phx2.redhat.com-general weight 0.010 < item mbukatov-usm1-node3.os1.phx2.redhat.com-general weight 0.010 < item mbukatov-usm1-node1.os1.phx2.redhat.com-general weight 0.010 < item mbukatov-usm1-node4.os1.phx2.redhat.com-general weight 0.010 --- > item mbukatov-usm1-node2.os1.phx2.redhat.com-general weight 0.000 > item mbukatov-usm1-node3.os1.phx2.redhat.com-general weight 0.000 > item mbukatov-usm1-node1.os1.phx2.redhat.com-general weight 0.000 > item mbukatov-usm1-node4.os1.phx2.redhat.com-general weight 0.000 ~~~ 8. Compile new broken CRUSH map and push it into the cluster: ~~~ # crushtool -c ceph-crushmap.err01 -o ceph-crushmap.err01.compiled # ceph --cluster alpha osd setcrushmap -i ceph-crushmap.err01.compiled ~~~ At this point, `ceph --cluster alpha status` should report something like: * health HEALTH_WARN * recovery ... objects misplaced (100.000%) 9. Check output of `ceph df` command. Since you haven't loaded any data into the RBDs, ceph should report that both pools are empty (0 for %USED and 0 for MAX AVAIL) . 10. Make sure the sync of cluster state happened before you go on. You can either wait for RHSC 2.0 UI to sync new cluster state automatically. But it means waiting for at least 24 hours (which is the current default value of clustersSyncInterval). Or you can force the sync by restarting skyring service: ~~~ systemctl restart skyring ~~~ 11. Check Pools list page. Actual results ============== While `ceph df` provides this information: ~~~ # ceph --cluster alpha df GLOBAL: SIZE AVAIL RAW USED %RAW USED 40915M 40760M 155M 0.38 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS rbd_pool 1 7486 0 0 16 def_pool 3 114 0 0 4 ~~~ The *Pools* page shown different storage capacity data for both pools: * rbd_pool: 100.0%, 7.3 KB of 7.3 KB used * def_pool: 100.0%, 114.0 B of 114.0 B used (see screenshot #2) When we ignore the wrong units there for now (there is another BZ for that, see https://bugzilla.redhat.com/show_bug.cgi?id=1340747#c4), the data presented there doesn't match data reported by `ceph df`: * '%USED' should not be reported as 100% by UI (ceph df stil shows 0%, moreover both pools are essentially empty). * MAX AVAIL is not shown in the pool list by UI, and the `7.3 KB of 7.3 KB used` statement is misleading in this case The related event states: > Pool utilization for pool rbd_pool on alpha cluster has moved to CRITICAL which while technically true, it doesn't cover the actual issue in any way and admin is again forced to use ceph command line tools to debug the problem. Expected results ================ The storage capacity information should not conflict with information as provided by `ceph df` command. Statement "7.3 KB of 7.3 KB used" would be better replaced by presenting 'USED' and 'MAX AVAIL' values instead. Related events should better describe the problem. At least information about zero value of 'MAX AVAIL' should be conveyed. Additional info =============== Besides the problem with interpreting the storage capacity data, the sync interval itself makes things harder to understand as well: * See screenshot #1 which I created after I did all steps from "Steps to Reproduce" section but without restarting skyring, so that the cluster state is still not synced. * And compare that with screenshot #2 which shows Pools list page after the skyring restart (which triggered the sync).
Created attachment 1178853 [details] screenshot 1: pools list before cluster sync
Created attachment 1178856 [details] screenshot 2: pools list after cluster sync
Created attachment 1178859 [details] screenshot 3: pool unitization event detail page
(In reply to Martin Bukatovic from comment #0) > Description of problem > ====================== > > Pools page which provides a list of pools shows information about maximal > available storage capacity for each pool. But this information is valid only > when there is no problem with the cluster. > > Eg. when (some) OSD are removed from cluster map (eg. in case of some > problem), > the pool item in the Pools list provides incorrect storage capacity > information. > > Version-Release > =============== > > On RHSC 2.0 server: > > rhscon-ceph-0.0.27-1.el7scon.x86_64 > rhscon-core-0.0.28-1.el7scon.x86_64 > rhscon-core-selinux-0.0.28-1.el7scon.noarch > rhscon-ui-0.0.42-1.el7scon.noarch > ceph-ansible-1.0.5-23.el7scon.noarch > ceph-installer-1.0.12-3.el7scon.noarch > > On Ceph Storage nodes: > > rhscon-agent-0.0.13-1.el7scon.noarch > ceph-osd-10.2.2-5.el7cp.x86_64 > > How reproducible > ================ > > 100 % > > Steps to Reproduce > ================== > > 1. Install RHSC 2.0 following the documentation. > > 2. Accept few nodes for the ceph cluster. > > 3. Create new ceph cluster named 'alpha'. > > 4. Create 2 RBD (along with new backing pool each time) in the cluster. > > 5. Check CRUSH cluster map, make sure it's ok and then make a backup of it: > > ~~~ > # ceph --cluster alpha osd getcrushmap -o ceph-crushmap.ok.compiled > # crushtool -d ceph-crushmap.ok.compiled -o ceph-crushmap.ok > ~~~ > > 7. Edit CRUSH cluster map so that there are no OSDs in cluster hierarchy > which is used by the backing pools of RBDs created in step 4: > > ~~~ > # cp ceph-crushmap.ok ceph-crushmap.err01 > # sed -i '/.*item\ osd\.[0-9]\+\ weight\ [0-9\.]\+$/d' ceph-crushmap.err01 > $ sed -i 's/weight\ [0-9\.]\+$/weight 0.000/' ceph-crushmap.err01 > ~~~ > > So that for example: > > ~~~ > # diff ceph-crushmap.ok ceph-crushmap.err01 > 65c65 > < # weight 0.010 > --- > > # weight 0.000 > 68d67 > < item osd.1 weight 0.010 > 72c71 > < # weight 0.010 > --- > > # weight 0.000 > 75d73 > < item osd.2 weight 0.010 > 79c77 > < # weight 0.010 > --- > > # weight 0.000 > 82d79 > < item osd.0 weight 0.010 > 86c83 > < # weight 0.010 > --- > > # weight 0.000 > 89d85 > < item osd.3 weight 0.010 > 93c89 > < # weight 0.040 > --- > > # weight 0.000 > 96,99c92,95 > < item mbukatov-usm1-node2.os1.phx2.redhat.com-general weight 0.010 > < item mbukatov-usm1-node3.os1.phx2.redhat.com-general weight 0.010 > < item mbukatov-usm1-node1.os1.phx2.redhat.com-general weight 0.010 > < item mbukatov-usm1-node4.os1.phx2.redhat.com-general weight 0.010 > --- > > item mbukatov-usm1-node2.os1.phx2.redhat.com-general weight 0.000 > > item mbukatov-usm1-node3.os1.phx2.redhat.com-general weight 0.000 > > item mbukatov-usm1-node1.os1.phx2.redhat.com-general weight 0.000 > > item mbukatov-usm1-node4.os1.phx2.redhat.com-general weight 0.000 > ~~~ > > 8. Compile new broken CRUSH map and push it into the cluster: > > ~~~ > # crushtool -c ceph-crushmap.err01 -o ceph-crushmap.err01.compiled > # ceph --cluster alpha osd setcrushmap -i ceph-crushmap.err01.compiled > ~~~ > > At this point, `ceph --cluster alpha status` should report something like: > > * health HEALTH_WARN > * recovery ... objects misplaced (100.000%) > > 9. Check output of `ceph df` command. Since you haven't loaded any data > into the RBDs, ceph should report that both pools are empty (0 for > %USED and 0 for MAX AVAIL) . > > 10. Make sure the sync of cluster state happened before you go on. > > You can either wait for RHSC 2.0 UI to sync new cluster state > automatically. But it means waiting for at least 24 hours (which is > the current default value of clustersSyncInterval). > > Or you can force the sync by restarting skyring service: > > ~~~ > systemctl restart skyring > ~~~ > > 11. Check Pools list page. > > Actual results > ============== > > While `ceph df` provides this information: > > ~~~ > # ceph --cluster alpha df > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 40915M 40760M 155M 0.38 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > rbd_pool 1 7486 0 0 16 > def_pool 3 114 0 0 4 > ~~~ Here: USED --> used space of the pool MAX AVAIL --> Maximum size available to the pool. this will be TOTAL-USED So the total is not provided by the ceph CLI so USM will calculate TOTAL as USED+MAX AVAIL Also note that USM execute json fromatted API call to get the pool stats(ceph df -f json) and this CLI won't provide the percentage value so we need to calculate which is done as USED/TOTAL * 100 where TOTAL is USED+MAX AVAIL. > > The *Pools* page shown different storage capacity data for both pools: > > * rbd_pool: 100.0%, 7.3 KB of 7.3 KB used > * def_pool: 100.0%, 114.0 B of 114.0 B used > > (see screenshot #2) > > When we ignore the wrong units there for now (there is another BZ for that, > see https://bugzilla.redhat.com/show_bug.cgi?id=1340747#c4), the data > presented > there doesn't match data reported by `ceph df`: > > * '%USED' should not be reported as 100% by UI (ceph df stil shows 0%, > moreover both pools are essentially empty). > * MAX AVAIL is not shown in the pool list by UI, and the > `7.3 KB of 7.3 KB used` statement is misleading in this case > > The related event states: > > > Pool utilization for pool rbd_pool on alpha cluster has moved to CRITICAL > > which while technically true, it doesn't cover the actual issue in any way > and admin is again forced to use ceph command line tools to debug the > problem. > > Expected results > ================ > > The storage capacity information should not conflict with information as > provided by `ceph df` command. > > Statement "7.3 KB of 7.3 KB used" would be better replaced by presenting > 'USED' and 'MAX AVAIL' values instead. > > Related events should better describe the problem. At least information about > zero value of 'MAX AVAIL' should be conveyed. > I have explained the logic above and it is clear that USM is doing the calculation as expected. So I don't treat this an issue. We can discuss this in the bug scrub meeting take a call > Additional info > =============== > > Besides the problem with interpreting the storage capacity data, the sync > interval itself makes things harder to understand as well: > > * See screenshot #1 which I created after I did all steps from "Steps to > Reproduce" section but without restarting skyring, so that the cluster > state is still not synced. > * And compare that with screenshot #2 which shows Pools list page after the > skyring restart (which triggered the sync).
We now pick the utilization from the normal cli command ceph df instead of its json form. So the percentage used and used size is now the same as what the cli returns and the total calculation is as follows(as per sjust's suggestions): POOL TOTAL SIZE = MAX AVAIL / (.01 * (100 - %USED)) where MAX AVAIL and %USED are directly from ceph df cli o/p.
As you can see on screenshots utilization charts are not correct. My configuration: 1 cluster 2 storage profiles: default - 2 OSDs, user defined - 2 OSDs 2 pools - double replication each; one on each storage profile(pool1 on default, pool2 on user_defined) each OSD has 10GB Please check screenshots. Tested with ceph-ansible-1.0.5-31.el7scon.noarch ceph-installer-1.0.14-1.el7scon.noarch rhscon-ceph-0.0.39-1.el7scon.x86_64 rhscon-core-0.0.39-1.el7scon.x86_64 rhscon-core-selinux-0.0.39-1.el7scon.noarch rhscon-ui-0.0.51-1.el7scon.noarch
Created attachment 1186442 [details] build 39 list of pools As you can see used is bigger than total size and % utilization is not correct. There are 7 objects 1GB each in double replicated pool.
Created attachment 1186443 [details] build 39 - main dashboard
Created attachment 1186444 [details] build 39 - cluster dashboard
Created attachment 1186445 [details] build 39 - cluster list
%USED doesn't show how much of the space for the pool is used, but how much of the space of cluster is used in the pool. With that said if I have several hierarchies the number is not correct. (At least not correct for calculating total size of pool from it.)
Looks good to me
This product is EOL now