Bug 1306842

Summary:	Adding a new OSD to a Ceph cluster with a CRUSH weight of 0 causes 'ceph df' to report invalid MAX AVAIL on pools
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Mike Hackett <mhackett>
Component:	RADOS	Assignee:	Kefu Chai <kchai>
Status:	CLOSED ERRATA	QA Contact:	Ramakrishnan Periyasamy <rperiyas>
Severity:	high	Docs Contact:	Bara Ancincova <bancinco>
Priority:	high
Version:	1.3.1	CC:	ceph-eng-bugs, dzafman, flucifre, hnallurv, icolle, kchai, kdreyer, ksquizza, tserlin, vumrao
Target Milestone:	rc
Target Release:	1.3.3
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	RHEL: ceph-0.94.7-5.el7cp Ubuntu: ceph_0.94.7-3redhat1trusty	Doc Type:	Bug Fix
Doc Text:	."ceph df" now shows proper value of "MAX AVAIL" When adding a new OSD node to the cluster by using the `ceph-deploy` utility with the `osd_crush_initial_weight` option set to `0`, the value of the `MAX AVAIL` field in the output of the `ceph df` command was `0` for each pool instead of the proper numerical value. As a consequence, other applications using Ceph, such as OpenStack Cinder, assumed that there is no space available to provision new volumes. This bug has been fixed, and `ceph df` now shows proper value of `MAX AVAIL` as expected.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-09-29 12:56:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1335269
Bug Blocks:	1348597, 1372735

Description Mike Hackett 2016-02-11 21:08:28 UTC

Description of problem:

When adding new OSD's to a cluster using Ceph Deploy with 'osd_crush_initial_weight = 0' set. The output of 'ceph df' reports 'MAX AVAIL' to be 0 instead of the proper value until the weight is changed to 0.01, then ceph df displays proper numerical values. This causes problems for OpenStack Cinder in Kilo because it thinks there isn't any available space for new volumes.

Before adding an OSD:

GLOBAL:
SIZE AVAIL RAW USED %RAW USED
589T 345T 243T 41.32
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
data 0 816M 0 102210G 376
metadata 1 120M 0 102210G 94
images 5 11990G 1.99 68140G 1536075
volumes 6 63603G 10.54 68140G 16462022
instances 8 5657G 0.94 68140G 1063602
rbench 12 260M 0 68140G 22569
scratch 13 40960 0 68140G 10

After adding an OSD:

GLOBAL:
SIZE AVAIL RAW USED %RAW USED
590T 346T 243T 41.24
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
data 0 816M 0 0 376
metadata 1 120M 0 0 94
images 5 11990G 1.98 0 1536075
volumes 6 63603G 10.52 0 16462022
instances 8 5657G 0.94 0 1063602
rbench 12 260M 0 0 22569
scratch 13 40960 0 0 10

Max Avail is showing 0's for all pools.

Version-Release number of selected component (if applicable): 
RHCS 1.3.1
RHEL 7.2


How reproducible:
Highly reproducible, I was able to reproduce on my cluster without any issues.

Steps to Reproduce:

2 separate networks cluster and public
1 admin deploy node
3 Mon nodes
3 OSD nodes - 3 OSD disk 3 separate journals
Running RHCS 1.3.1 and RHEL 7.2

I have edited my ceph conf to include:
osd_crush_initial_weight = 0

Action plan:

check ceph df - verfiy max avail shows expected value.
Add 1 additional OSD node with 3 OSD's and 3 separate journals taking the initial OSD CRUSH weight of 0 from the conf

check ceph df - verify issue is seen with max avail at 0

If issue is seen, change osd CRUSH weight to 0.01 and recheck that max avail shows proper numerical values


Actual results:
Max Avail shows 0

Expected results:
Max Avail to show proper numerical counts from the pools

Additional info:
Upstream Ceph tracker open as issue is also seen on 94.5
http://tracker.ceph.com/issues/14710

Comment 1 Mike Hackett 2016-02-11 21:10:41 UTC

Assigned bug to kchai per sjust request.

Comment 2 Mike Hackett 2016-02-11 21:18:56 UTC

Better representation of test from 'ceph df' output:

Before adding an OSD:

GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    589T      345T         243T         41.32
POOLS:
    NAME          ID     USED       %USED     MAX AVAIL     OBJECTS
    data          0        816M         0       102210G          376
    metadata      1        120M         0       102210G           94
    images        5      11990G      1.99        68140G      1536075
    volumes       6      63603G     10.54        68140G     16462022
    instances     8       5657G      0.94        68140G      1063602
    rbench        12       260M         0        68140G        22569
    scratch       13      40960         0        68140G           10

After adding an OSD:

GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED
    590T      346T         243T         41.24
POOLS:
    NAME          ID     USED       %USED     MAX AVAIL     OBJECTS
    data          0        816M         0             0          376
    metadata      1        120M         0             0           94
    images        5      11990G      1.98             0      1536075
    volumes       6      63603G     10.52             0     16462022
    instances     8       5657G      0.94             0      1063602
    rbench        12       260M         0             0        22569
    scratch       13      40960         0             0           10

Comment 4 Kefu Chai 2016-02-22 15:25:14 UTC

merged into hammer https://github.com/ceph/ceph/pull/6834.

Comment 8 Federico Lucifredi 2016-07-25 17:59:00 UTC

Fix in 0.94.7 -

Comment 13 Ramakrishnan Periyasamy 2016-09-09 10:31:29 UTC

Verified adding OSD to cluster having data & Journal partition on same disk and different disks, "ceph df" command has "MAX AVAIL" data. Output captured for reference. 

[root@host1]# ceph osd tree
ID WEIGHT   TYPE NAME         UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 10.79991 root default                                        
-2  2.69998     host magna105                                   
 0  0.89999         osd.0          up  1.00000          1.00000 
 1  0.89999         osd.1          up  1.00000          1.00000 
 2  0.89999         osd.2          up  1.00000          1.00000 
-3  2.69998     host magna107                                   
 3  0.89999         osd.3          up  1.00000          1.00000 
 4  0.89999         osd.4          up  1.00000          1.00000 
 5  0.89999         osd.5          up  1.00000          1.00000 
-4  2.69998     host magna108                                   
 6  0.89999         osd.6          up  1.00000          1.00000 
 7  0.89999         osd.7          up  1.00000          1.00000 
 8  0.89999         osd.8          up  1.00000          1.00000 
-5  2.69997     host magna109                                   
10  0.89999         osd.10         up  1.00000          1.00000 
 9  0.89999         osd.9          up  1.00000          1.00000 
11  0.89998         osd.11         up  1.00000          1.00000 
-6        0     host magna110                                   
12        0         osd.12         up  1.00000          1.00000 
[root@host1]# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    12043G     11947G       98290M          0.80 
POOLS:
    NAME      ID     USED      %USED     MAX AVAIL     OBJECTS 
    rbd       0      7173M      0.17         3655G     1836528 
    pool1     1      2456M      0.06         3655G      628740 
    pool2     2      2556M      0.06         3655G      654449 
[root@host1]# ceph -v
ceph version 0.94.9-1.el7cp (72b3e852266cea8a99b982f7aa3dde8ca6b48bd3)

Comment 16 errata-xmlrpc 2016-09-29 12:56:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1972.html