Description of problem:

10 volumes each 10TB were created in an RBD pool. The volumes were filled with unique data (non-sparse) to add used data to the cluster that shows as only 6.73% being used. After writing  65TB of non-sparse data to the cluster (total capacity 1176TB) % RAW USED shoots to only 6.74% which is impossible. The RAW USED shows 81165G. Before filing up the cluster, it was 81107G. Hence, the RAW USED data increased only by 58G as per ceph df and rados df!   

Version-Release number of selected component (if applicable):
1.2.3

How reproducible:
Reproducible every time 

Steps to Reproduce:
1. Create some volumes in a pool with the intention of filling up the pool to more than 50% 
2. After attaching it to some filesystem, execute the following command on a Linux system inside the mount point (say /mnt/vdb where /dev/sdb is the volume):

for i in {1..10}; do ruby -e 'a=STDIN.readlines;100000000.times do;b=[];1000.times do; b << a[rand(a.size)].chomp end; puts b.join(" "); end' < /usr/share/dict/words > file$i.txt; done
 
3. Repeat for all the volumes till the total usage shoots to > 50%

Actual results:

The total non-sparse data written to the pool is as per the df -h : 

1) [root@pavana-vm vdb]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        47G  1.3G   44G   3% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G   17M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vdc        985G  985G     0 100% /mnt/vdc
/dev/vdb        985G  985G     0 100% /mnt/vdb
/dev/vdd        985G  478G  458G  52% /mnt/vdd
/dev/vde        985G  985G     0 100% /mnt/vde
/dev/vdf        985G  932G  3.2G 100% /mnt/vdf
/dev/vdg        985G  476G  459G  51% /mnt/vdg
/dev/vdh        985G  474G  462G  51% /mnt/vdh
/dev/vdi        985G  392G  543G  42% /mnt/vdi
/dev/vdj        985G  377G  559G  41% /mnt/vdj
/dev/vdk        985G  396G  539G  43% /mnt/vdk


2) df -h | grep mnt | grep vd | awk '{print $3}' | cut -d"G" -f1 | awk '{sum += $1} END {print sum}'
6511

3) Before writing 65TB of data to the alln01-a-css-cinder-volume-1 pool, the ceph df shows this: 

POOLS:
    NAME                      ID     USED       %USED     MAX AVAIL     OBJECTS


alln01-a-css-cinder-volume-1   43    23142G      1.92         358T      7663661

After writing 65TB, below is the output of ceph df for the pool: 

alln01-a-css-cinder-volume-1    43   23144G      1.92          358T     7664486

Expected results:

The pool USED field in ceph df output of the pool alln01-a-css-cinder-volume-1 should have increased by 65TB. 


Additional info:
[Integration]root@css-a-ceph1-001:~# ceph -v
ceph version 0.80.8.5 (4ec83c2359fe63663ae28befbcaa65e755bcd462)