Bug 1268533 - ceph -s reports incorrect block size in the client I/O section
ceph -s reports incorrect block size in the client I/O section
Status: CLOSED NOTABUG
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD (Show other bugs)
1.2.3
x86_64 Linux
unspecified Severity medium
: rc
: 1.3.2
Assigned To: Josh Durgin
ceph-qe-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-02 20:48 EDT by Narendra Trivedi
Modified: 2017-07-30 11:32 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-06 12:13:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Narendra Trivedi 2015-10-02 20:48:32 EDT
Description of problem:

When running FIO performance tests of 4KB writes directed towards a ceph cluster that is not in use (there is no client activity before the starting the test), the "ceph -s" of output shows 2KB of writes instead of 4KB.  


Version-Release number of selected component (if applicable):

ceph version 0.80.9


How reproducible:

Run FIO tests of 4KB random writes directed towards a ceph cluster. In this case, a VM with boot volume resident on the Ceph cluster was created and FIO was run against it. Run "ceph -s" on the cluster and grep on "client io" to see the client I/O. The output shows only 2KB of writes although FIO is writing 4KB of blocks of data.  

Steps to Reproduce:
1. Create a VM with the boot volume or at least one volume resident on the ceph cluster
2. RUn FIO to so that 4KB of random writes are directed towards this ceph-resident volume
3. Observe the client I/Os like this: 
for i in {1..30}; do ceph -s 2>&1 | grep "client io" | tee -a output.txt; sleep 2; done

Actual results:

client io 0 B/s rd, 42285 kB/s wr, 21164 op/s
  client io 0 B/s rd, 38088 kB/s wr, 17240 op/s
  client io 0 B/s rd, 30999 kB/s wr, 13648 op/s
  client io 0 B/s rd, 34726 kB/s wr, 16193 op/s
  client io 0 B/s rd, 52878 kB/s wr, 25058 op/s
  client io 0 B/s rd, 38969 kB/s wr, 18347 op/s
  client io 44698 kB/s wr, 21358 op/s
  client io 0 B/s rd, 45960 kB/s wr, 19663 op/s
  client io 0 B/s rd, 45419 kB/s wr, 22476 op/s
  client io 0 B/s rd, 48987 kB/s wr, 23173 op/s
  client io 0 B/s rd, 60230 kB/s wr, 28496 op/s
  client io 0 B/s rd, 54499 kB/s wr, 26146 op/s
  client io 0 B/s rd, 34186 kB/s wr, 16025 op/s
  client io 0 B/s rd, 30595 kB/s wr, 13111 op/s
  client io 0 B/s rd, 36604 kB/s wr, 17799 op/s
  client io 0 B/s rd, 45193 kB/s wr, 19905 op/s
  client io 0 B/s rd, 35855 kB/s wr, 16933 op/s
  client io 0 B/s rd, 44043 kB/s wr, 19807 op/s
  client io 0 B/s rd, 38977 kB/s wr, 17639 op/s
  client io 0 B/s rd, 43831 kB/s wr, 20539 op/s
  client io 0 B/s rd, 45128 kB/s wr, 20306 op/s
  client io 0 B/s rd, 35571 kB/s wr, 16344 op/s
  client io 0 B/s rd, 32537 kB/s wr, 14783 op/s

Expected results:

client io to op/s ratio should be 4KB and not 2KB as the above shows. 

Additional info:

ceph -s output before the tests were run: 

[root@svl-csl-b-ceph2-001 tmp]# ceph -s
    cluster fa2e0156-7251-4e33-99cf-097b76bcdc92
     health HEALTH_WARN noscrub,nodeep-scrub flag(s) set
     monmap e5: 5 mons at {svl-csl-b-ceph2-001=10.114.194.11:6789/0,svl-csl-b-ceph2-002=10.114.194.12:6789/0,svl-csl-b-ceph2-018=10.114.194.28:6789/0,svl-csl-b-ceph2-035=10.114.194.45:6789/0,svl-csl-b-ceph2-036=10.114.194.47:6789/0}, election epoch 5286, quorum 0,1,2,3,4 svl-csl-b-ceph2-001,svl-csl-b-ceph2-002,svl-csl-b-ceph2-018,svl-csl-b-ceph2-035,svl-csl-b-ceph2-036
     osdmap e350614: 158 osds: 156 up, 156 in
            flags noscrub,nodeep-scrub
      pgmap v27090162: 25514 pgs, 29 pools, 22467 GB data, 7532 kobjects
            69939 GB used, 355 TB / 423 TB avail
               25514 active+clean
  client io 0 B/s rd, 0 kB/s wr, 0 op/s
Comment 2 Josh Durgin 2015-10-06 12:13:27 EDT
There are a few reasons the reported client i/o would not match the 4kb block size:

1) This is an estimate of total client io per second, not client i/o size - particular emphasis on the estimate - it would be too expensive to exactly track all client i/o and report it to the monitors for display in 'ceph -s'

2) The i/o size generated by fio in a vm may not translated directly to the same size i/o in rbd - it has to go through the filesystem and block layers in the vm, which may split up, merge, or generate extra i/o before getting to the rbd layer.

You can measure what is being sent to rbd via blktrace in the vm, or by enabling 'debug rbd = 20' logging for the vm and look at the logs for aio_read and aio_write.

Note You need to log in before you can comment on or make changes to this bug.