Bug 1332470 - rados bench sometimes numbers not separated by blank
Summary: rados bench sometimes numbers not separated by blank
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RADOS
Version: 1.3.2
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 1.3.3
Assignee: Ken Dreyer (Red Hat)
QA Contact: Vasishta
Bara Ancincova
URL:
Whiteboard:
Depends On:
Blocks: 1372735
TreeView+ depends on / blocked
 
Reported: 2016-05-03 09:37 UTC by Vikhyat Umrao
Modified: 2020-02-14 17:46 UTC (History)
8 users (show)

Fixed In Version: RHEL: ceph-0.94.7-5.el7cp Ubuntu: ceph_0.94.7-3redhat1trusty
Doc Type: Bug Fix
Doc Text:
.The columns in the "rados bench" command output are now separated correctly This update ensures that the columns in the `rados bench` command output are separated correctly.
Clone Of:
Environment:
Last Closed: 2016-09-29 12:58:14 UTC
Target Upstream Version:


Attachments (Terms of Use)
Patch (7.23 KB, patch)
2016-05-06 07:30 UTC, Brad Hubbard
no flags Details | Diff
rados bench test with fixes in PR : https://github.com/ceph/ceph/pull/8960 (5.04 KB, text/plain)
2016-05-12 11:14 UTC, Vikhyat Umrao
no flags Details


Links
System ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 15703 None None None 2016-05-03 09:41:50 UTC
Ceph Project Bug Tracker 15717 None None None 2016-05-12 08:18:07 UTC
Red Hat Knowledge Base (Solution) 2295271 None None None 2016-05-03 10:58:35 UTC
Red Hat Product Errata RHSA-2016:1972 normal SHIPPED_LIVE Moderate: Red Hat Ceph Storage 1.3.3 security, bug fix, and enhancement update 2016-09-29 16:51:21 UTC

Description Vikhyat Umrao 2016-05-03 09:37:46 UTC
Description of problem:

rados bench sometimes numbers not separated by blank

`rados bench` output  is not readable, as numbers sometimes are not separated by a blank.

Expected behavior: Numbers should be separated by a blank.



Actual output:

2016-04-27 17:40:30.470978min lat: 0.00161851 max lat: 4.61329 avg lat: 0.0769791
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
  1060     256   3524928   3524672   25.9746   25.3594 0.0149067 0.0769791
  1061     256   3527461   3527205   25.9688   19.7891 0.0144534 0.0769785
  1062     256   3530718   3530462   25.9683   25.44530.00639977 0.0769856   ---> here numbers are not separated




Version-Release number of selected component (if applicable):
Red  Hat Ceph Storage 1.3.2 
ceph-0.94.5-9.el7cp

How reproducible:
Always

Comment 2 Vikhyat Umrao 2016-05-03 09:48:11 UTC
Hammer Source File :  src/common/obj_bencher.cc

 if (i % 20 == 0) {
      if (i > 0)
        cur_time.localtime(cout) << "min lat: " << data.min_latency
             << " max lat: " << data.max_latency
             << " avg lat: " << data.avg_latency << std::endl;
      //I'm naughty and don't reset the fill
      bencher->out(cout, cur_time) << setfill(' ')
           << setw(5) << "sec"
           << setw(8) << "Cur ops"
           << setw(10) << "started"
           << setw(10) << "finished"
           << setw(10) << "avg MB/s"
           << setw(10) << "cur MB/s"
           << setw(10) << "last lat"
           << setw(10) << "avg lat" << std::endl;
    }

last lat and avg lat here has setw(10) : which is causing the issue.


But in Jewel it was changed to :

  bencher->out(cout, cur_time) << setfill(' ')
          << setw(5) << "sec"
          << setw(8) << "Cur ops"
          << setw(10) << "started"
          << setw(10) << "finished"
          << setw(10) << "avg MB/s"
          << setw(10) << "cur MB/s"
          << setw(12) << "last lat(s)"
          << setw(12) << "avg lat(s)" << std::endl;
    }

*last lat* and *avg lat * is changed to setw(12) :  I hope this will solve the issue.

   << setw(12) << "last lat(s)"
   << setw(12) << "avg lat(s)" << std::endl;

Comment 8 Vikhyat Umrao 2016-05-04 04:24:09 UTC
Upstream Hammer backport : http://tracker.ceph.com/issues/15717

Comment 9 Brad Hubbard 2016-05-06 03:29:56 UTC
https://github.com/ceph/ceph/commit/669b9329 does not fix this problem.

2016-04-27 17:40:30.470978min lat:

Working on making 669b9329 work with hammer and adding an additional patch for the timestampnogapmin issue.

Comment 10 Brad Hubbard 2016-05-06 04:06:36 UTC
Just so we are clear.

The actual problem with the hammer source code is that we have setw(10) for both the column header and the field itself so they can both be the same size.

<< setw(10) << "last lat"
...
<< setw(10) << (double)data.cur_latency

In the newer code the columns have been made bigger but the field is restricted to one less char than the header so they can't be the same size and a space must always exist between fields *and* we've added a forced space as well in the output.

<< setw(12) << "last lat(s)"
...
<< ' ' << setw(11) << (double)data.cur_latency

That was introduced by this commit, https://github.com/ceph/ceph/commit/474c98961fe0aa85881929fb8e08550ce1666197. I'll need to do a bit more investigation and testing to get the hammer commit right.

Comment 13 Brad Hubbard 2016-05-06 07:28:42 UTC
I've created https://github.com/ceph/ceph/pull/8960

I'm attaching hammer-rados-bench-formatting.patch which includes all of the patches in that PR. If it applies cleanly (or maybe with a bit of massaging) you could create a test package and you, and the customer, could test it. Since it only requires a client system it should be very low impact.

Comment 14 Brad Hubbard 2016-05-06 07:30:00 UTC
Created attachment 1154501 [details]
Patch

Diff including all commits from https://github.com/ceph/ceph/pull/8960

Comment 15 Vikhyat Umrao 2016-05-06 13:24:42 UTC
(In reply to Brad Hubbard from comment #14)
> Created attachment 1154501 [details]
> Patch
> 
> Diff including all commits from https://github.com/ceph/ceph/pull/8960

Thanks Brad. Sure I will try to test it.

Comment 17 Vikhyat Umrao 2016-05-12 11:14:11 UTC
Created attachment 1156637 [details]
rados bench test with fixes in PR : https://github.com/ceph/ceph/pull/8960

I have tested rados bench after applying the patches in above listed PR and now it looks perfect.

Comment 18 Vikhyat Umrao 2016-05-12 11:16:02 UTC
(In reply to Brad Hubbard from comment #13)
> I've created https://github.com/ceph/ceph/pull/8960
> 
> I'm attaching hammer-rados-bench-formatting.patch which includes all of the
> patches in that PR. If it applies cleanly (or maybe with a bit of massaging)
> you could create a test package and you, and the customer, could test it.
> Since it only requires a client system it should be very low impact.

Brad, Please find test results in comment#17 attachment.  Now rados bench output looks perfect.

Comment 26 errata-xmlrpc 2016-09-29 12:58:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1972.html


Note You need to log in before you can comment on or make changes to this bug.