Description of problem: rados bench sometimes numbers not separated by blank `rados bench` output is not readable, as numbers sometimes are not separated by a blank. Expected behavior: Numbers should be separated by a blank. Actual output: 2016-04-27 17:40:30.470978min lat: 0.00161851 max lat: 4.61329 avg lat: 0.0769791 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 1060 256 3524928 3524672 25.9746 25.3594 0.0149067 0.0769791 1061 256 3527461 3527205 25.9688 19.7891 0.0144534 0.0769785 1062 256 3530718 3530462 25.9683 25.44530.00639977 0.0769856 ---> here numbers are not separated Version-Release number of selected component (if applicable): Red Hat Ceph Storage 1.3.2 ceph-0.94.5-9.el7cp How reproducible: Always
Hammer Source File : src/common/obj_bencher.cc if (i % 20 == 0) { if (i > 0) cur_time.localtime(cout) << "min lat: " << data.min_latency << " max lat: " << data.max_latency << " avg lat: " << data.avg_latency << std::endl; //I'm naughty and don't reset the fill bencher->out(cout, cur_time) << setfill(' ') << setw(5) << "sec" << setw(8) << "Cur ops" << setw(10) << "started" << setw(10) << "finished" << setw(10) << "avg MB/s" << setw(10) << "cur MB/s" << setw(10) << "last lat" << setw(10) << "avg lat" << std::endl; } last lat and avg lat here has setw(10) : which is causing the issue. But in Jewel it was changed to : bencher->out(cout, cur_time) << setfill(' ') << setw(5) << "sec" << setw(8) << "Cur ops" << setw(10) << "started" << setw(10) << "finished" << setw(10) << "avg MB/s" << setw(10) << "cur MB/s" << setw(12) << "last lat(s)" << setw(12) << "avg lat(s)" << std::endl; } *last lat* and *avg lat * is changed to setw(12) : I hope this will solve the issue. << setw(12) << "last lat(s)" << setw(12) << "avg lat(s)" << std::endl;
Upstream Hammer backport : http://tracker.ceph.com/issues/15717
https://github.com/ceph/ceph/commit/669b9329 does not fix this problem. 2016-04-27 17:40:30.470978min lat: Working on making 669b9329 work with hammer and adding an additional patch for the timestampnogapmin issue.
Just so we are clear. The actual problem with the hammer source code is that we have setw(10) for both the column header and the field itself so they can both be the same size. << setw(10) << "last lat" ... << setw(10) << (double)data.cur_latency In the newer code the columns have been made bigger but the field is restricted to one less char than the header so they can't be the same size and a space must always exist between fields *and* we've added a forced space as well in the output. << setw(12) << "last lat(s)" ... << ' ' << setw(11) << (double)data.cur_latency That was introduced by this commit, https://github.com/ceph/ceph/commit/474c98961fe0aa85881929fb8e08550ce1666197. I'll need to do a bit more investigation and testing to get the hammer commit right.
I've created https://github.com/ceph/ceph/pull/8960 I'm attaching hammer-rados-bench-formatting.patch which includes all of the patches in that PR. If it applies cleanly (or maybe with a bit of massaging) you could create a test package and you, and the customer, could test it. Since it only requires a client system it should be very low impact.
Created attachment 1154501 [details] Patch Diff including all commits from https://github.com/ceph/ceph/pull/8960
(In reply to Brad Hubbard from comment #14) > Created attachment 1154501 [details] > Patch > > Diff including all commits from https://github.com/ceph/ceph/pull/8960 Thanks Brad. Sure I will try to test it.
Created attachment 1156637 [details] rados bench test with fixes in PR : https://github.com/ceph/ceph/pull/8960 I have tested rados bench after applying the patches in above listed PR and now it looks perfect.
(In reply to Brad Hubbard from comment #13) > I've created https://github.com/ceph/ceph/pull/8960 > > I'm attaching hammer-rados-bench-formatting.patch which includes all of the > patches in that PR. If it applies cleanly (or maybe with a bit of massaging) > you could create a test package and you, and the customer, could test it. > Since it only requires a client system it should be very low impact. Brad, Please find test results in comment#17 attachment. Now rados bench output looks perfect.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-1972.html